Publications, Patents, and Panels

Publications

Bader-Natal, A. CCCApply and OpenCCC Issues and Opportunities. [Memorandum] (2020, October)
Abstract PDF PDF

We appreciate the many benefits of all colleges in the California Community Colleges being able to share a common infrastructure for the student-facing college application process. CCCApply serves as a "gateway to the California Community Colleges," and its existence eliminates the need for each college or district to independently procure and customize (or build and maintain) its own application system, and its annual improvements benefit colleges system-wide. But while all colleges in the system share the benefits of this common infrastructure, all colleges also share the limitations of this infrastructure. We focus on six key issues with CCCApply and OpenCCC today that affect prospective students and the colleges that serve them. We believe these recommendations will result in (a.) a more streamlined, cohesive experience for incoming students system-wide, and (b.) an enhanced ability for colleges to remotely support all of their prospective students more equitably through the entire application process.

Bader-Natal, A., Fost, J., and Genone, J. Building Lesson Plans for Twenty-First-Century Active Learning. In Building the Intentional University: Minerva and the Future of Higher Education, edited by Stephen M. Kosslyn and Ben Nelson, MIT Press, 2017.
Katzman, J., Regan, M., and Bader-Natal, A. The Active Learning Forum. In Building the Intentional University: Minerva and the Future of Higher Education, edited by Stephen M. Kosslyn and Ben Nelson, MIT Press, 2017.
Levitt, R., Bader-Natal, A., and Chandler, V. Assessing Student Learning. In Building the Intentional University: Minerva and the Future of Higher Education, edited by Stephen M. Kosslyn and Ben Nelson, MIT Press, 2017.
Bader-Natal, A., Lotze, T., and Furr, D. A Comparison of the Effects of Nine Activities within a Self-Directed Learning Environment on Skill-Grained Learning. Proceedings of the 15th International Conference on Artificial Intelligence in Education, 2011.
Abstract Draft PDF Draft PDF Springer Springer

Self-directed learners value the ability to make decisions about their own learning experiences. Educational systems can accommodate these learners by providing a variety of different activities and study contexts among which learners may choose. When creating a software-based environment for these learners, system architects incorporate activities designed to be both effective and engaging. Once these activities are made available to students, researchers can evaluate these activities by analyzing observed usage and performance data by asking: Which of these activities are most engaging? Which are most effective? Answers to these questions enable a system designer to highlight and encourage those activities that are both effective and popular, to refine those that are either effective or popular, and to reconsider or remove those that are neither effective nor popular. In this paper, we discuss Grockit - a web-based environment offering self-directed learners a wide variety of activities - and use a mixed-effects logistic regression model to model the effectiveness of nine of these supplemental interventions on skill-grained learning.

Bader-Natal, A. and Lotze, T. Evolving a learning analytics platform. Proceedings of the 1st International Conference on Learning Analytics and Knowledge. ACM Press, 2011.
Abstract Draft PDF Draft PDF Talk Slides Talk Slides ACM Digital Library ACM Digital Library

Web-based learning systems offer researchers the ability to collect and analyze fine-grained educational data on the performance and activity of students, as a basis for better understanding and support- ing learning among those students. The availability of this data enables stakeholders to pose a variety of interesting questions, often specifically focused on some subset of students. As a system matures, the number of stakeholders, the number of interesting questions, and the number of relevant sub-populations of students also grow, adding complexity to the data analysis task. In this work, we describe an internal analytics system designed and developed to address this challenge, adding flexibility and scalability. Here we present several examples of typical examples of analysis, discuss a few uncommon but powerful use-cases, and share lessons learned from the first two years of iteratively developing the platform.

Bader-Natal, A. Methodologies and Technologies of the Grockit Learning Platform. (Unpublished report.)
Abstract Draft PDF Draft PDF

Grockit provides a place for students to master new concepts and exercise what they learn through a set of study modes designed to accommodate a variety of learning styles and learner preferences. These include: (1.) small group study, which leverages the power of collaborative learning dynamics to provide students with a social learning network that can help motivate and assist them, (2.) individual study, which builds and uses a data-driven model of a student's abilities to provide that student with appropriate challenges for learning, (3.) instructor-led classes, which draw on a teacher's domain knowledge and experience to provide a guided and structured path for groups of learners.

Bader-Natal, A. Combining peer-assistance and peer-assessment in a synchronous collaborative learning activity. Supplementary Proceedings of the Tenth International Conference on Intelligent Tutoring Systems (ITS-2010), Workshop on Opportunities for intelligent and adaptive behavior in collaborative learning systems.
Abstract Draft PDF Draft PDF Workshop proceedings Workshop proceedings

One unique characteristic of learning systems that support peer collaboration is that these systems have the potential to supplement or replace software-based representations of domain- and learner-models with the representations implicitly formed by peers. In order to realize this potential, a collaborative activity must sufficiently motivate peers to reflect, collect, and communicate these mental models. Peer-assessment represents a class of activities that address this challenge by design. In this work, we describe a project, currently under development, in which peer-assessment is melded with peer-instruction to create a new learning activity for an existing collaborative learning platform. We present the rationale behind the design of the activity, focusing specifically on how it draws from and synthesizes the three modes of learning supported by the Grockit platform: adaptive individual study, live collaborative small-group study, and instructor-led skill-focused lessons. By treating teaching as a demonstration of learning, we illustrate how a single activity can peer-assess mastery and peer-assist learning.

Bader-Natal, A. Interaction Synchronicity in Web-based Collaborative Learning Systems. Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2009 (E-LEARN 2009), pp. 1121-1129.
Abstract Draft PDF Draft PDF Talk Slides Talk Slides EdITLib EdITLib

While many web-based learning systems connect students asynchronously, fewer systems focus on facilitating synchronous interactions among learners. Given the value of real-time communication - the social and motivational benefits of having a cohort of peers and the ability for a student to get immediate answers to pressing questions - it is perhaps surprising that more systems do not support interaction synchronicity. We suggest that this is due, in part, to a mismatch between the hypertext document-oriented nature of the web and the social activity-oriented nature of learning, and we explore how several systems address this discrepancy. We discuss Grockit, a web-based learning environment that we designed to support both synchronous and asynchronous interactions, and share lessons learned from grappling with the choices enabled by this flexibility: Which interactions should to be synchronous? Which should be asynchronous? Which should be a mix? What should that mix be?

Bader-Natal, A. Incorporating game mechanics into a network of online study groups. Supplementary Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED-2009). Intelligent Educational Games workshop, 2009.
Abstract Draft PDF Draft PDF Full proceedings Full proceedings

The recent movement towards publishing open educational resources has increased the variety and quantity of learning materials available to students outside of the traditional classroom environment. Several core characteristics of the classroom environment, however, are difficult to offer through a web-based interface, including: (1) interaction and camaraderie among a cohort of peers, (2) the ability to get "real-time" answers to pressing questions, and (3) a motivating force to keep the student engaged over time. An online learning environment can approximate the value of peer cohorts and live question-answering by supporting (and encouraging) synchronous interactions among individuals studying a common topic. A learning system can motivate participation and collaboration by incorporating elements of game mechanics in the activity. We discuss Grockit, a recently-launched website that combines a virtual study group format with multi-player game dynamics to provide an engaging live collaborative learning environment for geographically-dispersed learners.

Bader-Natal, A. The Teacher's Dilemma: A game-based approach for motivating appropriate challenge among peers. Ph.D. Dissertation, Brandeis University, May 2008.
Abstract Full PDF Full PDF ProQuest ProQuest

In classroom-based studies, peer tutoring has proved to be an effective learning strategy, both for the tutees and for their peer tutors. Today, the increasingly widespread availability of computers and internet access in the homes and after-school programs of students offers a new venue for peer learning. In seeking to translate the successes of peer-assisted learning from the classroom to the Internet, one major hurdle to overcome is that of motivation. When teachers are no longer supervising student activity and when participation itself becomes voluntary, peer tutoring protocols may stop being educationally productive. In order to successfully leverage these peer interactions, we must find a way to facilitate and motivate learning among a group of unsupervised peers. In this dissertation, we respond to this challenge by reconceptualizing the interactions among peers within the context of a different medium: that of games. In designing a peer-tutoring experience as a two-player game, we gain a valuable set of tools and techniques for affecting student participation, engagement, goals, and strategies.

Our contributions:

  1. We define a criteria for games -- the Teacher's Dilemma criteria -- that motivates players to challenge one another with problems of appropriate difficulty;
  2. We show three games that satisfy the Teacher's Dilemma criteria when played by rational players under idealized conditions;
  3. We demonstrate, using computer simulations of strategic dynamics, that game-play will converge towards meeting these criteria, through time, under more realistic conditions;
  4. We design a suite of software that incorporates a Teacher's Dilemma game into several web-based activities for different learning domains;
  5. We collect data from thousands of students using these activities, and examine how the games actually affected the game-play strategy and learning among these students.
The game-theoretic analysis establishes the possibility for a game-based mechanism for motivating appropriate challenges, the simulations support the plausibility of this approach given non-optimal players, the implemented software systems demonstrate the scalability of this model, and the data analysis supports the real-world applicability of this game-based approach to motivating appropriate challenges for learning among unsupervised peers.

Bader-Natal, A. Incorporating students' probabilistic expectations into a peer-driven tutoring game. Brandeis CS Tech Report CS-08-269, 2008.
Abstract Draft PDF Draft PDF

Games provide a promising mechanism for intelligent tutoring systems in that they offer means to influence motivation and structure interactions. We have designed and released several game-based tutoring systems in which students learn to identify the best game strategies to adopt, and, in doing so, create for each other increasingly productive learning environments. Here, we first detail the core game underlying our deployed systems, designed to leverage human intelligence in tutoring systems through the tutor's identification of "appropriate" challenges for their tutee. While this game works well for task domains in which problem difficulty is known, it cannot be applied to domains if nothing is known about a problem beyond its correct solution. We introduce a second, more robust, game here capable of addressing this larger set of task domains. By incorporating player-generated probability estimates (in place of a difficulty metric), we show that a game can be designed to simultaneously elicit best-effort responses from tutees, honest statements of probability estimates from tutees, and appropriate challenges from tutors. We derive a set of constraints on the parameterized version of this game necessary for rational players to converge on this "Teacher's Dilemma" learning environment. Beyond providing a foundation for future tutoring systems, this work offers a new mechanism with which to simultaneously leverage and enhance the knowledge of peer learners.

Bader-Natal, A. and Pollack, J. Evaluating Problem Difficulty Rankings Using Sparse Student Response Data. Supplementary Proceedings of the 13th International Conference on Artificial Intelligence in Education, 2007.
Abstract Draft PDF Draft PDF Full proceedings Full proceedings

Problem difficulty estimates play important roles in a wide variety of educational systems, including determining the sequence of problems presented to students and the interpretation of the resulting responses. The accuracy of these metrics are therefore important, as they can determine the relevance of an educational experience. For systems that record large quantities of raw data, these observations can be used to test the predictive accuracy of an existing difficulty metric. In this paper, we examine how well one rigorously developed - but potentially outdated - difficulty scale for American-English spelling fits the data collected from seventeen thousand students using our SpellBEE peer-tutoring system. We then attempt to construct alternate metrics that use collected data to achieve a better fit. The domain-independent techniques presented here are applicable when the matrix of available student-response data is sparsely populated or non-randomly sampled. We find that while the original metric fits the data relatively well, the data-driven metrics provide approximately 10% improvement in predictive accuracy. Using these techniques, a difficulty metric can be periodically or continuously recalibrated to ensure the relevance of the educational experience for the student.

Bader-Natal, A. and Pollack, J. Assessing Learning in a Peer-Driven Tutoring System. Proceedings of the 13th International Conference on Artificial Intelligence in Education, IOS Press, 2007.
Abstract Draft PDF Draft PDF ACM Digital Library ACM Digital Library

In many intelligent tutoring systems, a detailed model of the task domain is constructed and used to provide students with assistance and direction. Reciprocal tutoring systems, however, can be constructed without needing to codify a full-blown model for each new domain. This provides various advantages: these systems can be developed rapidly and can be applied to complex domains for which detailed models are not yet known. In systems built on the reciprocal tutoring model, detailed validation is needed to ensure that learning indeed occurs. Here, we provide such validation for SpellBEE, a reciprocal tutoring system for the complex task domain of American-English spelling. Using a granular definition of response accuracy, we present a statistical study designed to assess and characterize student learning from collected data. We find that students using this reciprocal tutoring system exhibit learning at the word, syllable, and grapheme levels of task granularity.

Bader-Natal, A. and Pollack, J. BEEweb: A Multi-Domain Platform for Reciprocal Peer-Driven Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Springer, 2006.
Abstract Draft PDF Draft PDF Springer Springer

Tutoring systems that engage each student as both a tutee and a tutor can be powerfully enhanced by motivating each tutor to try to appropriately challenge their tutee. The BEEweb platform is presented as a foundation upon which to build such systems, based upon the Reciprocal Tutoring protocol and the Teacher's Dilemma. Three systems that have recently been built on the BEEweb platform are introduced.

Bader-Natal, A. and Pollack, J. Towards Metrics and Visualizations Sensitive to Coevolutionary Failures. AAAI Technical Report FS-05-03. Coevolutionary and Coadaptive Systems, AAAI Fall Symposium 2005, Washington D.C. 2005.
Abstract Draft PDF Draft PDF AAAI Press AAAI Press Talk Slides Talk Slides

The task of monitoring success and failure in coevolution is inherently difficult, as domains need not have any external metric to measure performance. Past metrics and visualizations for coevolution have been limited to identification and measurement of success but not failure. We suggest circumventing this limitation by switching from "best-of-generation"-based techniques to "all-of-generation"-based techniques. Using "all-of-generation" data, we demonstrate one such technique - a population-differential technique - that allows us to profile and distinguish an assortment of coevolutionary successes and failures, including arms-race dynamics, disengagement, cycling, forgetting, and relativism.

Bader-Natal, A. and Pollack, J.B. Motivating Appropriate Challenges in a Reciprocal Tutoring System. Proceedings of the 12th International Conference on Artificial Intelligence in Education, IOS Press, 2005. Nominated for Best Paper Award.
Abstract Draft PDF Draft PDF ACM Digital Library ACM Digital Library

Formalizing a student model for an educational system requires an engineering effort that is highly domain-specific. This model-specificity limits the ability to scale a tutoring system across content domains. In this work we offer an alternative, in which the task of student modeling is not performed by the system designers. We achieve this by using a reciprocal tutoring system in which peer-tutors are implicitly tasked with student modeling. Students are motivated, using the Teacher's Dilemma, to use these models to provide appropriately-difficult challenges. We implement this as a basic literacy game in a spelling-bee format, in which players choose words for each other to spell across the internet. We find that students are responsive to the game's motivational structure, and we examine the affect on participants' spelling accuracy, challenge difficulty, and tutoring skill.

Bader-Natal, A. and Pollack, J.B. A Population-Differential Method of Monitoring Success and Failure in Coevolution. Proceedings of the 2004 Genetic and Evolutionary Computation Conference, Springer-Verlag, 2004.
Abstract Draft PDF Draft PDF Springer Springer Poster Poster

Coevolutionary algorithms require no domain-specific measure of objective fitness, enabling these algorithms to be applied to domains for which no objective metric is known or for which known metrics are too expensive. But this flexibility comes at the expense of accountability. Past work on monitoring has focused on measuring success, but has ignored failure. This limitation is due to a common reliance on "best-of-generation" (BOG) based analysis, and we propose a population-differential analysis based on an alternate "all-of-generation" (AOG) framework that is not similarly limited.

Patents

Bader-Natal et al., 2022. “Apparatus, user interface, and method for authoring and managing lesson plans and course design for virtual conference learning environments”U.S. Patent 11,217,109Filed September 19, 2017Issued January 4, 2022
Bader-Natal et al., 2020. “System and method for a virtual conference interactive timeline”U.S. Patent 10,666,696Filed August 31, 2015Issued May 26, 2020
Bader-Natal et al., 2020. “System and method for scalable, interactive virtual conferencing”U.S. Patent 10,541,824Filed June 21, 2017Issued January 21, 2020
Bader-Natal et al., 2018. “Participation queue system and method for online video conferencing”U.S. Patent 9,935,987Filed December 12, 2016Issued April 3, 2018
Bader-Natal et al., 2017. “System and method for tracking events and providing feedback in a virtual conference”U.S. Patent 9,674,243Filed August 31, 2015Issued June 6, 2017
Bader-Natal et al., 2017. “System and method for decision support in a virtual conference”U.S. Patent 9,578,073Filed August 31, 2015Issued February 21, 2017
Bader-Natal et al., 2018. “System and method for managing virtual conferencing breakout groups”U.S. Patent 9,961,119Filed March 31, 2015Issued May 1, 2018
Bader-Natal et al., 2019. “Registering and displaying visual attention metadata in a web video conferencing and seminar system”U.S. Patent 10,356,364Filed September 9, 2014Issued July 16, 2019

Invited Talks & Panels

Future Trends ForumReal innovation in higher education14 Jan 2021 Visit external link
Community College League of California Annual ConventionCalbright College: Lessons from Year One18 Nov 2020
EDUCAUSE Annual ConferenceThe Innovation Landscape in Higher Education27 Oct 2020 Visit external link
UCLA Educational Leadership Program (2019 EdTech Conference)1 June 2019
Duke University (Duke NextEd Festival)Future-Proof Learning Design Workshop8 Oct 2018 Visit external link
MKThinkApplying Data in Education: Discussion & Data Design Challenge16 Nov 2017 Visit external link
Minerva Schools at KGI (ACM Student Chapter)Transforming Vision into Reality: WIL with Ari Bader-Natal13 Nov 2017 Visit external link
Santa Fe Institute (Workshop on Gaming, Training, and Learning Analytics)9-10 Nov 2016
Carnegie Mellon University (Discussion Affordances for Natural Collaborative Exchange talk series)Supporting and assessing classroom discussion in an active learning seminar Abstract

In a small seminar classroom, how might we actively engage every student in the discussion? How might we provide each student with timely and actionable feedback on their contributions to a classroom discussion? How might we structure summative feedback so that students are evaluated not on what they produce for a final exam, but instead on what they demonstrate that they have mastered and can apply in new contexts, years later? These are a few of the questions that have guided the design of the Active Learning Forum, the core technology used to support and assess learning in every class at the Minerva Schools at KGI.

In this talk, I will introduce the Active Learning Forum, and discuss ways in which this computer-supported collaborative learning environment has been designed to support effective teaching and learning. With this background established, we can shift the discussion to whichever aspects of the Active Learning Forum that are of the most interest to this DANCE audience. This might include: the use of technologies for activity orchestration, the use of realtime feedback to help instructors moderate discussion, the use of video annotation as a basis for formative feedback, the use of a summative assessment framework that cuts across courses and over time to identify mastery and far transfer.

28 Jan 2016 Visit external link
University of California at Berkeley (Social Data Revolution, School of Information)28 Oct 2014 Visit external link
Mozilla (Etherpad Foundation Meetup)12 March 2013 Visit external link
Open Education Summit (O'Reilly OSCON & Shared Learning Collaborative)Unlocking the Third Dimension22 May 2012
United States Distance Learning Association conferenceHow Pseudo-Synchronous Interactions Can Enhance Video Instruction Abstract

Video is playing an increasingly important role in online learning, and exciting new web technologies are now emerging that enable a new level of peer interactions and learner collaborations around video instruction. For self-paced distance learners engaging with long-form presentations of complex concepts, content-centered peer dialogue can be incredibly valuable. Most video-hosting websites today support some limited peer interactions through a shared comment thread, but this format inherently treats the video as a two-dimensional object, ignoring its most meaningful dimension: time. What could peer interactions around web video look like when temporal context is treated as critically-relevant? In this session, we explore a number of freely-available tools that provide answers to this question, and share a set of concrete examples of how distance learning educators and administrators can benefit from using these tools in their courses and programs. Join us to learn how to spark lively student discussions around (and within) video instruction. Help us discuss and explore ways that this coming wave of web technologies can transform the distance learning experience.

29 April 2012
SF Edtech MeetupLeveraging Web Videos to Truly Flip the Classroom15 March 2012
Digital Media and Learning conferenceReflecting on four communities built for learning, tinkering, and remixing with code Abstract

In this panel, we bring together four people who have actively engaged in making, tinkering, and remixing the designs of learning communities in which young people make, tinker, and remix with code. Drawing on experiences creating and experimenting with Scratch Online, Studio Sketchpad, Kodu, and AppJet, we seek to identify the socio-technical factors that impact peer learning with social media. We will talk about lessons learned and have a discussion about questions such as how to support young people's development of a maker mindset.

We will introduce each of the four systems, drawing particular attention to the various mechanisms built-in for remixing, iteratively refining, collaborating, and sharing with others. We then use this as a starting point for identifying commonalities and differences to discuss in more detail. What do we agree are valuable components in creating a remix culture? What have we discovered to be promising, but ultimately unsuccessful? What styles of remixing have emerged and what seems to be their purpose? How generalizable are our observations?

This panel aims at providing insights for meta-designers, practitioners and educators interested in learning how to support and inspire young people to learn to create computational artifacts within a community of peers.

1 March 2012 Visit external link
Mozilla Labs NightSketchpad and Programming Visual Media Abstract

Studio Sketchpad creator Ari Bader-Natal will be sharing from his experiences teaching a P2PU School of Webcraft class called Programming Visual Media...

28 October 2010 Visit external link
Games, Learning and SocietyCollaborative Learning Games in the Virtual Classroom10 June 2010 Visit external link

Professional Service

Mentor - First Round Fast Track (2022-2023) Visit external link
Dean's Cabinet Member - Brandeis University Graduate School of Arts and Sciences (2022-2025) Visit external link
Governance Council Member - Calbright College (2020-2021)
Accreditation Steering Committee Member (Institutional Effectiveness) - Calbright College (2020-2021)
Faculty Advisor - Minerva Schools at KGI Association of Computing Machinery (ACM) Student Chapter (ACM)
Thesis Advisor - Master of Science In Applied Analyses and Decision Making, Minerva Schools at KGI (2017)
Advisor / Program Committee - 2016 Workshop on Intelligent Support for Learning in Groups (at AIED 2016)
Reviewer - 2016 International Review of Research in Open and Distance Learning (IRRODL)
Reviewer - 2016 International Conference for the Learning Sciences (ICLS 2016)
Program Committee Member - 2015 Workshop on Intelligent Support for Learning in Groups (at AIED 2015)
Reviewer - 2014 International Journal on Artificial Intelligence in Education (IJAIED)
Reviewer - 2014 International Review of Research in Open and Distance Learning (IRRODL)
Reviewer - 2015 Computer Supported Collaborative Learning conference (CSCL 2015)
Program Committee Member - 2014 Workshop on Intelligent Support for Learning in Groups (at ITS 2014)
Co-Chair - Industry & Innovation track, 2013 Artificial Intelligence in Education conference (AIED 2013)
Reviewer - 2013 International Journal in Computer Supported Collaborative Learning (IJCSCL)
Program Committee Member - 2013 workshop on Intelligent Support for Learning in Groups (at AIED 2013)
Program Committee Member - 2012 Workshop on Intelligent Support for Learning in Groups (at ITS 2012)
Reviewer - 2011 IEEE Transactions on Learning Technologies Journal (IEEE-TLT)
Program Committee Member - 2011 Artificial Intelligence in Education conference (AIED 2011)
Reviewer - 2011 Computer Supported Collaborative Learning conference (CSCL 2011)
Workshop Organizer - Workshop on Opportunities for intelligent and adaptive behavior in collaborative learning systems (at ITS 2010)
Reviewer - 2010 International Conference for the Learning Sciences (ICLS 2010)