Abstract

This document addresses some of the challenges presented by Adaptive Instructional Systems (AIS). Adhering to standard best practices for AIS benefits many audiences within the teaching and learning sphere, including learners whose experience with technology can be made more seamless if standards are followed. This document addresses those best practices within the scope of recommendations for academic institutions and suppliers serving that audience and recommendations for the 1EdTech Consortium community. In this context, an AIS is a system that can create a personalized learning pathway to enable student success when learning a subject.

1EdTech formed an Adaptive Task Force in 2018 to review the current state of the adaptive learning market and determine how 1EdTech and its members should best address those needs. After a short pause in 2019, the group reassembled in June 2020 with a new focus: To pursue the creation of a Proof of Concept and accompanying deliverables to address particular problems directly. These solutions address multiple challenges for both users and suppliers of adaptive learning technologies. Read on to discover the many problems potentially solved by this solution.

The information in this document was authored by members of the 1EdTech Adaptive Task Force (2018-2021).

This document is intended for 1EdTech members. Anticipated complementary work includes revisions of this document and a charter to propose this work as a specification to the 1EdTech Technical Advisory Board if interest arises from suppliers.

 

Adaptive Instructional System Best Practices

 

Introduction

An Adaptive Instructional System (AIS) is an emerging teaching technology with great potential for increasing the effectiveness and efficiency of learning. AIS is designed to deliver the right lesson to the right student at the right time to help that person achieve mastery of the learning objective. To accomplish this, AIS applies sophisticated computer algorithms and learner analytics to generate a learner-specific instructional sequence to guide learning. The AIS models effective instructional strategies such as scaffolding to show the learner's mastery of a specific learning objective. Instructors use AIS to help students develop introductory conceptual knowledge, enabling greater instructor support for deeper, conditional learning and assessment.

 

Objectives

Problem Statement

Adaptive Instructional System standards are needed to provide a scalable and sustainable instructional infrastructure that can be configured by faculty and personalized for students. The current lack of standards makes it difficult for faculty to efficiently locate and integrate lessons from disparate sources into a learning sequence delivered through an adaptive system. Without a common lesson framework capable of linking the learning objectives, instructional resources, and assessment activities, it is difficult to manage and measure the student learning outcomes from the systems. Finally, the growing demand for personalized instruction necessitates a standard to ensure that each student's lessons can be delivered as efficiently and effectively as possible.

Solution Statement

Adaptable Technology Operating Model standard (ATOM)

ATOM defines a "Lesson" as containing one learning objective (LO), at least one instructional resource (IR), at least one assessment activity (AA), and a learning objective taxonomic reference (i.e., Bloom's Taxonomy). The ATOM facilitates the measurement of learning objectives to deliver information to students and faculty.

Potential 1EdTech standards implications include:

• Learning Objective (integrated with CASE)
• Instructional Resources (integrated with Thin Common Cartridge)
• Assessment Activities (integrated with QTI)
• Event tracking (via Caliper Analytics)
• Plus potential additional information tracking

Potentially related specifications:

• ATOM could be searchable with LTI Resource Search

Instructional Object Number (ION)

• Unique alphanumeric code for each object
• LTI compliant at the Object level
• Caliper Analytics
• Open tagging

EdMolecule

• A lesson developed in the Adaptable Technology Operating Model (ATOM) framework and including unique Instructional Object Number (ION) assignments for all instructional resources comprising the EdMolecule

Figure 1. An example of an adaptive instructional systems model.

 

Goals of the Adaptive Task Force

• Establish best practices to effectively manage instructional resources and data throughout 1EdTech specifications in the ecosystem so that systems can monitor and measure learning outcomes.
• Identify opportunities for improving alignment between 1EdTech specifications on the use of learning outcomes data for users and context to make it easier for institutions, districts, and suppliers to leverage the full power of instructional systems.
• Deliver multiple solution modalities to the market which address its needs:
  • A glossary to establish a foundational vocabulary for a technology solution, providers, and consumers
  • A Proof of Concept to illustrate the solutions to challenges in a technical model

 

Goals of This Guide

• Provide guidance on how to design and develop adaptive instructional systems and integrate them with third parties
• Share a glossary to establish a foundational vocabulary for a technology solution, its providers, and its consumers
• Introduce 1EdTech members to the Proof-of-Concept ideas and mechanism itself through a recorded demonstration
• Document recommended changes to 1EdTech specifications and implementation and certification practices to improve support for content and data sharing across adaptive instructional systems

 

Out of Scope

For this guide, the task force considers the following out of scope:

• An official 1EdTech charter—i.e., work catalyst—for the "specification ecosystem stack" demonstrated in the Proof of Concept
• Public availability of the Proof of Concept

 

User Stories

The task force used the following summary user stories to guide their work:

As an Instructor, I want to…

...modify the lesson sequence and content quickly and easily whenever I want while preserving the integrity of the educational data for doing learning analytics.

...author, customize, and select content within an adaptive course with the content in the order of their choosing and have the flexibility to make easy changes for their teaching or others.

...analyze the learning efficacy of each instructional resource based on the student performance on the linked assessment activities

...review the learning outcomes data to help me decide which students need help with which specific learning objectives

As an Instructional Systems Technologist, I want to…

...create an integration with my LMS and an Adaptive Learning System to facilitate a seamless entry point for faculty and students, establish the appropriate LTI Advantage ATOM component integrations between the systems, and—based on a curriculum/learning outcomes map—connect the desired learning outcomes and other data streams to an analytics dashboard (comprising LMS, ALS, or other learning tools), so that the dashboard represents the desired learner progress to outcomes, by activities, with usage stats, as identified by the faculty and their instructional designer.

As an Instructional Designer, I want to…

...assist faculty in preparing a curriculum/learning outcomes map that pulls together instructional resources and assessment results and clickstream activity within my LMS, an Adaptive Learning System, and other learning tools selected by faculty. My map will be organized by Learning Outcomes. It will reveal the pedagogical associations (including, but not limited to prerequisite or corequisite, sequence, mandatory or optional) among the resources and assessments in different systems, the type of resource (background information, instruction, practice, assessment, etc.), and how the resource or assessment is applied to the learning sequence (initial instruction, tutorial, maintenance check, and assessment value/weight), will define what "progress" to the outcome means—or what "mastery" means (as applicable), and will identify the data points needed related to outcome, resource, assessment, access, time, and progress/mastery for dashboard presentation and analysis. Works with the Instructional Technologist to implement between systems.

...understand where—and with what—adaptations are happening the most and use this understanding to shape the (re)design of materials and courses.

As a Program Administrator, I want to …

...evaluate the variety of individual adaptive pieces to guide design and development decisions around sequencing concepts and courses to provide students with the best-tailored (adaptive) learning experience.

As a Technical System Administrator, I want to…

...be able to draw together data from various adaptive learning systems adopted by our institution to create a holistic picture of students' performance and the extent to which a program or sequence of courses support student outcomes in concert. We have a mix of these systems under adoption given the different proprietary systems at some publishers and several adoptions of different off-the-shelf adaptive learning systems by individual departments or faculty members. These systems produce student learning and engagement data in a different structure and with different values. Transforming the data to enable a basis of comparison across systems is difficult.

...be able to adopt several adaptive learning systems and integrate them within our learning management system in a standard way. The integration should provide bi-directional passage of at least basic information: enrollments and roles from the LMS and performance and engagement data from the adaptive learning system. Further, each adaptive learning system should be interoperable with all major LMSs since different schools within institutions may have adopted their LMSs. The behavior of the adaptive learning management system should be comparable in each LMS to make it less burdensome on authoring end-users who I support in the implementation of the adaptive learning system.

As a Data Scientist/Learning Analyst, I want to…

...collect learning data from adaptive courseware and platforms to understand better and visualize learner-specific mastery and product usage data, and present this information to students, instructors, and advisors in meaningful ways to help inform student recruitment and retention plans; program, curriculum, and course design; grouping students and differentiation of instruction; and student intervention and enrichment measures.

...identify the atomic bits of content/questions being used—and where—and then use that and other data to determine which pieces are the most effective at which locations and for which learners.

As an Adaptive Learning Systems Procurement Professional, I want to…

...be able to identify, review, assess, and select adaptive learning systems using an interoperability rubric and certification program to ensure its compatibility with other platforms comprising my university's adaptive learning ecosystem

 

The Proof of Concept

The Proof of Concept for this project is a simple previewer for Common Cartridges that emphasizes the use of CASE-aligned content and other relevant metadata. We sought to help content evaluators and designers to answer the question, "Will this content work for our adaptive systems?"

The Proof of Concept allows the user to easily see what content is aligned to their CASE framework and provide easy access to any of those standards hosted on CASE Network. Other metadata such as version numbers, target age ranges, and LTI vendor information allow an evaluator to understand what the content covers and what integrations are involved.

 

Demo Video

 

Recommendations

Academic Institutions and Suppliers

•  RFPs should include
  • Language requiring content aligned to standards.
  • Proof of interoperability (i.e., the 1EdTech Product Directory).
• Learning Objects should be created with standards in mind and then aligned to standards when complete and available.
• The Teaching and Learning community should share recommendations, best practices, and findings so that gaps are identified, and opportunities for improvement are shared.

1EdTech Consortium (1EdTech)

1EdTech should track interest from suppliers and institutions regarding the ideas presented in this paper.

1. If suppliers express interest in the Proof of Concept on adaptive learning, the Adaptive Learning Task Force could form again and write a charter to propose to the Technical Advisory Board.
2. If suppliers and institutions express interest in the Proof of Concept as it pertains to the 1EdTech ecosystem, 1EdTech could pursue those interests. Deliverables may be best practices regarding specification interplay, recommended "stacks" of specifications expressed via one-pagers posted on the 1EdTech website, and other communications.

→Specification-Level Recommendations

Key 1EdTech specifications and adaptive content best practices are outlined below. Details are not comprehensive to the 1EdTech catalog nor data management best practices. See imsglobal.org for the most current updates.

 

Common Cartridge^®

Common Cartridge is a set of open standards developed by the 1EdTech member community that enable interoperability between content and systems. Common Cartridge solves two problems. The first is to provide a standard way to represent digital course materials for use in online learning systems so that such content can be developed in one format and used across a wide variety of learning systems (e.g., course management systems, learning management systems, virtual learning environments, or instructional management systems). The second is to enable new publishing models for online course materials and digital books that are modular, web-distributed, interactive, and customizable.

Common Cartridge in the Proof of Concept

Common Cartridge was the focus of the Proof-of-Concept work. As the mechanism for moving content between systems, the adaptive courseware community must be able to exchange the content itself and all the relevant metadata about that content. This Proof of Concept showed that the necessary information could be transmitted via Common Cartridge and quickly evaluated.

Proposed Best Practices

For vendors looking to start adding more relevant information into their Common Cartridges for adaptive systems, here are a few items to consider:

• Include a lot of metadata! While not all possibilities are outlined in this document, the need for more information was a common discussion point during this work.
• Aligning Common Cartridge resources to CASE is the most critical piece of the adaptive content exchange puzzle.
• Consider how content should be chunked into cartridges. Sometimes it makes sense to provide content for a whole course, but smaller individual lessons or topic cartridges also offer great value to practitioners.

 

Competencies and Academic Standards Exchange^® (CASE^®)

The 1EdTech CASE standard facilitates the exchange of information about learning and education competencies. CASE also transmits information about rubrics, criteria for performance tasks, which may or may not align to competencies. Implementing CASE makes it possible to electronically exchange competency definitions so that applications, systems, and tools can readily access and manage this data. Having universal identifiers for education competencies makes it possible for any tool or application to share information between systems easily. This includes learning management systems, assessment tools, curriculum management apps, certificate and competency-based evaluation systems, and any other tool, process, or content that aligns to or references a competency or framework. This framework makes it possible to define relationships within a competency framework or between two different frameworks.

CASE in the Proof of Concept

A primary objective of the Proof-of-Concept work is to show how a cartridge and its resources align to CASE academic standards. Each lesson shows the standards alignments and links to the source definition.

For example, in this screenshot, you can see a lesson with a resource, practice activity, and assessment activity all aligned to a specific CASE standard that is hosted on CASE Network.

CASE Identifier Best Practices

→Institutions, Districts, and Schools

Institutions, Districts, and Schools that desire to track adaptive content and provide consistent analytics should publish their academic standard frameworks in CASE format. This allows local and supplier-provided content to be consistently aligned with authoritative standards.

→Suppliers

Suppliers that provide educational software platforms must work closely with their educational customers and partners to ensure fidelity and consistency in using CASE identifiers passed between their systems.

Suppliers should work with their institutional customers to coordinate the use of identifiers and ensure a one-to-one relationship between institutional identifiers and any supplier identifiers in instances where the supplier may be using an academic standard framework that differs from that of the institution.

 

Caliper Analytics^®

Caliper Analytics enables institutions to collect learning data from digital resources to understand better and visualize learning activity and product usage data, and present this information to students, instructors, and advisors in meaningful ways to help inform:

• Student recruitment and retention plans
• Program, curriculum, and course design
• Student intervention measures

Caliper Analytics defines a number of metric profiles, each of which models a learning activity or a supporting activity that helps facilitate learning. Each profile provides a domain-specific set of terms and concepts that application designers and developers can draw upon to describe common user interactions consistently using a shared vocabulary. Annotating a reading, playing a video, taking a test, or grading an assignment submission represent a few examples of the many activities or events that Caliper Analytics' metric profiles attempt to describe.

Caliper in the Proof of Concept

The Proof of Concept for this iteration did not directly utilize Caliper Analytics. However, the content and metadata are foundational to building analytics dashboards and reports for adaptive content. The CASE alignments and resource IDs would need to be communicated via Caliper events to allow consistent processing of these resources.

 

Learning Tools Interoperability^® (LTI^®)

The 1EdTech LTI standard prescribes a way to easily and securely connect learning applications and tools with platforms like learning management systems (LMS), portals, and learning object repositories in a secure and standard way without the need for custom programming. Using LTI, if you have a tool such as an interactive assessment application or virtual chemistry lab, you can securely connect to your LMS with a few clicks. LTI consists of a central core that handles the launch and discrete services to add standard features and functions. The LTI core establishes a secure connection and confirms the tool's authenticity. At the same time, the services add features like the exchange of assignment and grade data between an assessment tool and your LMS gradebook.

LTI in the Proof of Concept

The Proof of Concept shows how LTI resources packages in a Common Cartridge can provide CASE alignment and vendor information for evaluation and processing purposes. LTI is broadly used for content distribution, so proper handling of the relevant metadata is essential for adaptive system interoperability.

Proposed Best Practices

For vendors looking to start adding more relevant LTI link information into their Common Cartridges for adaptive systems, here are a few items to consider:

• Include thorough vendor info for evaluators. LTI–Common Cartridge Documentation.
• LTI content requires integration and authentication to access, and when someone is evaluating a cartridge, they often won't have easy access to preview the content. For this purpose, it is essential to include a quality description for each LTI resource.

 

Question Test Interoperability^® (QTI^®)

The 1EdTech QTI standard enables the exchange of item and test content and results data among authoring tools, item banks, test construction tools, learning platforms, assessment delivery systems, and scoring/analytics engines.

QTI in the Proof of Concept

The Proof of Concept for this iteration did not directly utilize QTI. However, all the CASE alignment and additional metadata recommendations throughout this document apply to all QTI resources. A QTI assessment can have CASE alignments inside the cartridge's metadata and within the QTI items themselves. All assessments and their items should be aligned to CASE academic standards to be used within adaptive systems.

 

Future Plans

The Adaptive Learning Task Force expects that sharing thought leadership and recommendations may:

• Result in new specification work
• Inform new initiatives at 1EdTech
• Catalyze Adaptive Task Force members to get involved at 1EdTech in other domain areas

 

Task Force Members

 

 

Appendix

Resources

• CASE
• Caliper Analytics
• Common Cartridge
• LTI Advantage
• QTI

 

References

[CASE-10]

Competencies and Academic Standards Exchange (CASE) Service Version 1.0. 1EdTech Consortium. July 2017. 1EdTech Final Release. URL: https://www.imsglobal.org/activity/case/

[CALIPER-11]

Caliper® Analytics Specification 1.1. Whyte, Anthony; Haag, Viktor; Feng, Linda; Gylling, Markus; Ashbourne, Matt; LaMarche, Wes; Pelaprat, Etienne. 1EdTech Consortium. URL: https://www.imsglobal.org/sites/default/files/caliper/v1p1/caliper-spec-v1p1/caliper-spec-v1p1.html

[CC-1.4]

Common Cartridge 1.4. Anderson, Thor; Smythe, Colin. 1EdTech Consortium. URL: https://www.imsglobal.org/spec/v1p4/

[LTI-13]

Learning Tools Interoperability (LTI): Core Specification v1.3. C. Vervoort; N. Mills. 1EdTech Consortium. March 2018. URL: https://www.imsglobal.org/spec/lti/v1p3/

[QTI-22]

Question and Test Interoperability v2.2 Implementation Guide. Jérôme Bogaerts (OAT), Thomas Hoffmann (ETS), Rob Howard (NWEA), Wilbert Kraan (JISC/CETIS), Mark McKell (1EdTech), Colin Smythe (1EdTech), 1EdTech Consortium. September 2015. URL: https://www.imsglobal.org/question/qtiv2p2/imsqti_v2p2_impl.html

Question & Test Interoperability Results Reporting v2.2. Colin Smythe, 1EdTech (UK), Mark McKell, 1EdTech (USA), Wilbert Kraan, JISC (UK). 1EdTech Consortium. August 2016. URL: https://www.imsglobal.org/question/qtiv2p2p1/QTIv2p2-Results-InfoBindModelv1p0p1/imsqtiv2p2_result_v1p0_InfoBindv1p0p1.html

 

Terms/Definitions

Assessment. Any activities which can be used to demonstrate a learners' mastery of the lesson.

ATOM. Adaptable Technology Operating Model. The objective of the ATOM is to enable the delivery of the right lesson to the right student at the right time

Audience. Group of people served by the Proof of Concept.

Data Point. A measurable, uniquely identifiable piece of information that has the potential to help the machine learn/recommend/produce meaningful data visualization(s). Examples of a data point can be clickstream data: times visited, sequence/frequency/speed/direction (backward/forward) of clicks, time spent, times attempted, accuracy of attempts, etc.

Instructional Resource. Content delivered for the students for learning.

ION. Instructional Object Number or sometimes referred to as Learning Object Identifier in the industry.

Learning Objective. A detailed description of what students will be able to do when they complete a unit of instruction.

Learning Outcome. Learning outputs generated from the assessments data that measure the knowledge or skills students have acquired by the end of a particular assignment, class, course, or program.

Lesson. An amount of instruction given at one time for a student to master. A lesson combines one learning objective, multiple instructional resources, and multiple assessment activities

Module. A module is a container for content that contains several lessons.

Page. A small piece of instructional materials within a module, a lesson, or a unit.

Tagging. The action of attaching a piece of instructional materials, e.g., an assessment item or choice option, to a learning objective or outcome.

Tool. Software and hardware used in an educational environment for teaching and learning. Examples can be Learning Management Systems, mobile devices, etc.

Unit. A module is a container for content that contains a number of modules.

Variable. A factor is liable to adapt or change. Variables can be used in instructional materials to adapt content and assessments for students.