Digital Assessment in IMS Global Ecosystems

IMS Global TECH TALK

Contributed by Dr. Colin Smythe, IMS Chief Architect

Digital Assessment in IMS Ecosystems: The Set of IMS Specifications Making Online Assessment a Reality

In June 2000, IM published version 1.0 of the Question and Test Interoperability^® specification—or QTI^® as we all know it. This specification described a structure for exchanging tests and questions to enable the authoring and distribution of online assessments. The following 36 months saw several new versions resulting in the publication of version 1.2.1 in March 2003. This version was later included in the IMS Common Cartridge^® 1.0 specification. Our work from 2004 to 2015 resulted in QTI versions 2.0, 2.1, and 2.2. Between QTI 2.1 and 2.2, IMS released the Accessible Portable Item Protocol^® (APIP^®) 1.0, which added new accessibility features to the core QTI specification. In September 2015, the IMS Technical Advisory Board (TAB) authorized the Charter for QTI version 3.0 and, in July 2020, made available the Candidate Final Public documents for public review.

The longevity of the QTI specification is based on the early research undertaken by long-time IMS Contributing Member ETS and published as the Four Process Architecture. The four-process architecture gives rise to two of the key QTI structures: Assessment and Item. Notice that we use the term ‘Item’ instead of ‘Question.’ This is because an Item includes more than just the actual question: for example, the response processing rules for determining the correctness of the learner’s response (their answer); alternative content to reflect accessibility needs; and much more. An assessment, test, or quiz is the combination and organization of a set of Items. QTI also defines Section. A Section is a collection of Items and Sections. Sections enable the creation of flexible and adaptive assessments to reflect the assessment pedagogy being used and ease the management of the assessment structure. An Item is based upon one or more Interactions. QTI has nineteen predefined interactions to support the most typical question types, e.g., multiple-choice, hotspot, etc.

IMS was an early adopter of the use of XML for data exchange. Document Type Definitions were used for the 1.x versions of QTI to validate the QTI XML instances, but XML Schema Definitions (XSDs) are used for all later versions. Certification of compliant products is an essential aspect of the IMS specification support process. For XML-based specifications, IMS uses a proprietary online XML validator for certification. Unlimited access to this validator is available to all members, and many make extensive use of this benefit as part of their content development validation process. For QTI versions 2.1, 2.2, and 3.0, the specification is composed of several individual specifications:

• QTI Assessment, Section & Item (ASI) – the core definition of the QTI structures and their representation in XML

• QTI Metadata – the QTI-specific metadata to be included in the accompanying content packaging manifest file

• QTI Results Reporting – the information used to represent the exchange of the results awarded to a learner as part of the response and outcomes processing as defined in the core ASI

• QTI Usage Data & Item Statistics – the information used to describe the usage of the QTI Items in terms of a variety of standardized statistics

Several more IMS and non-IMS specifications are used to provide the complete QTI interoperability capability:

• IMS Portable Custom Interaction (PCI) – the definition of a JavaScript API used to connect proprietary Items, also commonly known as Technology Enhanced Items (TEIs), with a QTI-compliant assessment

  • This enables the set of predefined QTI interaction sets to be extended

• IMS Content Packaging 1.2 – a collection of all the XML instance and related asset files (images, video, etc.) combined with the controlling manifest file to form the zip file for the actual interoperability exchange

• IMS Curriculum Standards Metadata (CSM) 1.1 – the metadata used to associate learning standards information to each of the QTI XML resource and asset files

• IMS Access for All Personal Needs & Preferences (AfA PNP) 3.0 – the metadata that is used to configure the assessment system environment to reflect the accessibility needs of the learner

  • This information is also used to tailor the actual assessment/items being delivered

• IMS Access for All Digital Resource Definition (AfA DRD)– the metadata used to annotate the resources available to enable the presentation of alternative content

• IEEE Learning Object Metadata (LOM) – used to define the manifest-level, resource-level, and generic metadata in the Content Packaging manifest file

• Speech Synthesis Markup Language (SSML) and Pronunciation Lexicon Specification (PLS) from W3C – combined to specify how input text is converted into synthesized speech

• MathML – to enable the inclusion of mathematical representation in the content

• HTML 5.0 – used to provide the actual content contained in the QTI XML instances

• W3C Accessibility Standards – support of Accessible Rich Applications (WAI-ARIA) is provided

These specifications are combined at the level of the XML binding to create a single zip file as required in the QTI Profile of IMS Content Packaging.

Actionable e-Assessment is an important aspect of IMS Ecosystems. While QTI is the keystone specification, many IMS specifications can be combined to support an extensive range of assessment processes and workflows.

This broader set of specifications is:

• IMS Common Cartridge 1.x – QTI 1.2.1 is used to insert native tests/quizzes into the content available through all versions of Common Cartridge

  • Common Cartridge 1.3 includes support for APIP zip files

• IMS Caliper Analytics 1.2 – use of the Assessment Profile enables assessment systems/tools/apps to report on the learner’s activities

• IMS Competency & Academic Standards Exchange (CASE) 1.0 – a QTI package supports the use of curriculum standards identifiers, and if these are CASE compliant, then a CASE service can be used to obtain the actual learning standard definitions

• IMS Proctoring Service 1.0 ­– this provides support for proctoring services through the definition of a new extension to Learning Tools Interoperability (LTI) Advantage

• IMS Assessment Results Profile for Gradebook Service 1.0 – a new service that combines QTI Results Reporting with the OneRoster 1.2 Gradebook Service to provide detailed results reports on assessment activities in a OneRoster-based service framework

• IMS Computer Adaptive Testing (CAT) – a work-in-progress REST-based service to enable the adaptive selection and sequencing of QTI Items through an external CAT engine

As can be seen, IMS has a thriving specification development activity in e-Assessment. We will continue to improve how our specifications can be combined to support your assessment processes and workflows.