IMS Global QTI v3 Conformance and Certification

IMS Global QTI v3 Conformance and Certification

IMS Candidate Final Public
Version 3.0
IMS Candidate Final Public
Date Issued: 20 July 2020
Status: This document is for review and adoption by the IMS membership.
This version: https://www.imsglobal.org/spec/qti/v3p0/conf/
Latest version: https://www.imsglobal.org/spec/qti/latest/conf/
Errata: https://www.imsglobal.org/spec/qti/v3p0/errata/

IPR and Distribution Notice

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS's procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

The following participating organizations have made explicit license commitments to this specification:

Org name Date election made Necessary claims Type
HMH September 16, 2019 No RF RAND (Required & Optional Elements)

Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/speclicense.html.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

Public contributions, comments and questions can be posted here: http://www.imsglobal.org/forums/ims-glc-public-forums-and-resources.

© 2020 IMS Global Learning Consortium, Inc. All Rights Reserved.

Trademark information: http://www.imsglobal.org/copyright.html

Abstract

IMS Question and Test Interoperability® QTI® is a technical standard for digital interchange of assessment content (items and tests), assessment usage data and results reporting. QTI 3 also describes digital delivery options for a range of common accessibility needs. Use of QTI 3 enables assessment materials to be exchanged digitally among a wide variety of products, such as item/test authoring products, item banks, and test delivery systems.

QTI 3 is based on the well established QTI standard that has been used internationally for two decades and IMS Accessible Portable Item Protocol® (APIP®). Implementations of QTI 3 may require use of related standards, including the IMS Content Packaging 1.0 specification, and the AccessForAll® (AfA) 3.0 specification.  

One of the principal ways that IMS promotes conformance to the QTI 3 standard is by creating a certification process, which is detailed in this documentation. Additionally, and as a part of the certification process, IMS employs online validators for use by member organizations to ensure conformance to the QTI 3 specification.

The objective of the QTI 3 Conformance and Certification process is to increase interoperability between vendors and promote the adoption of the QTI 3 standard at entry (minimum) and more advanced implementation levels.

1. Introduction

1.1 Scope and Context

The purpose of this document is to define the different areas of conformance available for QTI 3 and to describe the procedures for achieving certification for version 3.0. QTI 3 is a standard created and governed by IMS Global Learning Consortium, a non-profit learning standards organization. This document is intended to be helpful to vendors who wish to certify their systems as officially conforming to QTI version 3.0, and for prospective consumers of QTI 3 for use in their assessment programs.

This document includes the specification certification tables within Section 4: Conformance Statements, and detailed descriptions of all the QTI conformance features are available in Appendix A.

1.2 Status of this Document

IMS strongly encourages its members and the community to provide feedback to continue the evolution and improvement of the QTI standard. To join the IMS developer and conformance certification community focused on QTI 3 please visit the QTI Specification online here: https://www.imsglobal.org/question/index.html

Public contributions, comments and questions can be posted here: Public QTI 3 Forums .

2. Conformance and Certification Overview

A QTI 3 certification statement will make a declaration of:

  1. the certification level (or community profile)
  2. the kind of system seeking certification
  3. the capabilities of the system
  4. the specific version for which they have received certification (e.g. QTI version 3.0.1)

Details related to items 1–3 above are provided in Sections 3.1–3.3.

QTI 3 Results Reporting will have a separate certification program. Details for this certification program are available on the IMS member website.

At this time, certification is not available for PNP (candidate registration and profiling) systems. There is an online validator available to IMS members for the validation of specific PNP instances, which validates to the AfA 3.0 QTI 3 Profile. In addition, Delivery/Presentation systems may be required to support PNP data, depending on their certification level, or based on the requirements of any particular assessment program.

Validation of instances of exchangeable QTI 3 materials can verify the materials as conformant to the QTI 3 standard for the available certification levels. However, the validation can only check if the materials are technically valid, not whether the materials themselves are usable by people.

If an organization using the official IMS QTI 3 validator believes the validator to be in error, the organization must report the error to the IMS immediately.

2.1 Certification Levels for QTI 3 Content

Certification for QTI 3 is available for two IMS Levels:

  1. Entry (Level 1)
  2. Core (Level 2)

Entry certification includes all of the required features for the most rudimentary QTI 3 systems or for generated instances of content from those systems. Core certification includes all of features and functionality of the Entry level, and adds additional conformance requirements. Systems that are Core certified are also considered to be Entry certified.

In addition, any certification level can also achieve a QTI Elevated Accessibility certification. This certification includes a bundle of recommended accessibility and accommodation features that help support candidates taking assessments. The Elevated Accessibility certification involves the use of AfA 3.0 PNP data to meet the specific needs of candidates. Systems must achieve one of the IMS certification levels to be eligible to receive an Elevated Accessibility certification. For example, a system that achieved Core certification and Elevated Accessibility certification is said to be QTI 3 Core + Elevated Accessibility Certified (or Level 2A Certified).

Features in the certification context means a named and numbered feature included in the various conformance areas: Tests, Sections, Items, Questions (interactions), Accessibility, Packaging, Metadata, Results. These features might include a bundle of required elements (classes) that enable that feature.

2.2 Certification Systems

Currently, there are 3 kinds of systems that can receive certification for QTI 3, including:

  1. Authoring
  2. Delivery
  3. Scoring

The definitions for the above systems in the context of QTI conformance are described below:

AUTHORING: asserts the system is capable of creating new assessment content, or modifying stored assessment content. While QTI certified AUTHORING systems are not required to store content in QTI, the system must be capable of creating or modifying content that aligns with QTI features, which could include any of the required features of the certification levels or profiles for which they have been certified.

DELIVERY: asserts the system is capable of presenting QTI content to candidates for the feature requirements of the certification level or profile for which they have been certified, and may include the ability to combine candidate PNP data with the QTI content, and within the context of the PNP and available content, deliver content that is specifically designed for the candidate’s needs. Delivery systems may need to meet standards of visual presentation for QTI-specific classes and attributes (based on the certification level or profile requirements), as detailed in the QTI 3 Best Practices and Implementation Guide. Delivery systems must be able to collect and store the responses provided by the candidate, and if the test is designed to allow the candidate to return to questions, accurately present the candidate’s stored response back to the candidate. Delivery systems must be able to meet any test-specific parameters (e.g. time allotted). Delivery systems may need to be able to adjust the test-specific parameters if a candidate’s PNP indicates accommodations for those parameters.

SCORING: consumes QTI Content Packages and QTI Results Reporting data to allow for external manual or automated (Artificial-Intelligence based) scoring. Scoring systems need item definitions metadata, including scoring rubrics.

Additional Systems may be added to the QTI 3 Certification program in the future.

2.3 Certification Capabilities

QTI Certified systems can have the following capabilities:

  1. EXPORT
  2. IMPORT

The definitions for the above capabilities are described below:

EXPORT: asserts the system is capable of exporting QTI content, and the system is capable of meeting the feature requirements of the certification levels or profiles for which they have been certified, including packaging requirements. Exported content must be able to pass the IMS validator without errors for the certification version claimed.

IMPORT: asserts the system is capable of importing QTI content, and the system is capable of storing (without loss of data) the feature requirements of the certification levels or profiles for which they have been certified. Importing systems must be able to differentiate between valid and invalid instances of QTI content for the certification levels or profiles claimed by the system.

Additional capabilities may be added to the QTI 3 Certification program in the future.

2.4 Community Conformance Profiles

The IMS will support a registration system for common, broader profiles for vendors to access, known as a Community Conformance Profile. All Community profiles must be conformant with the IMS QTI 3 Entry Certification set of required features. All the features within a community profile are a subset of the full QTI 3 feature set.

2.5 Certification Publication

After receiving your IMS certification, your organization’s certification for any particular IMS level or Community Profile, will be listed on the IMS web site at http://imscert.org.

2.6 Annual Certification Requirement

Certification is to be done annually for specific QTI 3 versions (final and public releases), beginning with QTI 3 (version 3.0). A vendor may certify in multiple QTI versions, or may wish to only certify in the most recent version, or certify in only one of the specific legacy versions.

Test materials will incorporate any new test materials (errata) into the test suite. Systems seeking certification or re-certification should always pull the most recent version of the testing materials from the IMS certification site as updated materials are added as needed.

Certification submissions will be reviewed and approved by the IMS, including the QTI 3 Certification Working Group, which is comprised of participating IMS Members.

2.7 Interoperability Assurances

In addition to IMS Certifications (IMS Levels or Community Conformance Profiles), organizations can also establish a process that raises confidence of interoperability between the involved parties.

Participating vendors will need to establish which of the QTI 3 features the contract needs (or that the contract requires) to support, then ensure that the trial packages they exchange contain instances of all the listed features. Essentially, organizations create a Contract Assurance that uses existing (or proposed) QTI 3 features. The exchange process should always start with using the official IMS validator for any data/materials used in any of the exchanges.

When a Contract Assurance is across many vendors, or established by a government agency, the participating organizations may wish to involve the IMS by establishing an official Community Conformance Profile. Certification to those published profiles may also be possible or desirable.

Below is a list of examples where a Contract Assurance may be useful to the involved organizations.

  • Two vendors are contracted by a government agency to author and deliver a suite of assessments.
  • A single vendor is contracted by a government agency to deliver an assessment.
  • Two vendors contract together, where one of the vendors is the preferred delivery system for the other vendor.

3. Conformance Features for Certification Levels

This section contains a series of tables that detail the required features for IMS Certification Levels. The tables only include features that are required in at least one of the certification levels. For a complete list of QTI 3 features and their descriptions, see Appendix A: QTI 3 Feature List.

QTI 3 supports many possible features, and it is unlikely (though not impossible) that any particular system supports all the QTI 3 features. In fact, many of the features are not included in any IMS certification levels or Community Conformance Profiles, and are considered “elective.” However, many of these elective features are used for specific programs and projects. These features may be added to certification levels or profiles in the future.

All QTI 3 certified systems must be able to operate without failure when features beyond the minimum required features are included in the content or PNP instance. While certified systems don’t need to support, or even store the “extra” features/data, the systems must be able to ignore or remove the unsupported data.

Exceptions to the above requirement include:

  1. A conformant system does not have to support a PNP bulk record instance.

Exceptions would not apply where the Community Conformance Profile requires the feature, or where the feature is specified in a Contract Assurance.

Delivery systems are expected to be able to identify (on import) and display notifications for content that is not valid. Delivery systems may continue to present that invalid content to the candidate at their own risk.

3.1 Content Import/Export & Delivery Certification Requirements

Below is a list of features required for certification for IMS certification levels. For a full description of each feature, see Appendix A: QTI 3 Feature List.

An empty cell indicates that the feature is not required for certification, but the system may optionally support the feature without affecting the system’s certification, except that the optional feature is published in conformance statements.

R = REQUIRED. A system MUST support this feature to achieve certification.

3.1.1 INTERACTIONS

Feature ID Feature Name

Level 1

Entry

Level 2

Core

Q-2 Choice Interaction R R
Q-5 Extended Text Interaction R R
Q-6 Gap Match Interaction R
Q-8 Graphic Gap Match Interaction R
Q-10 Hot-spot Interaction R
Q-11 Hot-text Interaction R
Q-12 Inline Choice Interaction R
Q-13 Match Interaction R
Q-15 Order Interaction  R*
Q-20 Text Entry Interaction R R

*Requirement excludes support for the min/max attribute

Interaction Conformance Notes:

  1. Entry level systems need only support the required elements and attributes for the interactions. Unless otherwise noted, Core systems need to support the required and optional elements and attributes for interactions.
  2. Template and Feedback features within Interactions are only required if implementing these elective features.
  3. The shuffle attribute is a discretionary trait of all the interactions, and is not required at any IMS certification level.
  4. Interaction in the Entry Level must also support a related subset of QTI shared vocabulary terms (see I-19a).

3.1.2 ITEM LEVEL FEATURES

Feature ID Feature Name

Level 1

Entry

Level 2

Core

I-0 Assessment Item Root R R
I-1 Response Declaration R R
I-2 Outcome Declaration R R
I-4 Shared Stimulus R
I-7 Item Body (some exclusions) R R
I-8 HTML5 (QTI subset) R R
I-9a Response Processing – Full R
I-9b Response Processing – Template R R
I-11 Rubric Block R
I-14 Catalog Resources R
I-17 Composite Items R
I-18 MathML R
I-19a QTI Shared Vocabulary (subset) R R
I-19b QTI Shared Vocabulary (full) R

3.1.3 ACCESSIBILITY*

Feature ID Feature Name

Level 1

Entry

Level 2

Core

A-1 Alt text for graphics R R
A-13b Captions (track support) R
A-15 Glossary (on-screen) R
A-30a Magnification (all-content) – Delivery R
A-42a Text-appearance, text and background color– Delivery R

*See also the Elevated Accessibility Conformance table.

3.1.4 TEST LEVEL FEATURES

Feature ID Feature Name

Level 1

Entry

Level 2

Core

T-1 Outcome Declaration R
T-2 Time Limits R
T-4 Test Part R R
T-5 Item Session Control in TestPart R
T-7 Sections in Test Parts R R
T-9 Outcomes Processing for Tests R
T-12 Multiple Sections in TestPart R
T-13 Rubric Blocks for AssessmentTest R
T-14 Record and Restore Responses (Delivery) R R

3.1.5 SECTION LEVEL FEATURES

Feature ID Feature Name

Level 1

Entry

Level 2

Core

S-0 Section Root (AssessmentSection.Type) R R
S-2 Item Session Control in Sections R
S-3 Selection R
S-4 Ordering R
S-5 Rubric Block in Sections R
S-7 Items R R
S-9 Assessment Section Referencing R

3.1.6 CONTENT PACKAGING FEATURES

ID Feature Name Level 1Entry Level 2Core
P-1 Test Instances R R
P-4 Item Instances R R
P-7 Resource Metadata (QTI) R
P-11 Stimulus Instances R

3.2 Certification for Elective Features

In addition to systems achieving certification for the IMS pre-defined levels, community profiles, and Elevated Accessibility, systems may also make conformance statements (added to their published certification conformance statements) about individual features that are outside of the defined list of required features for particular certification levels or profiles. Any feature (as listed in Appendix A: QTI Feature List) that is outside of the defined list of required features for certification levels or profiles is considered “elective” – meaning it is optional for a system to implement the feature.

For systems seeking to make conformance statements for elective features for the EXPORT capability, systems must export at least two examples of content that makes use of the feature (and it’s required classes and attributes) and verify that the content passes the official online IMS validator for the version for which the system is seeking certification. The examples should be provided to the IMS, and may be reviewed by the QTI 3 Specification & Development group before conformance is granted to the submitting system.

For systems seeking to make conformance statements for the IMPORT capability, systems must be able to import the feature (and present the feature if it is a Delivery system) and retain all the information provided with the content. The certification test content will be provided on the IMS QTI 3 Conformance and Certification GitHub repository in the Elective Features directory (though certain features may be found in higher level certification directories).

The additional feature(s) should be listed in the documentation submitted to the IMS for certification. As is true of certification for levels and profiles, conformance statements for specific elective features will need to be renewed annually (usually done in conjunction with certification at a particular level or profile).

3.3 Elevated Accessibility Conformance Requirements

Below is a table which lists the accessibility and accommodation supports required to achieve an additional Elevated Accessibility certification for your IMS Certification. All systems seeking Elevated Accessibility certification will need to support candidate PNP data for at least the features listed in the Elevated Accessibility feature list. All the named requirements are required for Elevated Accessibility certification. All other accessibility and accommodations are considered elective, unless specifically required by a Community Profile or Contract Assurance.

Feature ID  Feature Name
I-14 Catalog Resources
A-2a WAI-ARIA Subset – IMPORT and Delivery
A-6 additional-testing-time – Delivery
A-8 answer-masking – Delivery
A-9 audio-description
A-10a braille
A-11 breaks – Delivery
A-13b Captions (track support)
A-29 long-description
A-30a Magnification (all-content) – Delivery
A-36 sign-language
A-40a spoken (reading-type=screen-reader) – Delivery
A-40b spoken (reading-type=computer-read-aloud) – Delivery
A-41 tactile
A-42a text-appearance, text and background color – Delivery
A-44 transcript

3.4 PNP Conformance Requirements

Below is a table which lists the supports expected to be supported in the systems achieving Elevated Accessibility certification that rely on PNP data for their assessment programs. While there is no current certification process for the QTI 3 PNP profile of AfA 3.0, the IMS validator will check for valid instances of QTI 3 PNPs. In the future, there may be a process that allows systems to certify their PNP conformance. The table below is for reference purposes only.

Feature ID Feature Name
A-1 Alt text for graphics
A-8 answer-masking
A-9 audio-description
A-10a braille
A-11 breaks
A-29 long-description
A-30a Magnification (all-content)
A-36 sign-language
A-40a Spoken (reading-type)
A-40c Spoken (restriction-type)
A-41 tactile
A-42a Text-appearance, text and background color
A-44 transcript

4. Achieving Certification

Validation for QTI 3 conformance consists of the following strategies:

  • Self-Testing. Systems seeking certification will test themselves, and report the results to the IMS.
  • Automated Validation. An electronic process to verify conformance using automated tools provided by the IMS QTI Evolution & Certification Committee.
  • Documented or Demonstrable Validation. Documented statements and/or demonstrable capabilities showing conformance that can be reviewed by the IMS QTI Specification Development Task Force or by the user.
  • Test Case Based Validation. Using a standard set of QTI 3 content packages and PNP files with known characteristics provided by the IMS QTI Specification Development Task Force that must be used as part of the validation process and have known expected results that can be documented and/or demonstrated.

Because the QTI 3 standard is primarily a data transfer format, conformance and certification does not govern how applications manage or modify data within the system itself. For example, delivery systems must be able to combine the information provided by imported content and PNP files, but need not natively use the QTI 3 format during delivery. There are limits to what can be verified in an automated fashion. Export functions are much more easily verified using automation than import functions. The output from an export process must adhere to the QTI 3 XML specifications and can be verified using automation. Import is much more difficult to verify because the import process results in a proprietary system consuming the information and representing it in an internal format that does not need to adhere to the XML specification and cannot be verified using automation.  Each of these approaches is discussed in detail below.

4.1 Certification through Self-testing

A certifying system would self-test by filling out the relevant checklist and submit that to the IMS. Systems certifying exported packages need to also submit proof of passing validation. The validator will report the content level achieved, and list all the features correctly used in the content, as well as the published conformance statements. The validator will report any errors in the content as well. Instances containing errors are not considered valid instances for purposes of certification or exchange.

4.2 Automated Validation

Capabilities to be validated using automated tools include:

  • QTI 3 Content Package export
  • QTI 3 PNP file export

For export, the QTI 3 conformance approach focuses on determining if the system can generate an object (QTI 3 item or AfA PNP instance) that is non-problematic. IMS provides an online tool available for individuals or organization to submit an object (item or PNP) and then receive information about the existence (and sometimes nature) of problems with syntax and semantics of the object.

It is expected that the automated validation tool will:

  1. Accept Entry, Core, Advanced, and all elective QTI 3 conformance content elements identified in the conformance matrix.
  2. Accept custom (and profiled) extensions that are agreed to between exchanging parties (interdependent vendors), as allowed for extension by the QTI 3 XSD.
  3. Identify which QTI 3 features are implemented in the supplied instance.

4.3 Documented or Demonstrable Validation

Capabilities to be documented and/or demonstrated by the solution provider include:

  • A statement from the IMS validator for exported QTI 3 content,
  • Render QTI 3 content in accordance with the QTI 3 Best Practices & Implementation Guide, supporting differences in presentation as required by PNP file information,
  • A computer generated report from an import system that verifies the content was imported properly, noting any content, or portions of content that were not fully imported.

As each test delivery system may provide a unique user experience and use different technologies to deliver to content, providing automated validation of each implementation is virtually impossible. Therefore, the QTI Certifications Committee will rely on the solution provider to document that the required QTI 3 content and PNP features have been implemented and can document and/or demonstrate them. The documents and demo systems can be submitted to the QTI Certifications Committee for consideration and review but the Certifications Committee will not be able verify the exact system behaviors.

4.4 Test Case Validation

IMS’s Assessment Product Steering Committee members will contribute reference content for interoperability/conformance testing. Test packages will be created by the IMS for the 3 Certification Levels and the full QTI 3 feature set.

The test set includes sample content that includes the complete set of QTI 3 features for the specific version for which a system is seeking certification. All certified QTI 3 systems are required to ignore QTI 3 features outside the required set of features for which that system is seeking, or has received, certification. Systems that do not support specific features can leave the exchangeable materials unaltered, but not implement the unsupported features. Alternatively, systems may elect to remove the unsupported features from the exchangeable materials for use within their systems.

Systems certified at a particular level, or for a specific Community Profile, are not required to return, or pass on, exchangeable content in the same exact form in which it was received, except to the extent that the content must still contain all the data and features for which that system has received certification, and that the materials exported from that system are valid QTI 3 instances.

To conduct import validation, the test sets contain some materials that are problematic and others that are non-problematic. Import functionality focuses on determining if the system can (a) flag as problematic objects that are known to have problems in syntax or semantics and (b) recognize as non-problematic the objects that are known to be without problems.

Capabilities to be validated using a standard set of test cases include:

  • QTI 3 Content Package import
  • AfA 3.0 (QTI 3 Profile) PNP file import
  • AfA 3.0 (QTI 3 Profile) PNP bulk records file import

A. APPENDIX A: QTI 3 Feature List

Below is a list of all identified QTI 3 features. Only features fully supported and provided in these lists will be counted listed in published conformance statements.

A.1 QTI Tests

ID Feature Name Feature Description
T-1 Outcome Declaration Outcome variables are declared by outcome declarations on the test-level. Their value is set either from a default given in the declaration itself or by a rule set by outcomes processing, e.g. to sum all item-scores on a test.
T-2 Assessment Time Limits In the context of a specific AssessmentTest, an item or group of items, may be subject to a time constraint. This specification supports both minimum and maximum time constraints.
T-3 Stylesheet A reference to a stylesheet.
T-4 Test Part A TestPart represents a major division of the test and is used to control the basic mode parameters that apply to all sections and sub-sections within that TestPart.
T-5 Item Session Control (ISC) in Test Part When items are referenced as part of a test, the test may impose constraints on how many attempts, and which states are allowed. These constraints can be specified for individual items, for whole sections or for an entire TestPart. By default, a setting at TestPart level affects all items in that part unless the setting is overridden at the AssessmentSection level or ultimately at the individual AssessmentItemRef. The defaults for an ItemSessionControl are used only in the absence of any applicable constraint.
T-6 ISC Time Limits in Test Part Time Limits as they apply to individual TestPart(s).
T-7 Sections in TestPart An assessment section groups together individual item references and/or sub-sections. A section can be composed of any hierarchy/combination of items and sections. A section can only reference an item using an qti-assessment-item-ref object but it may contain or reference other sections. The grouping of the sections/items depends upon the nature of the parent section i.e. each section can be used for different grouping criteria e.g. organizational, pedagogic, etc.
T-8 Test Feedback in TestPart Feedback provided to the candidate based on conditions set at the TestPart level.
T-9 Outcomes Processing for Test
T-10 Test Feedback (test-level) Feedback provided to the candidate based on conditions set at the test level.
T-11 Multiple Test Parts Support for more than one TestPart in a TestAssessment
T-12 Multiple Sections in TestPart Support for more than one AssessmentSection in a TestPart
T-13 Rubric Block for AssessmentTest Support for TestRubricBlock content in an AssessmentTest, including all the attributes for TestRubricBlock and their vocabularies
T-14 Record and Restore Responses Delivery: A candidate’s responses are correctly saved. When the session is discontinued and reinitiated the candidate’s responses are restored correctly.

A.2 QTI Sections

ID Feature Name Feature Description
S-0 Section Root Support for root class AssessmentSection.Type
S-1 Item Session Control Parameters used to control the allowable states (maximum attempts, show feedback, allow review, show solution, allow comment, allow skipping, validate responses) of each item session (may be overridden at sub-section or item level).
S-2 Time Limits Section Level Optionally controls the amount of time a candidate is allowed for this item or section.
S-3 Selection The selection class specifies the rules used to select the child elements of a section for each test session. If no selection rules are given it must be assumed that all elements are to be selected. The selection class also provides an opportunity for extensions to this specification to include support for more complex selection algorithms.
S-4 Ordering The ordering class specifies the rule used to arrange the child elements of a section following selection. If no ordering rule is given, assume that the elements are to be ordered in the order in which they are defined.
S-5 Rubric Block The container for the rubric block content. A rubric block identifies part of the content that represents instructions to one or more of the actors that view the item.
S-6 Section Include This allows assessment content that would usually be included in the same file as the AssessmentSection to be provided as a separate  (possibly shared by more than one item or test) file which is included in the referencing file using the XML Inclusions standard ( https://www.w3.org/TR/xinclude/  )
S-7 Items Support for references to AssessmentItems.
S-8 Nested Sections Support for sections that are within sections.
S-9 Section References Using a reference to an AssessmentSection via qti-assessment-section-ref

A.3 QTI 3 Items

ID Feature Name Feature Description
I-0 Assessment Item Root Support for the AssessmentItem Element’s attributes.
I-1 Response Declaration Response variables are declared using the qti-response-declaration node and the declarations are bound to interactions in the qti-item-body. The response declaration supplies the values for the scoring of the item. It indicates what the correct response is, or the partially correct response(s).
I-2 Outcome Declaration Outcome variables are declared using the qti-outcome-declarations node. The outcome is the score that the candidate can receive for responding to, or not responding to, the item.
I-3 Template Declaration Use the TemplateDeclaration node to declare item variables that are to be used specifically for the purposes of cloning items. They can have their value set only during TemplateProcessing. They are referred to within the qti-item-body in order to individualize the clone and possibly also within the response processing rules if the cloning process affects the way the item is scored.
I-4 Shared Stimulus reference The AssessmentStimulusRef structure that enables reference to an 'AssessmentStimulus' instance. The stimulus must be contained within its own instance and the item uses the 'qti-assessment-stimulus-ref' structure to provide the link between the item and the stimulus.
I-5 Template Processing Use the qti-template-processing node for template processing. Template processing consists of one or more templateRules that are followed by a cloning engine or delivery system in order to assign values to the template variables. Template processing is identical in form to response processing except that the purpose is to assign values to template variables, not outcome variables.
I-6 stylesheet The stylesheet node is used to associate an external stylesheet for the item. QTI 3 supports CSS 2.1 and CSS 3.0 stylesheets (except CSS3 Speech). Any number of stylesheets can be included in an item.
I-7 Item Body

The ItemBody (element qti-item-body) contains the text, graphics, media objects and interactions that describe the item's content and information about how it is structured.

Interactions have their own certification level requirements. (See the QTI 3 Interactions table.)

Support for qti-rubric-block is required at the Core Level.

Support for InfoControl, Feedback, and Template features are elective, or as required by a Community Profile or contract.

I-8 HTML5 (QTI Subset) Support for all the HTML 5 elements as detailed within the QTI 3 Schema. QTI does not support every HTML element. Rather, it lists specific elements that are permitted for use within QTI.
I-9a Response Processing: Full The process by which a delivery engine assigns outcomes (the score) based on the candidate's responses. Response processing can be achieved using either a response processing template (a reference to a common processing pattern), or by using customized response processing.
I-9b Response Processing: Fixed-Template The process by which a delivery engine assigns outcomes (the score) based on the candidate's responses. Response processing is achieved using a response processing template (a reference to a common processing pattern). You need only to support the templates for the interactions required for the certification level you are seeking.
I-10 Modal Feedback The qti-modal-feedback node is used for modal feedback to be presented to the candidate directly following response processing. The value of an outcome variable is used in conjunction with the show-hide and identifier characteristics to determine whether or not the feedback is presented. The content of the qti-modal-feedback cannot contain any interactions.
I-11 Rubric Block The container for the rubric block content. A rubric block identifies part of the content that represents instructions to one or more of the actors that view the item. Although rubric blocks are defined as SimpleBlocks they must not contain interactions. The visibility of nested qti-rubric-blocks is determined by the outermost element. In other words, if an element is determined to be hidden then all of its content is hidden including conditionally visible elements for which the conditions are satisfied and that therefore would otherwise be visible.
I-12 Item Include This allows assessment content that would usually be included in the same file as the AssessmentItem to be provided as a separate  (possibly shared by more than one item) file which is included in the referencing file using the XML Inclusions standard ( https://www.w3.org/TR/xinclude/  )
I-13 Companion Materials

Item content provides information within the qti-companion-materials-info node that indicates that the item must be presented in conjunction with specific materials, including: calculators, protractors, rules, digital materials, and physical materials. Delivery Systems would provide the listed materials during the assessment session in conjunction with the item content.

Calculator: Content that indicates a particular type of calculator (basic, standard, scientific, or graphing) must be provided by the delivery system.

Protractor: Content that indicates a protractor must be provided by the delivery system, using the parameters provided.

I-14 Catalog Resources Support for content held within a qti-content-info node, including qti-catalog, qti-card, and qti-card-entry containers, and the qti-html-content & qti-file-href nodes.
I-15 Context Declarations The qti-context-declaration node is used to make contextual declarations. When evaluating TemplateProcessing and ResponseProcessing instructions in an AssessmentItem, having contextual information available in the information model can improve the effectiveness and efficiency of TemplateProcessing and ResponseProcessing instructions. Contextual information can include candidate information, test information, and system (or environmental) information.
I-16 Time Limits For an individual item, time limits (as set by the Session definition) are enforced by the delivery system (unless PNP information for specific candidates allows for additional time).
I-17 Composite Items Support for multiple interactions within a single item.
I-18 MathML Support for MathML v2.0 and v3.0
I-19a QTI Shared Vocabulary Subset

A subset of the classes used in the QTI Shared Vocabulary for interactions used in Entry Level Certification.

These classes include:

ChoiceInteraction:

qti-labels-noneqti-labels-decimal

qti-labels-lower-alpha

qti-labels-upper-alpha

qti-labels-suffix-noneqti-labels-suffix-periodqti-labels-suffix-parenthesis

qti-input-control-hidden

TextEntryInteraction:

qti-input-width-1qti-input-width-2qti-input-width-3qti-input-width-4qti-input-width-6qti-input-width-10qti-input-width-15qti-input-width-20qti-input-width-72

ExtendedTextInteraction:

qti-height-lines-3

qti-height-lines-6

qti-height-lines-15

I-19b QTI Shared Vocabulary Full Support for the full set of shared vocabulary, including those interactions which are required for a particular level of certification. Implementers are NOT required to support shared vocabulary for interactions not included in the certification level they are seeking. See Appendix B of the QTI 3 Best Practices Implementation Guide for the complete list of QTI Shared Vocabulary.

A.4 QTI 3 Interactions

ID Feature Name Feature Description & Notes
Q-1 Associate Interaction

An Associate Interaction (qti-associate-interaction) is a block interaction that presents candidates with a number of choices and allows them to create associations between them.

The “shuffle” attribute is not required for any certification level.

Q-2 Choice Interaction

The choice interaction (qti-choice-interaction) presents a set of choices to the candidate. The candidate's task is to select one or more of the choices, up to a maximum of max-choices. The interaction is always initialized with no choices selected. Though the max-choices is an optional attribute, it is required for Entry Level Certification.

The “shuffle” attribute and associated SimpleChoice “fixed” attribute are not required for any certification level.

Q-3 Drawing Interaction The drawing interaction (qti-drawing-interaction) allows the candidate to use a common set of drawing tools to modify a given graphical image (the canvas). It must be bound to a response variable with base-type file and single cardinality. The result is a file in the same format as the original image.
Q-4 End Attempt Interaction The end attempt interaction (qti-end-attempt-interaction) is a special type of interaction which allows item authors to provide the candidate with control over the way in which the candidate terminates an attempt. The candidate can use the interaction to terminate the attempt (triggering response processing) immediately, typically to request a hint.
Q-5 Extended Text Interaction An Extended Text Interaction (extended-text-interaction) is a block interaction that allows the candidate to enter an extended amount of text.
Q-6 Gap Match Interaction A gap match interaction (qti-gap-match-interaction) is a block interaction that contains a number gaps that the candidate can fill from an associated set of choices. The candidate must be able to review the content with the gaps filled in context, as indicated by their choices. The GapMatchInteraction must be bound to a response variable with base-type directedPair and either single or multiple cardinality, depending on the number of gaps. The choices represent the source of the pairing and gaps the targets. Each gap can have at most one choice associated with it. The maximum occurrence of the choices is controlled by the match-max characteristic of GapChoiceInteraction.
Q-7 Graphic Associate Interaction A graphic associate interaction (qti-graphic-associate-interaction) is Te The delivery engine must clearly indicate all defined area(s) of the image.
Q-8 Graphic Gap Match Interaction A graphic gap match interaction (qti-graphic-gap-match-interaction) is a graphical interaction with a set of gaps that are defined as areas (hotspots) of the graphic image and an additional set of gap choices that are defined outside the image. The candidate must associate the gap choices with the gaps in the image and be able to review the image with the gaps filled in context, as indicated by their choices.
Q-9 Graphic Order Interaction A  graphic order interaction (qti-graphic-order-interaction) is a graphic interaction with a corresponding set of choices that are defined as areas of the graphic image. The candidate's task is to impose an ordering on the areas (hotspots).
Q-10 Hot-spot Interaction A hotspot interaction (qti-hotspot-interaction) is a graphical interaction with a corresponding set of choices that are defined as areas of the graphic image. The candidate's task is to select one or more of the areas (hotspots).
Q-11 Hot-text Interaction The hot text interaction (qti-hottext-interaction) presents a set of choices to the candidate represented as selectable runs of text embedded within a surrounding context, such as a simple passage of text. Like qti-choice-interaction, the candidate's task is to select one or more of the choices, up to a maximum of max-choices. Must support the HotTextInteraction.Type class and the HotText.Type class.
Q-12 Inline Choice Interaction An inline choice interaction (qti-inline-choice-interaction) is an inline interaction that presents the user with a set of choices, each of which is a simple piece of text. The candidate's task is to select one of the choices. Unlike the qti-choice-interaction, the delivery engine must allow the candidate to review their choice within the context of the surrounding text.
Q-13 Match Interaction A match interaction (qti-match-interaction) is a block interaction that presents candidates with two sets of choices and allows them to create associates between pairs of choices in the two sets, but not  between pairs of choices in the same set. Further restrictions can still be placed on the allowable associations using the match-max characteristic of the choices. The qti-match-interaction must be bound to a response variable with base-type directedPair and either single or multiple cardinality.
Q-14 Media Interaction The media interaction (qti-media-interaction) allows more control over the way the candidate interacts with a time-based media object and allows the number of times the media object was experienced to be reported in the value of the associated response variable.
Q-15 Order Interaction In an order interaction (qti-order-interaction) the candidate's task is to reorder the choices, the order in which the choices are displayed initially is significant. By default the candidate's task is to order all of the choices but a subset of the choices can be requested using the max-choices and min-choices attributes. When specified the candidate must select a subset of the choices and impose an ordering on them.
Q-16 Portable Custom Interaction (PCI) The portable custom interaction provides an opportunity for extensibility of this specification to include support for interactions not currently documented. The “custom interaction” is considered deprecated and replaced by use of Portable Custom Interaction (PCI) .
Q-17 Position Object Interaction The position object interaction (qti-position-object-interaction) consists of a single image which must be positioned on another graphic image (the stage) by the candidate. Like qti-select-point-interaction, the associated response may have an AreaMapping that scores the response on the basis of comparing  it against predefined areas but the delivery engine must not indicate these areas of the stage. Only the actual position(s) selected by the candidate shall be indicated.
Q-18 Selection Point Interaction The selection point interaction (qti-select-point-interaction) is a graphic interaction. The candidate's task is to select one or more points. The associated response may have an AreaMapping that scores the response on the basis of comparing it against predefined areas but the delivery engine must not indicate these areas of the image. Only the actual point(s) selected by the candidate shall be indicated.
Q-19 Slider Interaction The slider interaction (qti-slider-interaction) presents the candidate with a control for selecting a numerical value between a lower and upper bound. It must be bound to a response variable with single cardinality with a base-type of either integer or float.
Q-20 Text Entry Interaction A text entry interaction (qti-text-entry-interaction) is an InlineInteraction that obtains a simple piece of text from the candidate. Like qti-inline-choice-interaction, the delivery engine must allow the candidate to review their choice within the context of the surrounding text.
Q-21 Upload Interaction The upload interaction (qti-upload-interaction) allows the candidate to upload a pre-prepared file representing their response.
Q-31 Custom Interaction (proprietary) The custom interaction (qti-custom-interaction) provides an opportunity for extensibility of this specification to include support for interactions not currently documented, using methods that are proprietary for the supplier. The use of this interaction is deprecated in favor of qti-portable-custom-interaction
Q-32 Custom Interaction (LTI) The LTI custom interaction provides an opportunity for extensibility of this specification to include support for interactions not currently documented, by use of LTI.

A.5 QTI 3 Accessibility

ID Feature Name Feature Description & Notes
A-1 Alternate text for graphics All graphics that could be presented to candidates that are not for decorative-only purposes have a text alternative provided. Text for <img> is provided in the alt attribute.
A-2a ARIA Subset Support for the following ARIA features for all content that could be delivered to candidates, proctors, scorers, and tutors:aria-describedby, aria-hidden, aria-label, aria-labelledby, aria-live, role
A-2b ARIA Full Support for all the ARIA 1.0 features for all content that could be delivered to candidates, proctors, scorers, and tutors.
A-3a Inline Namespaced SSML Support for namespaced SSML 1.1 markup within all content that could be delivered to candidates.
A-3b Inline SSML using JSON markup Support for SSML 1.1 vocabulary using JSON markup within all content that could be delivered to candidates.
A-4a Internationalization (bi-directional text) Support for bi-directional text and content
A-4b Internationalization (Ruby) Support for Ruby markup
A-5 additional-directions When item content contains additional-directions within a catalog, that content is presented to candidates who have the support listed in their PNP.
A-6 additional-testing-time Candidates who have additional-testing-time in their PNP will be allowed the amount of additional time requested to finish the assessment.
A-7 alternative-text (PNP assignment) If a candidate is specifically assigned to “alternative-text” in their PNP, the expectation during an assessment session is that the candidate will have the option to have the alternative text presented visually. The candidate must have the ability to switch between the visual presentation of the alternative text, and the programmatically available (but visually hidden) alternative text content. By default, alternative text is required to be visually hidden but programmatically available to candidates. Alternative text can be provided in the <img> alt attribute or aria-label attribute (for any element).
A-8 answer-masking By default, choices for items with choice interactions are covered (visually hidden) when the item is first presented to the candidate. The candidate has the ability to remove the mask(s) at a time of their choosing. This feature is generally used as a cognitive support, as a test-taking strategy, to enable candidates to think of the correct response before being presented with the answer choices.
A-9 audio-description Narration added to a soundtrack to describe important visual details that cannot be understood from the main soundtrack alone (from WCAG 2.0 http://www.w3.org/TR/WCAG20/#audiodescdef ). Audio-descriptions can be useful for video-only media, or synchronized media (video & audio) content. Candidates with "audio-description" in their PNP indicate a user's preference that they have access to audio descriptions for media containing video.
A-10a braille Candidates who have braille in their PNP have a preference that content be available to them in braille form. The PNP allows for setting the specific delivery-mode of the braille support, using either the "refreshable" or "embossed" vocabularies. Refreshable is the default value. While refreshable braille displays should use the default content of the assessment, encoded braille files can be associated with all or portions of the content using catalog resources and making the file(s) available to the candidate.
A-10b braille preferences (delivery-mode, grade, braille-type, math-type) For candidates who have supplied these optional braille parameters in their PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, although the candidate is still capable of modifying the settings during the assessment session.
A-11 breaks For candidates who have breaks in their PNP, they can pause an assessment to take a break, and return to the assessment without losing any response data, or incurring scoring penalties.
A-12 calculator-on-screen Candidates who have calculator-on-screen must have access to the calculator type specified in their PNP, even when an assessment would normally prohibit the use of a calculator.
A-13a Captions: PNP Candidates who have captions in their PNP would be presented captions for videos that have captions.
A-13b Captions: track Content supplies a captions track in the video or audio element. Delivery systems ensure the captions track is available to candidates.
A-15 Glossary (on-screen) An on-screen feature which allows a user to see definitions for designated words or phrases. The word or phrase will have some form of highlighting to indicate a glossary term that can be clicked to reveal the definition. When item content contains glossary-on-screen content within a catalog, that content is presented to candidates who have the support listed in their PNP or where the program allows access to glossary terms to all candidates.
A-16 hazard-avoidance If a candidate has a specific hazard-avoidance listed in their PNP, the candidate must not be exposed to content with those characteristics.
A-17 high-contrast When item content contains high-contrast content within a catalog (usually as an alternative to an image), that content is presented to candidates who have the support listed in their PNP.
A-18 Homophone-checker (on-screen) A resource that allows the user to understand the difference between words that share the same pronunciation regardless of how they are spelled. Candidates with homophone-checker in their PNP would access to an on-screen homophone checking tool.
A-19 input-requirements The requirements for the user to fully operate a computer interface using only their prefered input method (i.e., keyboard, mouse, etc.). For candidates who have input-requirements in their PNP, they must be able to access content using the input-requirement specified. It does not mean the user cannot use other methods of input (the interface must not prevent these other methods), but that the interface must have full control of the interface using the method provided in their PNP.
A-20 invert-display-polarity The values (the dark and light aspect) of the default presentation colors are inverted in the user interface. White is black, black is white. The purpose of inverting the values is to display text as light text over a dark background. While some inversion tools also reverse the polarity of the color hue (red to green, blue to yellow, etc.), the color shift is not necessarily needed or desired for the display of text. For candidates who have invert-polarity-display in their PNP, the user must have the ability to invert the polarity of the device display. In practice, the user may also have to capability to switch between the inverted and default display colors of the content.
A-21 item-translation A translated version (a variant) of the assessment content is presented to the user in place of, or in addition to, the primary language of the content. For candidates with item-translation listed in their PNP, assessment content is available to them in the language specified.
A-22 keyboard-directions When content contains keyboard-directions within a catalog, that content is presented to candidates who have the support listed in their PNP.
A-23 keyword-emphasis When content uses the special class “qti-keyword-emphasis”, any text within that class is presented with additional text emphasis to candidates who have the support listed in their PNP.
A-24 keyword-translation Certain specific words would have language translations available to users who need some assistance with difficult or important words in the content. When content contains keyword-translation content within a catalog, that content is presented to candidates who have the support listed in their PNP.
A-25 language-of-interface The language used on the interface elements (navigation buttons, labels, page headings, etc.), as differentiated from the language of content within the interface. For example, in an assessment, the interface could be in English, while the assessment questions and directions are in Spanish. For candidates with language-of-interface listed in their PNP, the assessment user interface is available to them in the language specified.
A-26 layout-single-column The delivery system has the capability of presenting content in a single column of text (when possible) for candidates who have the layout-single-column support in their PNP.
A-27 line-reader A tool that allows the user to move a marker (a colored bar or an underscore) up or down text, line by line, to assist in reading the content. Candidates who have line-reader in their PNP must have access to a line-reader tool in the assessment user interface during the assessment session.
A-28 linguistic-guidance When content contains linguistic-guidance within a catalog, that content is presented to candidates who have the support listed in their PNP.
A-29 long-description When content contains long description(s) for object/images.
A-30a Magnification Content on a computer device's screen is magnified by the amount specified by the user. This is distinct from enlarging a font size or providing a static large print version. Candidates who have magnification in their PNP must have access to a magnification controls in the assessment user interface during the assessment session.
A-30b Magnification Zoom-amount For candidates who have supplied zoom-amount values within the text, non-text, and all-content values for magnification in the PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, although the candidate is still capable of modifying the settings during the assessment session.
A-31 Note-taking (on-screen) A feature providing the ability to take notes in a virtual notepad while working on another activity on the computer. Candidates who have note-taking-on-screen in their PNP must have access to an on-screen note-taking tool in the assessment user interface during the assessment session.
A-32 outliner-on-screen A feature providing the ability to easily create an outline of ideas (commonly a hierarchical list) while working on another activity on the computer. Candidates who have outliner-on-screen in their PNP must have access to an on-screen outliner tool in the assessment user interface during the assessment session.
A-33 Peer-interaction (on-screen) Candidates who have peer-interaction-on-screen in their PNP must have access to an on-screen peer interaction tool in the assessment user interface during the assessment session.
A-34 Prohibit set Within a candidate’s PNP, a set of supports that are specifically not available to the candidate in the user interface during an assessment session.
A-35 Pronunciation Lexicon Specification (PLS) For candidates who have indicated the spoken support in their PNP, any PLS files provided in the qti-external-supplemental-accessibility node are used to inform the text-to-speech engine how to pronounce words.
A-36 sign-language When content contains sign-language within a catalog, that content is presented to candidates who have the support listed in their PNP, in the sign language specified. In the context of a web-based presentation of content, sign language is generally presented as simultaneous content in a video alongside the default visually presented content (including text in a written language). The video playback must be controlled by the user.
A-37 simplified-graphics When content contains simplified-graphics within a catalog (usually as an alternative to a default image), that content is presented to candidates who have the support listed in their PNP.
A-38 simplified-language-portions When content contains simplified-language-portions within a catalog, that content is available for presentation to candidates who have the support listed in their PNP.
A-39 Spell-checker (on-screen) Candidates who have spell-checker in their PNP must have access to an on-screen spell-checking tool in the assessment user interface during the assessment session.
A-40a Spoken (reading-type = screen-reader) Candidates who have the spoken support in their PNP with a reading-type of “screen-reader” must be able to use a screen-reader during the assessment session, and content within the assessment is expected to be perceivable, understandable, and usable.
A-40b Spoken (reading-type = computer-read-aloud) Candidates who have the spoken support in their PNP with a reading-type of “computer-read-aloud” must have access to a computer interface that reads assessment content text aloud during the assessment session.
A-40c Spoken Restriction Type Candidates who have the spoken support in their PNP with a specific restriction-type must not have the restricted content read out loud to them during the assessment session.
A-40d Spoken, user preferences for speech-rate, pitch, volume, and link-indication For candidates who have supplied these optional spoken user preference parameters in their PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, though the candidate is still capable of modifying the settings during the assessment session.
A-40e Spoken (typing-echo) For candidates who have the spoken support in their PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, though the candidate is still capable of modifying the settings during the assessment session.
A-40f Text-to-Speech Suppression For candidates who have the spoken support in their PNP, content that is within an element that contains the attribute “data-qti-suppress-tts” must not be read aloud to the candidate based on their reading-type and the vocabulary provided in the attribute.
A-41 Tactile

Term: An object that can be felt with the hands.

QTI 3: A tactile representation of the graphic information is made available outside of the user's computer.

PNP: The term "tactile" indicates that the user must have access to tactile materials when available, and must be presented with any tactile tour content that describes the tactile materials.

A tactile tour is text content that provides information that orients the user about the tactile object. When item content contains tactile qti-html-content within a catalog, that content is presented to candidates who have the support listed in their PNP.

A-42a Text-appearance, text and background color For candidates with text-appearance in their PNP, the delivery system allows the user to at least change the text color and the color of the text background. This feature may also be available to all candidates regardless of their PNP.
A-42b Text-appearance, other For candidates with text-appearance with any of the optional user preferences in their PNP, the delivery system presents the text content using the preferences supplied in the PNP.
A-43 Thesaurus (on-screen) A resource that allows the candidate to reference words of similar meaning. Candidates with thesaurus-on-screen in their PNP have access to an on-screen thesaurus tool.
A-44 transcript

A nonsynchronous written alternative for time-based media content, which includes dialogue (including the speaker name/role) and descriptions of sounds and/or important information that is presented visually. When item content contains transcript content within a catalog, that content is presented to candidates who have the transcript support listed in their PNP.

PNP: The term "transcript" indicates that the user must have access to transcript content if available in a catalog resource.

A-45 Visual-organizer (on-screen) A tool that allows people to organize their thoughts by constructing a visual representation of their ideas. Candidates with thesaurus-on-screen in their PNP would access to an on-screen thesaurus tool.
A-46 Activate-as-option The term "activate-as-option-set" is a set of user preferences in a PNP. Assigning a feature or support to this set indicates that each feature must be made available to the user as an option to use, but the feature is not activated (the feature isn't actively in use) until requested by the user. A method for making the feature available to the user must be obvious, like (but not limited to) a button on a toolbar, an item in a drop down menu, or an option in an options window.
A-47 Activate-at-initialization The term "activate-at-initialization-set" is a set of user preferences in a PNP. Assigning a feature or support to this set indicates that each feature must be made available to the user as soon as the user initiates their working session (generally after logging into a content presentation system).

A.6 Results Reporting

ID Feature Name Feature Description
R-1 Context Reports the contextual information of an assessment
R-2 Test result Report on the results of all AssessmentTests from an assessment
R-3 Item Result Report on the results of all AssessmentItems from an assessment

A.7 QTI Metadata

ID Feature Name Feature Description
M-1 Item Template An AssessmentItem that contains a TemplateProcessing section. The value is true if the item is actually an item template, in other words, the item changes its appearance based on some random or external factor.
M-2 Time Dependent A time dependent item takes the length of time taken for an attempt into consideration when scoring.
M-3 Composite Value is True if the item comprises more than one interaction, for example, an AssessmentItem that contains more than one interaction.
M-4 Interaction Type The interaction type(s) of the item. The vocabulary is comprised of the names, as defined in the information model, of the leaf classes derived from interaction.
M-5 Feedback Type Describes the type of feedback, if any, available in the item. If feedback is available then it is described as being non-adaptive or adaptive depending on whether the item is itself adaptive. A non-adaptive item generates feedback based on the responses submitted as part of (the last) attempt only. An adaptive item generates feedback that takes into consideration the path taken through the item, in other words, feedback based on the accumulation of all attempts and not just the last.
M-6 Solution Available Set to the value to “true” if a model solution is available for the item. For example, an AssessmentItem that provides correct responses for all declared response variables.
M-7 Tool Name The name of the tool used to author the evaluation object.
M-8 Tool Version The version of the tool used to author the evaluation object.
M-9 Tool Vendor The company which produced the tool used to author the evaluation object.
M-10 Scoring Mode Specifies the scoring mode: human, external machine, or response processing.

A.8 Content Packaging

ID Feature Name Feature Description
P-1 Test Instances The package contains test (assessment) level structures.
P-2 Nested Section Instances The package contains nested section level structures.
P-3 Item Banks Item bank structure.
P-4 Item Instances The package contains qti-assessment-item instances.
P-5 Manifest Metadata (LOM) Information about the package as a whole is based upon the IMS binding of IEEE Learning Object Metadata (LOM).
P-6 Resource Metadata (LOM) Information about an Item, Section, Test, etc. is described as part of the resources specifically and is based upon the IMS binding of IEEE Learning Object Metadata (LOM).
P-7 Resource Metadata (QTI) Information about an Item, Section, Test, etc. is described as part of the resources specifically and is based upon the IMS QTI-specific metadata.
P-8 Curriculum Standards Metadata Learning Standards annotations for the resources and the manifest as a whole is supported using the IMS Curriculum Standards Metadata (CSM) approach developed as part of IMS Common Cartridge [CSM, 12]. These annotations are added in the corresponding metadata structures for the manifest and resource i.e. in the same manner as for the LOM entries. When using CSM the following XML should be added to the manifest element.
P-9 External Metadata Reference to a metadata file that is maintained and stored outside of the package.
P-10 Learning Tools Interoperability v1.1 The IMS Learning Tools Interoperability (LTI) standard prescribes a way to easily and securely connect learning applications and tools with platforms, portals, learning environments, etc. LTI could be used to launch an assessment, some external tool or resource, or to specific points in assessment content.
P-11 Stimulus Instances The package contains a stimulus file, and the manifest references the stimulus.
P-12 Variant This is a feature in the IMS CPv1.2 that is used to contain alternative resource information. A resource can have many variants   (for example, an English variant, a Spanish variant, and a Chinese Variant). In QTI 3, the variant structure is used to contain the metadata that describes the conditions under which the alternative forms of an item are available to be used. Variants for an item are packaged together in a single item content package.
P-13 CASE Competencies and Academic Standards Exchange (IMS CASE v1.0). Used to associate item, section, test level learning and education competencies that can be exchanged with competency systems or frameworks.

B. Conformance Statements

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words "MAY", "MUST", "MUST NOT", "OPTIONAL", "RECOMMENDED", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", and "SHOULD NOT" in this document are to be interpreted as described in [RFC2119].

An implementation of this specification that fails to implement a MUST/REQUIRED/SHALL requirement or fails to abide by a MUST NOT/SHALL NOT prohibition is considered nonconformant. SHOULD/SHOULD NOT/RECOMMENDED statements constitute a best practice. Ignoring a best practice does not violate conformance but a decision to disregard such guidance should be carefully considered. MAY/OPTIONAL statements indicate that implementers are entirely free to choose whether or not to implement the option.

The Conformance and Certification Guide for this specification may introduce greater normative constraints than those defined here for specific service or implementation categories.

C. Revision History

This section is non-normative.

C.1 Version History

Version No. Release Date Comments
IMS Public Candidate Final 3.0 20 July 2020 The first draft release.

D. References

D.1 Normative references

[AFA-30]
AccessForAll v3.0. IMS Global Learning Consortium. September 2012. IMS Public Draft. URL: https://www.imsglobal.org/activity/accessibility/
[CP-12]
Content Packaging v1.2. IMS Global Learning Consortium. March 2007. IMS Public Draft v2.0. URL: https://www.imsglobal.org/content/packaging/index.html
[QTI-IMPL-30]
QTI Best Practice and Implementation Guide v3. IMS Global Learning Consortium. April 2020. IMS Final Release. URL: https://www.imsglobal.org/spec/qti/v3p0/impl/
[QTI-OVIEW-30]
QTI Overview v3. IMS Global Learning Consortium. April 2020. IMS Final Release. URL: https://www.imsglobal.org/spec/qti/v3p0/oview/
[QTI-TERM-30]
QTI Terms and Definitions v3. IMS Global Learning Consortium. April 2020. IMS Final Release. URL: https://www.imsglobal.org/spec/qti/v3p0/term/
[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. IETF. March 1997. Best Current Practice. URL: https://tools.ietf.org/html/rfc2119

E. List of Contributors

The following individuals contributed to the development of this document:

Name Organization Role
Arjan AarninkCito
Vijay AmbatiACT, Inc.
Jérôme BogaertsOpen Assessment Technologies
Shiva BojjawarMcGraw-Hill Education
Catriona BuhayarNWEA
Jason CarlsonACT, Inc.
Gary DriscollETS
Michael FingerMeasured Progress
Paul GrudnitskiPearson
Mark HakkinenETS
Thomas HoffmannETSEditor
Rob HowardNWEA
Stephen KacsmarkInstructure
Justin MarksNWEA
Amy MarrichOpen Assessment Technologies
Mark McKellIMS Global
Mark MolenaarOpen Assessment Technologies
Brian MottersheadRiverscape Software
Nate OberQuestar
Padraig O'hiceadhaHoughton Mifflin Harcourt
Madeline RothbergWGBH
Colin SmytheIMS Global
Tjeerd Hans TerpstraCito
Travis ThompsonData Recognition Corporation
Jason WhiteETS

IMS Global Learning Consortium, Inc. ("IMS Global") is publishing the information contained in this document ("Specification") for purposes of scientific, experimental, and scholarly collaboration only.

IMS Global makes no warranty or representation regarding the accuracy or completeness of the Specification.

This material is provided on an "As Is" and "As Available" basis.

The Specification is at all times subject to change and revision without notice.

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

IMS Global would appreciate receiving your comments and suggestions.

Please contact IMS Global through our website at http://www.imsglobal.org.

Please refer to Document Name: IMS Global QTI v3 Conformance and Certification 3.0

Date: 20 July 2020