1EdTech QTI v3 Conformance and Certification
Question & Test Interoperability (QTI) 3.0 Conformance and Certification
Spec Version 3.0
| Document Version: | 2.0 |
| Date Issued: | 15 March 2022 |
| Status: | This document is made available for adoption by the public community at large. |
| This version: | https://www.imsglobal.org/spec/qti/v3p0/conf/ |
| Latest version: | https://www.imsglobal.org/spec/qti/latest/conf/ |
| Errata: | https://www.imsglobal.org/spec/qti/v3p0/errata/ |
IPR and Distribution Notice
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.
IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS's procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights webpage: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf .
The following participating organizations have made explicit license commitments to this specification:
| Org name | Date election made | Necessary claims | Type |
|---|---|---|---|
| HMH | September 16, 2019 | No | RF RAND (Required & Optional Elements) |
Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/speclicense.html.
Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.
The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.
THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.
Public contributions, comments and questions can be posted here: http://www.imsglobal.org/forums/ims-glc-public-forums-and-resources .
© 2023 IMS Global Learning Consortium, Inc. All Rights Reserved.
Trademark information: http://www.imsglobal.org/copyright.html
Abstract
The 1EdTech Question and Test Interoperability® (QTI®) specification is a technical standard for digital interchange of assessment content (items and tests), assessment usage data and results reporting. Use of QTI 3.0 enables assessment materials to be exchanged digitally among a wide variety of products, such as item/test authoring products, item banks, and test delivery systems [QTI-OVIEW-30].
QTI 3.0 is based on the well established QTI standard that has been used internationally for two decades and 1EdTech Accessible Portable Item Protocol® (APIP®). QTI 3.0 adds support of digital delivery options for a range of common accessibility needs and enables transform-free authoring to delivery workflow. Implementations of QTI 3.0 may require use of related standards, including the 1EdTech Content Packaging 1.0 specification, and the AccessForAll® (AfA) 3.0 specification.
One of the principal ways that 1EdTech promotes conformance to the QTI 3.0 standard is by creating a certification process, which is detailed in this documentation. Additionally, and as a part of the certification process, 1EdTech employs online validators for use by member organizations to ensure conformance to the QTI 3.0 specification.
The objective of the QTI 3.0 Conformance and Certification process is to increase interoperability between vendors and promote the adoption of the QTI 3.0 standard at entry (minimum) and more advanced implementation levels.
1. Introduction
1.1 Scope and Context
The purpose of this document is to define the different areas of conformance available for QTI 3.0 and to describe the procedures for achieving certification for version 3.0. QTI 3 is a standard created and governed by 1EdTech Consortium, a non-profit learning standards organization. This document is intended to be helpful to vendors who wish to certify their systems as officially conforming to QTI version 3.0, and for prospective consumers of QTI 3.0 for use in their assessment programs.
This document includes the specification certification tables within Section 4: Conformance Statements, and detailed descriptions of all the QTI conformance features are available in Appendix A.
1.2 Conformance Statements
As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.
The key words MAY, MUST, MUST NOT, OPTIONAL, RECOMMENDED, REQUIRED, SHALL, SHALL NOT, SHOULD, and SHOULD NOT in this document are to be interpreted as described in [RFC2119].
An implementation of this specification that fails to implement a MUST/REQUIRED/SHALL requirement or fails to abide by a MUST NOT/SHALL NOT prohibition is considered nonconformant. SHOULD/SHOULD NOT/RECOMMENDED statements constitute a best practice. Ignoring a best practice does not violate conformance but a decision to disregard such guidance should be carefully considered. MAY/OPTIONAL statements indicate that implementers are entirely free to choose whether or not to implement the option.
The Conformance and Certification Guide for this specification may introduce greater normative constraints than those defined here for specific service or implementation categories.
1.3 Status of this Document
1EdTech strongly encourages its members and the community to provide feedback to continue the evolution and improvement of the QTI standard. To join the 1EdTech developer and conformance certification community focused on QTI 3 please visit the QTI Specification online here: https://www.imsglobal.org/question/index.html.
Public contributions, comments and questions can be posted here: Public QTI 3 Forums.
2. Conformance and Certification Overview
A QTI 3.0 certification statement will make a declaration of:
- The certification profile: 1EdTech Level or community profile name
- The kind of certification (Application or Content Package)
- The capabilities of an application certification
- The specific version for which they have received certification (e.g. QTI version 3.0.1).
Details related to items 1–3 above are provided in Sections 3.1–3.3.
QTI 3.0 Results Reporting and QTI Usage Data & Item Statistics will have separate certification programs. Details for these certification programs are available on the 1EdTech member website.
At this time, certification is not available for PNP (candidate registration and profiling) systems. There is an online validator available to 1EdTech members for the validation of specific PNP instances, which validates to the AfA 3.0 QTI 3.0 Profile. In addition, Delivery/Presentation systems may be required to support PNP data, depending on their certification level, or based on the requirements of any particular assessment program.
Validation of instances of exchangeable QTI 3.0 materials can verify the materials as conformant to the QTI 3.0 standard for the available certification levels. However, the validation can only check if the materials are technically valid, not whether the materials themselves are usable by people.
If an organization using the official 1EdTech QTI 3.0 validator thinks the validator to be in error, the organization must immediately report the error to the 1EdTech Support email: support@1edtech.org.
2.1 Certification Levels for QTI 3.0
Certification for QTI 3.0 is available for two 1EdTech Certification Profile Levels, as approved by the 1EdTech Assessment Product Steering Committee:
- Basic Level
- Advanced Level
The Basic certification level includes all of the required features for the most rudimentary QTI 3.0 systems or for generated instances of content from those systems. Advanced certification includes all of the features and functionality of the Basic level, and adds additional conformance requirements. Applications that are Advanced certified are also considered to be Basic certified.
In addition, any certification level (for Applications and Content Packages) can also achieve a QTI Elevated Accessibility certification. This certification includes a bundle of recommended accessibility and accommodation features that help support candidates taking assessments. The Elevated Accessibility certification involves the use of AfA 3.0 PNP data to meet the specific needs of candidates. Applications must achieve one of the 1EdTech certification levels to be eligible to receive an Elevated Accessibility certification. For example, a system that achieved Core certification and Elevated Accessibility certification is said to be QTI 3.0 Core + Elevated Accessibility Certified.
Features in the QTI certification context means a named and numbered feature included in the various conformance areas: Tests, Sections, Items, Questions (interactions), Accessibility, Packaging and Metadata. These features might include a bundle of required elements (classes) that enable that feature.
2.2 Certification Capabilities for Applications
A certified QTI application imports and/or exports QTI content packages, QTI Results files, or QTI Usage Data files.
In the context of QTI certification, a “capability” is a set of functions, behaviors, or group of features that the application can perform, as defined by the specific capability. Applications may elect to get certified beyond their IMPORT or EXPORT capabilities.
Currently, there are a number of capabilities that a certified application can achieve for QTI 3, including:
- EXPORT – Tests with Item Packages
- EXPORT – Item Only Packages
- IMPORT – Tests with Item Packages
- IMPORT – Item Only Packages
- Authoring – Items
- Authoring – Tests
- Tests with Items
Future capabilities may include:
- IMPORT – PCI v1.0
- EXPORT – PCI v1.0
- Delivery – PCI v1.0 Required Features
- IMPORT – Usage Data
- EXPORT – Usage Data
- IMPORT – QTI Results
- EXPORT – QTI Results
- Delivery – CAT v1.0 Engine (Provider)
- Delivery – CAT v1.0 Presentation (Consumer)
Additional capabilities may be added to the QTI 3.0 Certification program in the future.
The definitions for the currently available capabilities in the context of QTI conformance are described below:
AUTHORING: asserts the application is capable of creating new assessment content, or modifying stored assessment content. While QTI certified AUTHORING applications are not required to store content in QTI, the system must be capable of creating or modifying content that aligns with QTI features, which could include any of the required features of the certification levels or profiles for which they have been certified.
The authoring capability will state whether it can author Tests (assessments) and/or Items (questions). If the authoring system can author both items and tests, both capabilities will be listed in the certification.
DELIVERY: asserts the application is capable of presenting QTI content to candidates for the feature requirements of the certification level or profile for which they have been certified, and may include the ability to combine candidate PNP data with the QTI content, and within the context of the PNP and available content, deliver content that is specifically designed for the candidate’s needs. Applications with delivery capability may need to meet standards of visual presentation for QTI-specific classes and attributes (based on the certification level or profile requirements), as detailed in the QTI 3 Best Practices and Implementation Guide. Applications with delivery capability must be able to collect and store the responses provided by the candidate, and if the test is designed to allow the candidate to return to questions, accurately present the candidate’s stored response back to the candidate. Applications with delivery capability must be able to meet any test-specific parameters (e.g. time allotted). Applications with delivery capability may need to be able to adjust the test-specific parameters if a candidate’s PNP indicates accommodations for those parameters.
Unless otherwise stated applications with the delivery capability deliver compliant QTI items and tests. Specific capabilities for the delivery of PCI, CAT, or other specific QTI related standards will be listed separately in the certification.
SCORING: the application consumes QTI Content Packages and QTI Results Reporting data to allow for external manual or automated (Artificial-Intelligence based) scoring. Applications with scoring capability need item definitions metadata, including scoring rubrics.
EXPORT: asserts the system is capable of exporting QTI content, and the system is capable of meeting the feature requirements of the certification levels or profiles for which they have been certified, including packaging requirements. Exported content must be able to pass the 1EdTech validator without errors for the certification version claimed.
IMPORT: asserts the system is capable of importing QTI content, and the system is capable of storing (without loss of data) the feature requirements of the certification levels or profiles for which they have been certified. Importing systems must be able to differentiate between valid and invalid instances of QTI content for the certification levels or profiles claimed by the system.
2.3 Certification of Content Packages
Members can submit specific named assessment content products for Content Package certification. Online instructions will guide members to submit all or representative portions of their named product to the online validator. If the validator confirms there are no errors, members can submit the validation report along with the company and product information and the Content Package will be listed in the 1EdTech TrustEd Apps directory.
The certification may list a specific profile that aligns with the content, or members may request that specific features are listed with the certification.
Content Package certifications do not require annual recertification.
2.4 Community Conformance Profiles
The 1EdTech will support a registration system for common, broader profiles for vendors to access, known as a Community Conformance Profile. All Community profiles must be conformant with the 1EdTech QTI 3.0 Entry Certification set of required features. All the features within a community profile are a subset of the full QTI 3.0 feature set.
2.5 Certification Publication
After receiving your 1EdTech certification, your organization’s certification for any particular 1EdTech level or Community Profile, will be listed on the 1EdTech web site at http://imscert.org.
2.6 Annual Re-Certification Requirement
Certification is to be done annually for specific QTI 3 versions (final and public releases), beginning with QTI 3.0. A vendor may certify in multiple QTI versions, or may wish to only certify in the most recent version, or certify in only one of the specific legacy versions.
Test materials will incorporate any new test materials (errata) into the test suite. Systems seeking certification or re-certification should always pull the most recent version of the testing materials from the 1EdTech certification site as updated materials are added as needed.
Certification submissions will be reviewed and approved by 1EdTech.
2.7 Interoperability Assurances
In addition to 1EdTech Certifications (1EdTech Levels or Community Conformance Profiles), organizations can also establish a process that raises confidence of interoperability between the involved parties.
Participating vendors will need to establish which of the QTI 3.0 features the contract needs (or that the contract requires) to support, then ensure that the trial packages they exchange contain instances of all the listed features. Essentially, organizations create a Contract Assurance that uses existing (or proposed) QTI 3.0 features. The exchange process should always start with using the official 1EdTech validator for any data/materials used in any of the exchanges.
When a Contract Assurance is across many vendors, or established by a government agency, the participating organizations may wish to involve the 1EdTech by establishing an official Community Conformance Profile. Certification to those published profiles may also be possible or desirable.
Below is a list of examples where a Contract Assurance may be useful to the involved organizations.
- Two vendors are contracted by a government agency to author and deliver a suite of assessments.
- A single vendor is contracted by a government agency to deliver an assessment.
- Two vendors contract together, where one of the vendors is the preferred delivery system for the other vendor.
2.7.1 Recommended Interoperability Assurance Process
While the numbered steps below detail specific steps, the exchanging organizations may find different processes are more effective for their specific contexts. The list provided is a suggested approach only:
- Decide which features are needed for your project. You many need assistance from 1EdTech, consultants, or 1EdTech groups to understand the specific features. It is recommended that all contracts that use QTI 3.0 follow at least the 1EdTech Entry Certification features. All the participating parties need to agree to the list of features.
- Create sample content (assessment content and/or PNP files) that incorporates ALL of the features included for the project.
- Verify that materials created are VALID QTI 3.0 using the official 1EdTech validators.
- Exchange the content in the direction of the work flow expected for the contracted scope of work.
- Verify that the final endpoint (for example: import in a content management system, or candidate presentation in a delivery system) matches the expectations of the contract. Also verify that the examples use all the agreed upon features of the contract.
- If your project allows for the use of Assistive Technologies (AT), it is recommended that users familiar with the specified AT interact with the sample content to ensure that content not only meets the technical expectations of the project, but that the content is actually accessible (perceivable, operable, understandable, robust – the WCAG 2.0 Principles of accessibility) by users of the specified AT.
- For any features that do not meet the expectations of the project, determine if the problem stems from the supplied source or any of downstream systems which receive and use the materials.
- Iterate the above process (steps 2–7) until all features are verified from all the participating systems.
- Report the Interoperability Assurance to the contracting agency or agencies.
3. Conformance Features for Certification Levels
This section contains a series of tables that detail the required features for 1EdTech Certification Levels. The tables only include features that are required in at least one of the certification levels. For a complete list of QTI 3.0 features and their descriptions, see Appendix A: QTI 3.0 Feature List.
QTI 3 supports many possible features, and it is unlikely (though not impossible) that any particular application supports all the QTI 3.0 features. In fact, many of the features are not included in any 1EdTech certification levels or Community Conformance Profiles, and are considered “elective.” However, many of these elective features are used for specific programs and projects. These features may be added to certification levels or community profiles in the future.
All QTI 3.0 certified applications must be able to operate without failure when features beyond the minimum required features are included in the content or PNP instance. While certified applications don’t need to support, or even store the “extra” features/data, the applications must be able to ignore or remove the unsupported data.
Exceptions to the above requirement include:
- A conformant application does not have to support a PNP bulk record instance.
Exceptions would not apply where the Community Conformance Profile requires the feature, or where the feature is specified in a Contract Assurance.
Delivery applications are expected to be able to identify (on import) and display notifications for content that is not valid. Applications with delivery capability may continue to present that invalid content to the candidate at their own risk.
3.1 1EdTech Levels Certification Requirements
Below is a list of features required for certification for 1EdTech profiled certification levels. For a full description of each feature, see Appendix A: QTI 3.0 Feature List.
An empty cell indicates that the feature is not required for certification, but the system may optionally support the feature without affecting the system’s certification, except that the optional feature is published in conformance statements.
R = REQUIRED. An application MUST support this feature to achieve certification.
3.1.1 INTERACTIONS
| Feature ID | Feature Name | Basic | Advanced |
|---|---|---|---|
| Q-2 | Choice Interaction | R | R |
| Q-5 | Extended Text Interaction | R | R |
| Q-6 | Gap Match Interaction | R | |
| Q-8 | Graphic Gap Match Interaction | R | |
| Q-10 | Hot-spot Interaction | R | |
| Q-11 | Hot-text Interaction | R | |
| Q-12 | Inline Choice Interaction | R | |
| Q-13 | Match Interaction | R | |
| Q-15 | Order Interaction | R* | |
| Q-20 | Text Entry Interaction | R | R |
*Requirement excludes support for the min/max attribute
Interaction Conformance Notes:
- Basic level applications need only support the required elements and attributes for the interactions. Unless otherwise noted, Advanced systems need to support the required and optional elements and attributes for interactions.
- Template and Feedback features within Interactions are only required if implementing these elective features.
- The shuffle attribute is a discretionary trait of all the interactions, and is not required at any 1EdTech certification level.
- Interaction in the Basic Level must also support a related subset of QTI shared vocabulary terms (see I-19a).
3.1.2 ITEM LEVEL FEATURES
| Feature ID | Feature Name | Basic | Advanced |
|---|---|---|---|
| I-0 | Assessment Item Root | R | R |
| I-1 | Response Declaration | R | R |
| I-2 | Outcome Declaration | R | R |
| I-4 | Shared Stimulus | R | |
| I-7 | Item Body (some exclusions) | R | R |
| I-8 | HTML5 (QTI subset) | R | R |
| I-9a | Response Processing – Full | R | |
| I-9b | Response Processing – Template | R | R |
| I-11 | Rubric Block | R | |
| I-14 | Catalog Resources | R | |
| I-17 | Composite Items | R | |
| I-18 | MathML | R | |
| I-19a | QTI Shared Vocabulary (subset) | R | R |
| I-19b | QTI Shared Vocabulary (full) | R |
3.1.3 ACCESSIBILITY*
| Feature ID | Feature Name | Basic | Advanced |
|---|---|---|---|
| A-1 | Alt text for graphics | R | R |
| A-13b | Captions (track support) | R | |
| A-15 | Glossary (on-screen) | R | |
| A-30a | Magnification (all-content) – Delivery | R | |
| A-42a | Text-appearance, text and background color – Delivery | R |
*See also the Elevated Accessibility Conformance table.
3.1.4 TEST LEVEL FEATURES
| Feature ID | Feature Name | Basic | Advanced |
|---|---|---|---|
| T-1 | Outcome Declaration | R | |
| T-2 | Time Limits | R | |
| T-4 | Test Part | R | R |
| T-5 | Item Session Control in TestPart | R | |
| T-7 | Sections in Test Parts | R | R |
| T-9 | Outcomes Processing for Tests | R | |
| T-12 | Multiple Sections in TestPart | R | |
| T-13 | Rubric Blocks for AssessmentTest | R | |
| T-14 | Record and Restore Responses (Delivery) | R | R |
3.1.5 SECTION LEVEL FEATURES
| Feature ID | Feature Name | Basic | Advanced |
|---|---|---|---|
| S-0 | Section Root (AssessmentSection.Type) | R | R |
| S-2 | Item Session Control in Sections | R | |
| S-3 | Selection | R | |
| S-4 | Ordering | R | |
| S-5 | Rubric Block in Sections | R | |
| S-7 | Items | R | R |
| S-9 | Assessment Section Referencing | R |
3.1.6 CONTENT PACKAGING FEATURES
| ID | Feature Name | Basic | Advanced |
|---|---|---|---|
| P-1 | Test Instances | R | R |
| P-4 | Item Instances | R | R |
| P-7 | Resource Metadata (QTI) | R | |
| P-11 | Stimulus Instances | R |
3.2 Certification for Elective Features
In addition to applications achieving certification for the 1EdTech profiled levels, community profiles, and Elevated Accessibility, applications may also make conformance statements (added to their published certification conformance statements) about individual features that are outside of the defined list of required features for particular certification levels or profiles. Any feature (as listed in Appendix A: QTI Feature List) that is outside of the defined list of required features for certification levels or profiles is considered “elective” – meaning it is optional for an application to implement the feature.
For applications seeking to make conformance statements for elective features for the EXPORT capability, applications must export at least two examples of content that makes use of the feature (and it’s required classes and attributes) and verify that the content passes the official online 1EdTech validator for the version for which the system is seeking certification. The examples MUST be provided to 1EdTech, and may be reviewed by the QTI 3.0 Specification & Development group before conformance is granted to the submitting application.
For applications seeking to make conformance statements for the IMPORT capability, applications MUST be able to import the feature (and present the feature if it is a Delivery system) and retain all the information provided with the content. The certification test content will be provided on the 1EdTech QTI 3.0 Conformance and Certification GitHub repository in the Elective Features directory (though certain features may be found in higher level certification directories).
The additional feature(s) should be listed in the documentation submitted to the 1EdTech for certification. As is true of certification for levels and profiles, conformance statements for specific elective features will need to be renewed annually (usually done in conjunction with certification at a particular level or profile).
3.3 Elevated Accessibility Conformance Requirements
Below is a table which lists the accessibility and accommodation supports required to achieve an additional Elevated Accessibility certification for your 1EdTech Certification. All systems seeking Elevated Accessibility certification will need to support candidate PNP data for at least the features listed in the Elevated Accessibility feature list. All the named requirements are required for Elevated Accessibility certification. All other accessibility and accommodations are considered elective, unless specifically required by a Community Profile or Contract Assurance.
| Feature ID | Feature Name |
|---|---|
| I-14 | Catalog Resources |
| A-2a | WAI-ARIA Subset – IMPORT and Delivery |
| A-6 | additional-testing-time – Delivery |
| A-8 | answer-masking – Delivery |
| A-9 | audio-description |
| A-10a | braille |
| A-11 | breaks – Delivery |
| A-13b | Captions (track support) |
| A-29 | long-description |
| A-30a | Magnification (all-content) – Delivery |
| A-36 | sign-language |
| A-40a | spoken (reading-type=screen-reader) – Delivery |
| A-40b | spoken (reading-type=computer-read-aloud) – Delivery |
| A-41 | tactile |
| A-42a | text-appearance, text and background color – Delivery |
| A-44 | transcript |
3.4 PNP Conformance Requirements
Below is a table which lists the supports expected to be supported in the applications achieving Elevated Accessibility certification that rely on PNP data for their assessment programs. While there is no current certification process for the QTI 3.0 PNP profile of AfA 3.0, the 1EdTech validator will check for valid instances of QTI 3.0 PNPs. In the future, there may be a process that allows applications to certify their PNP conformance. The table below is for reference purposes only.
| Feature ID | Feature Name |
|---|---|
| A-1 | Alt text for graphics |
| A-8 | answer-masking |
| A-9 | audio-description |
| A-10a | braille |
| A-11 | breaks |
| A-29 | long-description |
| A-30a | Magnification (all-content) |
| A-36 | sign-language |
| A-40a | Spoken (reading-type) |
| A-40c | Spoken (restriction-type) |
| A-41 | tactile |
| A-42a | Text-appearance, text and background color |
| A-44 | transcript |
4. Achieving Certification
Validation for QTI 3.0 conformance consists of the following strategies:
- Self-Testing. Applications seeking certification will test themselves, and report the results to the 1EdTech;
- Automated Validation. An electronic process to verify conformance using automated tools provided by the 1EdTech Assessment Product Steering Committee.
- Documented or Demonstrable Validation. Documented statements and/or demonstrable capabilities showing conformance that can be reviewed by the 1EdTech QTI Assessment Product Steering Committee or by the user;
- Test Case Based Validation. Using a standard set of QTI 3 content packages and PNP files with known characteristics provided by the 1EdTech QTI Working Group (as approved by the 1EdTech Assessment Product Steering Committee) that must be used as part of the validation process and have known expected results that can be documented and/or demonstrated.
Because the QTI 3.0 standard is primarily a data transfer format, conformance and certification does not govern how applications manage or modify data within the system itself. For example, delivery applications must be able to combine the information provided by imported content and PNP files, but need not natively use the QTI 3.0 format during delivery.
There are limits to what can be verified using automated software tools. Export functions are much more easily verified using automation than import functions. The output from an export process must adhere to the QTI 3.0 XML specifications and can be verified using automation. Import is much more difficult to verify because the import process results in a proprietary system consuming the information and representing it in an internal format that does not need to adhere to the XML specification and cannot be verified using automation. Each of these approaches is discussed in detail below.
4.1 Certification through Self-testing
A certifying application would self-test by filling out the relevant checklist and submit that to the 1EdTech. Applications certifying exported packages need to also submit proof of passing validation. The validator will report the content level achieved, and list all the features correctly used in the content, as well as the published conformance statements. The validator will report any errors in the content as well. Instances containing errors are not considered valid instances for purposes of certification or exchange.
4.2 Automated Validation
Capabilities to be validated using automated tools include:
- QTI 3.0 Content Package export;
- QTI 3.0 PNP file export.
For export, the QTI 3.0 conformance approach focuses on determining if the application can generate an object (QTI 3.0 item or AfA PNP instance) that is non-problematic. 1EdTech provides an online tool available for individuals or organization to submit an object (item or PNP) and then receive information about the existence (and sometimes nature) of problems with syntax and semantics of the object.
It is expected that the automated validation tool will:
- Accept Entry, Core, Advanced, and all elective QTI 3.0 conformance content elements identified in the conformance matrix.
- Accept custom (and profiled) extensions that are agreed to between exchanging parties (interdependent vendors), as allowed for extension by the QTI 3.0 XSD.
4.3 Documented or Demonstrable Validation
Capabilities to be documented and/or demonstrated by the solution provider include:
- A statement from the 1EdTech validator for exported QTI 3.0 content,
- Render QTI 3.0 content in accordance with the QTI 3.0 Best Practices & Implementation Guide, supporting differences in presentation as required by PNP file information,
- A computer generated report from an application with import capability that verifies the content was imported properly, noting any content, or portions of content that were not fully imported.
As each test delivery application may provide a unique user experience and use different technologies to deliver content, providing automated validation of each implementation is virtually impossible. Therefore, 1EdTech Staff (at the direction of the Assessment Product Steering) will rely on the solution provider to document that the required QTI 3.0 content and PNP features have been implemented and can document and/or demonstrate them. The documents and demo applications can be submitted to 1EdTech for consideration and review but 1EdTech will not be able to verify the exact application behaviors.
4.4 Test Case Validation
1EdTech’s Assessment Product Steering Committee members will contribute reference content for interoperability/conformance testing. Test packages will be created by the 1EdTech for the two Certification Levels and the full QTI 3.0 feature set.
The test set includes sample content that includes the complete set of QTI 3.0 features for the specific version for which an application is seeking certification. All certified QTI 3.0 applications are required to ignore QTI 3.0 features outside the required set of features for which that application is seeking, or has received, certification. Applications that do not support specific features can leave the exchangeable materials unaltered, but not implement the unsupported features. Alternatively, applications may elect to remove the unsupported features from the exchangeable materials for use within their applications.
Applications certified at a particular level, or for a specific Community Profile, are not required to return, or pass on, exchangeable content in the same exact form in which it was received, except to the extent that the content must still contain all the data and features for which that application has received certification, and that the materials exported from that application are valid QTI 3.0 instances.
To conduct import validation, the test sets contain some materials that are problematic and others that are non-problematic. Import functionality focuses on determining if the application can (a) flag problematic objects that are known to have problems in syntax or semantics and (b) recognize as non-problematic the objects that are known to be without problems.
Capabilities to be validated using a standard set of test cases include:
- QTI 3 Content Package import
- AfA 3.0 (QTI 3 Profile) PNP file import
- AfA 3.0 (QTI 3 Profile) PNP bulk records file import
A. APPENDIX A: QTI 3.0 Feature List
Below is a list of all identified QTI 3.0 features. Only features fully supported and provided in these lists will be counted listed in published conformance statements.
A.1 QTI Tests
| ID | Feature Name | Feature Description |
|---|---|---|
| T-1 | Outcome Declaration | Outcome variables are declared by outcome declarations on the test-level. Their value is set either from a default given in the declaration itself or by a rule set by outcomes processing, e.g. to sum all item-scores on a test. |
| T-2 | Assessment Time Limits | In the context of a specific AssessmentTest, an item or group of items, may be subject to a time constraint. This specification supports both minimum and maximum time constraints. |
| T-3 | Stylesheet | A reference to a stylesheet. |
| T-4 | Test Part | A TestPart represents a major division of the test and is used to control the basic mode parameters that apply to all sections and sub-sections within that TestPart. |
| T-5 | Item Session Control (ISC) in Test Part | When items are referenced as part of a test, the test may impose constraints on how many attempts, and which states are allowed. These constraints can be specified for individual items, for whole sections or for an entire TestPart. By default, a setting at TestPart level affects all items in that part unless the setting is overridden at the AssessmentSection level or ultimately at the individual AssessmentItemRef. The defaults for an ItemSessionControl are used only in the absence of any applicable constraint. |
| T-6 | ISC Time Limits in Test Part | Time Limits as they apply to individual TestPart(s). |
| T-7 | Sections in TestPart | An assessment section groups together individual item references and/or sub-sections. A section can be composed of any hierarchy/combination of items and sections. A section can only reference an item using an qti-assessment-item-ref object but it may contain or reference other sections. The grouping of the sections/items depends upon the nature of the parent section i.e. each section can be used for different grouping criteria e.g. organizational, pedagogic, etc. |
| T-8 | Test Feedback in TestPart | Feedback provided to the candidate based on conditions set at the TestPart level. |
| T-9 | Outcomes Processing for Test | |
| T-10 | Test Feedback (test-level) | Feedback provided to the candidate based on conditions set at the test level. |
| T-11 | Multiple Test Parts | Support for more than one TestPart in a TestAssessment |
| T-12 | Multiple Sections in TestPart | Support for more than one AssessmentSection in a TestPart |
| T-13 | Rubric Block for AssessmentTest | Support for TestRubricBlock content in an AssessmentTest, including all the attributes for TestRubricBlock and their vocabularies |
| T-14 | Record and Restore Responses | Delivery: A candidate’s responses are correctly saved. When the session is discontinued and reinitiated the candidate’s responses are restored correctly. |
A.2 QTI Sections
| ID | Feature Name | Feature Description |
|---|---|---|
| S-0 | Section Root | Support for root class AssessmentSection.Type |
| S-1 | Item Session Control | Parameters used to control the allowable states (maximum attempts, show feedback, allow review, show solution, allow comment, allow skipping, validate responses) of each item session (may be overridden at sub-section or item level). |
| S-2 | Time Limits Section Level | Optionally controls the amount of time a candidate is allowed for this item or section. |
| S-3 | Selection | The selection class specifies the rules used to select the child elements of a section for each test session. If no selection rules are given it must be assumed that all elements are to be selected. The selection class also provides an opportunity for extensions to this specification to include support for more complex selection algorithms. |
| S-4 | Ordering | The ordering class specifies the rule used to arrange the child elements of a section following selection. If no ordering rule is given, assume that the elements are to be ordered in the order in which they are defined. |
| S-5 | Rubric Block | The container for the rubric block content. A rubric block identifies part of the content that represents instructions to one or more of the actors that view the item. |
| S-6 | Section Include | This allows assessment content that would usually be included in the same file as the AssessmentSection to be provided as a separate (possibly shared by more than one item or test) file which is included in the referencing file using the XML Inclusions standard ( https://www.w3.org/TR/xinclude/ ) |
| S-7 | Items | Support for references to AssessmentItems. |
| S-8 | Nested Sections | Support for sections that are within sections. |
| S-9 | Section References | Using a reference to an AssessmentSection via qti-assessment-section-ref |
A.3 QTI 3.0 Items
| ID | Feature Name | Feature Description |
|---|---|---|
| I-0 | Assessment Item Root | Support for the AssessmentItem Element’s attributes. |
| I-1 | Response Declaration | Response variables are declared using the qti-response-declaration node and the declarations are bound to interactions in the qti-item-body. The response declaration supplies the values for the scoring of the item. It indicates what the correct response is, or the partially correct response(s). |
| I-2 | Outcome Declaration | Outcome variables are declared using the qti-outcome-declarations node. The outcome is the score that the candidate can receive for responding to, or not responding to, the item. |
| I-3 | Template Declaration | Use the TemplateDeclaration node to declare item variables that are to be used specifically for the purposes of cloning items. They can have their value set only during TemplateProcessing. They are referred to within the qti-item-body in order to individualize the clone and possibly also within the response processing rules if the cloning process affects the way the item is scored. |
| I-4 | Shared Stimulus reference | The AssessmentStimulusRef structure that enables reference to an 'AssessmentStimulus' instance. The stimulus must be contained within its own instance and the item uses the 'qti-assessment-stimulus-ref' structure to provide the link between the item and the stimulus. |
| I-5 | Template Processing | Use the qti-template-processing node for template processing. Template processing consists of one or more templateRules that are followed by a cloning engine or delivery system in order to assign values to the template variables. Template processing is identical in form to response processing except that the purpose is to assign values to template variables, not outcome variables. |
| I-6 | stylesheet | The stylesheet node is used to associate an external stylesheet for the item. QTI 3 supports CSS 2.1 and CSS 3.0 stylesheets (except CSS3 Speech). Any number of stylesheets can be included in an item. |
| I-7 | Item Body | The ItemBody (element qti-item-body) contains the text, graphics, media objects
and interactions that describe the item's content and information about how it is
structured.
Interactions have their own certification level requirements. (See the QTI 3.0 Interactions table.) Support for qti-rubric-block is required at the Core Level. Support for InfoControl, Feedback, and Template features are elective, or as required by a Community Profile or contract. |
| I-8 | HTML5 (QTI Subset) | Support for all the HTML 5 elements as detailed within the QTI 3.0 Schema. QTI does not support every HTML element. Rather, it lists specific elements that are permitted for use within QTI. |
| I-9a | Response Processing: Full | The process by which a delivery engine assigns outcomes (the score) based on the candidate's responses. Response processing can be achieved using either a response processing template (a reference to a common processing pattern), or by using customized response processing. |
| I-9b | Response Processing: Fixed-Template | The process by which a delivery engine assigns outcomes (the score) based on the candidate's responses. Response processing is achieved using a response processing template (a reference to a common processing pattern). You need only to support the templates for the interactions required for the certification level you are seeking. |
| I-10 | Modal Feedback | The qti-modal-feedback node is used for modal feedback to be presented to the candidate directly following response processing. The value of an outcome variable is used in conjunction with the show-hide and identifier characteristics to determine whether or not the feedback is presented. The content of the qti-modal-feedback cannot contain any interactions. |
| I-11 | Rubric Block | The container for the rubric block content. A rubric block identifies part of the content that represents instructions to one or more of the actors that view the item. Although rubric blocks are defined as SimpleBlocks they must not contain interactions. The visibility of nested qti-rubric-blocks is determined by the outermost element. In other words, if an element is determined to be hidden then all of its content is hidden including conditionally visible elements for which the conditions are satisfied and that therefore would otherwise be visible. |
| I-12 | Item Include | This allows assessment content that would usually be included in the same file as the AssessmentItem to be provided as a separate (possibly shared by more than one item) file which is included in the referencing file using the XML Inclusions standard ( https://www.w3.org/TR/xinclude/ ) |
| I-13 | Companion Materials | Item content provides information within the qti-companion-materials-info node that
indicates that the item must be presented in conjunction with specific materials,
including: calculators, protractors, rules, digital materials, and physical materials.
Delivery Systems would provide the listed materials during the assessment session
in conjunction with the item content.
Calculator: Content that indicates a particular type of calculator (basic, standard, scientific, or graphing) must be provided by the delivery system. Protractor: Content that indicates a protractor must be provided by the delivery system, using the parameters provided. |
| I-14 | Catalog Resources | Support for content held within a qti-content-info node, including qti-catalog, qti-card, and qti-card-entry containers, and the qti-html-content & qti-file-href nodes. |
| I-15 | Context Declarations | The qti-context-declaration node is used to make contextual declarations. When evaluating TemplateProcessing and ResponseProcessing instructions in an AssessmentItem, having contextual information available in the information model can improve the effectiveness and efficiency of TemplateProcessing and ResponseProcessing instructions. Contextual information can include candidate information, test information, and system (or environmental) information. |
| I-16 | Time Limits | For an individual item, time limits (as set by the Session definition) are enforced by the delivery system (unless PNP information for specific candidates allows for additional time). |
| I-17 | Composite Items | Support for multiple interactions within a single item. |
| I-18 | MathML | Support for MathML v2.0 and v3.0 |
| I-19a | QTI Shared Vocabulary Subset | A subset of the classes used in the QTI Shared Vocabulary for interactions used in Entry Level Certification. These classes include:
ChoiceInteraction: qti-labels-noneqti-labels-decimal qti-labels-lower-alpha qti-labels-upper-alpha qti-labels-suffix-noneqti-labels-suffix-periodqti-labels-suffix-parenthesis qti-input-control-hidden TextEntryInteraction: qti-input-width-1qti-input-width-2qti-input-width-3qti-input-width-4qti-input-width-6qti-input-width-10qti-input-width-15qti-input-width-20qti-input-width-72 ExtendedTextInteraction: qti-height-lines-3 qti-height-lines-6 qti-height-lines-15 |
| I-19b | QTI Shared Vocabulary Full | Support for the full set of shared vocabulary, including those interactions which are required for a particular level of certification. Implementers are NOT required to support shared vocabulary for interactions not included in the certification level they are seeking. See Appendix B of the QTI 3 Best Practices Implementation Guide for the complete list of QTI Shared Vocabulary. |
A.4 QTI 3.0 Interactions
| ID | Feature Name | Feature Description & Notes |
|---|---|---|
| Q-1 | Associate Interaction | An Associate Interaction (qti-associate-interaction) is a block interaction that
presents candidates with a number of choices and allows them to create associations
between them.
The “shuffle” attribute is not required for any certification level. |
| Q-2 | Choice Interaction | The choice interaction (qti-choice-interaction) presents a set of choices to the
candidate. The candidate's task is to select one or more of the choices, up to a
maximum of max-choices. The interaction is always initialized with no choices selected.
Though the max-choices is an optional attribute, it is required for Entry Level Certification.
The “shuffle” attribute and associated SimpleChoice “fixed” attribute are not required for any certification level. |
| Q-3 | Drawing Interaction | The drawing interaction (qti-drawing-interaction) allows the candidate to use a common set of drawing tools to modify a given graphical image (the canvas). It must be bound to a response variable with base-type file and single cardinality. The result is a file in the same format as the original image. |
| Q-4 | End Attempt Interaction | The end attempt interaction (qti-end-attempt-interaction) is a special type of interaction which allows item authors to provide the candidate with control over the way in which the candidate terminates an attempt. The candidate can use the interaction to terminate the attempt (triggering response processing) immediately, typically to request a hint. |
| Q-5 | Extended Text Interaction | An Extended Text Interaction (extended-text-interaction) is a block interaction that allows the candidate to enter an extended amount of text. |
| Q-6 | Gap Match Interaction | A gap match interaction (qti-gap-match-interaction) is a block interaction that contains a number gaps that the candidate can fill from an associated set of choices. The candidate must be able to review the content with the gaps filled in context, as indicated by their choices. The GapMatchInteraction must be bound to a response variable with base-type directedPair and either single or multiple cardinality, depending on the number of gaps. The choices represent the source of the pairing and gaps the targets. Each gap can have at most one choice associated with it. The maximum occurrence of the choices is controlled by the match-max characteristic of GapChoiceInteraction. |
| Q-7 | Graphic Associate Interaction | A graphic associate interaction (qti-graphic-associate-interaction) is Te The delivery engine must clearly indicate all defined area(s) of the image. |
| Q-8 | Graphic Gap Match Interaction | A graphic gap match interaction (qti-graphic-gap-match-interaction) is a graphical interaction with a set of gaps that are defined as areas (hotspots) of the graphic image and an additional set of gap choices that are defined outside the image. The candidate must associate the gap choices with the gaps in the image and be able to review the image with the gaps filled in context, as indicated by their choices. |
| Q-9 | Graphic Order Interaction | A graphic order interaction (qti-graphic-order-interaction) is a graphic interaction with a corresponding set of choices that are defined as areas of the graphic image. The candidate's task is to impose an ordering on the areas (hotspots). |
| Q-10 | Hot-spot Interaction | A hotspot interaction (qti-hotspot-interaction) is a graphical interaction with a corresponding set of choices that are defined as areas of the graphic image. The candidate's task is to select one or more of the areas (hotspots). |
| Q-11 | Hot-text Interaction | The hot text interaction (qti-hottext-interaction) presents a set of choices to the candidate represented as selectable runs of text embedded within a surrounding context, such as a simple passage of text. Like qti-choice-interaction, the candidate's task is to select one or more of the choices, up to a maximum of max-choices. Must support the HotTextInteraction.Type class and the HotText.Type class. |
| Q-12 | Inline Choice Interaction | An inline choice interaction (qti-inline-choice-interaction) is an inline interaction that presents the user with a set of choices, each of which is a simple piece of text. The candidate's task is to select one of the choices. Unlike the qti-choice-interaction, the delivery engine must allow the candidate to review their choice within the context of the surrounding text. |
| Q-13 | Match Interaction | A match interaction (qti-match-interaction) is a block interaction that presents candidates with two sets of choices and allows them to create associates between pairs of choices in the two sets, but not between pairs of choices in the same set. Further restrictions can still be placed on the allowable associations using the match-max characteristic of the choices. The qti-match-interaction must be bound to a response variable with base-type directedPair and either single or multiple cardinality. |
| Q-14 | Media Interaction | The media interaction (qti-media-interaction) allows more control over the way the candidate interacts with a time-based media object and allows the number of times the media object was experienced to be reported in the value of the associated response variable. |
| Q-15 | Order Interaction | In an order interaction (qti-order-interaction) the candidate's task is to reorder the choices, the order in which the choices are displayed initially is significant. By default the candidate's task is to order all of the choices but a subset of the choices can be requested using the max-choices and min-choices attributes. When specified the candidate must select a subset of the choices and impose an ordering on them. |
| Q-16 | Portable Custom Interaction (PCI) | The portable custom interaction provides an opportunity for extensibility of this specification to include support for interactions not currently documented. The “custom interaction” is considered deprecated and replaced by use of Portable Custom Interaction (PCI) . |
| Q-17 | Position Object Interaction | The position object interaction (qti-position-object-interaction) consists of a single image which must be positioned on another graphic image (the stage) by the candidate. Like qti-select-point-interaction, the associated response may have an AreaMapping that scores the response on the basis of comparing it against predefined areas but the delivery engine must not indicate these areas of the stage. Only the actual position(s) selected by the candidate shall be indicated. |
| Q-18 | Selection Point Interaction | The selection point interaction (qti-select-point-interaction) is a graphic interaction. The candidate's task is to select one or more points. The associated response may have an AreaMapping that scores the response on the basis of comparing it against predefined areas but the delivery engine must not indicate these areas of the image. Only the actual point(s) selected by the candidate shall be indicated. |
| Q-19 | Slider Interaction | The slider interaction (qti-slider-interaction) presents the candidate with a control for selecting a numerical value between a lower and upper bound. It must be bound to a response variable with single cardinality with a base-type of either integer or float. |
| Q-20 | Text Entry Interaction | A text entry interaction (qti-text-entry-interaction) is an InlineInteraction that obtains a simple piece of text from the candidate. Like qti-inline-choice-interaction, the delivery engine must allow the candidate to review their choice within the context of the surrounding text. |
| Q-21 | Upload Interaction | The upload interaction (qti-upload-interaction) allows the candidate to upload a pre-prepared file representing their response. |
| Q-31 | Custom Interaction (proprietary) | The custom interaction (qti-custom-interaction) provides an opportunity for extensibility of this specification to include support for interactions not currently documented, using methods that are proprietary for the supplier. The use of this interaction is deprecated in favor of qti-portable-custom-interaction |
| Q-32 | Custom Interaction (LTI) | The LTI custom interaction provides an opportunity for extensibility of this specification to include support for interactions not currently documented, by use of LTI. |
A.5 QTI 3.0 Accessibility
| ID | Feature Name | Feature Description & Notes |
|---|---|---|
| A-1 | Alternate text for graphics | All graphics that could be presented to candidates that are not for decorative-only purposes have a text alternative provided. Text for <img> is provided in the alt attribute. |
| A-2a | ARIA Subset | Support for the following ARIA features for all content that could be delivered to candidates, proctors, scorers, and tutors:aria-describedby, aria-hidden, aria-label, aria-labelledby, aria-live, role |
| A-2b | ARIA Full | Support for all the ARIA 1.0, 1.1 and 1.2 features for all content that could be delivered to candidates, proctors, scorers, and tutors. |
| A-3a | Inline Namespaced SSML | Support for namespaced SSML 1.1 markup within all content that could be delivered to candidates. |
| A-3b | Inline SSML using JSON markup | Support for SSML 1.1 vocabulary using JSON markup within all content that could be delivered to candidates. |
| A-4a | Internationalization (bi-directional text) | Support for bi-directional text and content |
| A-4b | Internationalization (Ruby) | Support for Ruby markup |
| A-5 | additional-directions | When item content contains additional-directions within a catalog, that content is presented to candidates who have the support listed in their PNP. |
| A-6 | additional-testing-time | Candidates who have additional-testing-time in their PNP will be allowed the amount of additional time requested to finish the assessment. |
| A-7 | alternative-text (PNP assignment) | If a candidate is specifically assigned to “alternative-text” in their PNP, the expectation during an assessment session is that the candidate will have the option to have the alternative text presented visually. The candidate must have the ability to switch between the visual presentation of the alternative text, and the programmatically available (but visually hidden) alternative text content. By default, alternative text is required to be visually hidden but programmatically available to candidates. Alternative text can be provided in the <img> alt attribute or aria-label attribute (for any element). |
| A-8 | answer-masking | By default, choices for items with choice interactions are covered (visually hidden) when the item is first presented to the candidate. The candidate has the ability to remove the mask(s) at a time of their choosing. This feature is generally used as a cognitive support, as a test-taking strategy, to enable candidates to think of the correct response before being presented with the answer choices. |
| A-9 | audio-description | Narration added to a soundtrack to describe important visual details that cannot be understood from the main soundtrack alone (from WCAG 2.0 http://www.w3.org/TR/WCAG20/#audiodescdef ). Audio-descriptions can be useful for video-only media, or synchronized media (video & audio) content. Candidates with "audio-description" in their PNP indicate a user's preference that they have access to audio descriptions for media containing video. |
| A-10a | braille | Candidates who have braille in their PNP have a preference that content be available to them in braille form. The PNP allows for setting the specific delivery-mode of the braille support, using either the "refreshable" or "embossed" vocabularies. Refreshable is the default value. While refreshable braille displays should use the default content of the assessment, encoded braille files can be associated with all or portions of the content using catalog resources and making the file(s) available to the candidate. |
| A-10b | braille preferences (delivery-mode, grade, braille-type, math-type) | For candidates who have supplied these optional braille parameters in their PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, although the candidate is still capable of modifying the settings during the assessment session. |
| A-11 | breaks | For candidates who have breaks in their PNP, they can pause an assessment to take a break, and return to the assessment without losing any response data, or incurring scoring penalties. |
| A-12 | calculator-on-screen | Candidates who have calculator-on-screen must have access to the calculator type specified in their PNP, even when an assessment would normally prohibit the use of a calculator. |
| A-13a | Captions: PNP | Candidates who have captions in their PNP would be presented captions for videos that have captions. |
| A-13b | Captions: track | Content supplies a captions track in the video or audio element. Delivery systems ensure the captions track is available to candidates. |
| A-15 | Glossary (on-screen) | An on-screen feature which allows a user to see definitions for designated words or phrases. The word or phrase will have some form of highlighting to indicate a glossary term that can be clicked to reveal the definition. When item content contains glossary-on-screen content within a catalog, that content is presented to candidates who have the support listed in their PNP or where the program allows access to glossary terms to all candidates. |
| A-16 | hazard-avoidance | If a candidate has a specific hazard-avoidance listed in their PNP, the candidate must not be exposed to content with those characteristics. |
| A-17 | high-contrast | When item content contains high-contrast content within a catalog (usually as an alternative to an image), that content is presented to candidates who have the support listed in their PNP. |
| A-18 | Homophone-checker (on-screen) | A resource that allows the user to understand the difference between words that share the same pronunciation regardless of how they are spelled. Candidates with homophone-checker in their PNP would access to an on-screen homophone checking tool. |
| A-19 | input-requirements | The requirements for the user to fully operate a computer interface using only their prefered input method (i.e., keyboard, mouse, etc.). For candidates who have input-requirements in their PNP, they must be able to access content using the input-requirement specified. It does not mean the user cannot use other methods of input (the interface must not prevent these other methods), but that the interface must have full control of the interface using the method provided in their PNP. |
| A-20 | invert-display-polarity | The values (the dark and light aspect) of the default presentation colors are inverted in the user interface. White is black, black is white. The purpose of inverting the values is to display text as light text over a dark background. While some inversion tools also reverse the polarity of the color hue (red to green, blue to yellow, etc.), the color shift is not necessarily needed or desired for the display of text. For candidates who have invert-polarity-display in their PNP, the user must have the ability to invert the polarity of the device display. In practice, the user may also have to capability to switch between the inverted and default display colors of the content. |
| A-21 | item-translation | A translated version (a variant) of the assessment content is presented to the user in place of, or in addition to, the primary language of the content. For candidates with item-translation listed in their PNP, assessment content is available to them in the language specified. |
| A-22 | keyboard-directions | When content contains keyboard-directions within a catalog, that content is presented to candidates who have the support listed in their PNP. |
| A-23 | keyword-emphasis | When content uses the special class “qti-keyword-emphasis”, any text within that class is presented with additional text emphasis to candidates who have the support listed in their PNP. |
| A-24 | keyword-translation | Certain specific words would have language translations available to users who need some assistance with difficult or important words in the content. When content contains keyword-translation content within a catalog, that content is presented to candidates who have the support listed in their PNP. |
| A-25 | language-of-interface | The language used on the interface elements (navigation buttons, labels, page headings, etc.), as differentiated from the language of content within the interface. For example, in an assessment, the interface could be in English, while the assessment questions and directions are in Spanish. For candidates with language-of-interface listed in their PNP, the assessment user interface is available to them in the language specified. |
| A-26 | layout-single-column | The delivery system has the capability of presenting content in a single column of text (when possible) for candidates who have the layout-single-column support in their PNP. |
| A-27 | line-reader | A tool that allows the user to move a marker (a colored bar or an underscore) up or down text, line by line, to assist in reading the content. Candidates who have line-reader in their PNP must have access to a line-reader tool in the assessment user interface during the assessment session. |
| A-28 | linguistic-guidance | When content contains linguistic-guidance within a catalog, that content is presented to candidates who have the support listed in their PNP. |
| A-29 | long-description | When content contains long description(s) for object/images. |
| A-30a | Magnification | Content on a computer device's screen is magnified by the amount specified by the user. This is distinct from enlarging a font size or providing a static large print version. Candidates who have magnification in their PNP must have access to a magnification controls in the assessment user interface during the assessment session. |
| A-30b | Magnification Zoom-amount | For candidates who have supplied zoom-amount values within the text, non-text, and all-content values for magnification in the PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, although the candidate is still capable of modifying the settings during the assessment session. |
| A-31 | Note-taking (on-screen) | A feature providing the ability to take notes in a virtual notepad while working on another activity on the computer. Candidates who have note-taking-on-screen in their PNP must have access to an on-screen note-taking tool in the assessment user interface during the assessment session. |
| A-32 | outliner-on-screen | A feature providing the ability to easily create an outline of ideas (commonly a hierarchical list) while working on another activity on the computer. Candidates who have outliner-on-screen in their PNP must have access to an on-screen outliner tool in the assessment user interface during the assessment session. |
| A-33 | Peer-interaction (on-screen) | Candidates who have peer-interaction-on-screen in their PNP must have access to an on-screen peer interaction tool in the assessment user interface during the assessment session. |
| A-34 | Prohibit set | Within a candidate’s PNP, a set of supports that are specifically not available to the candidate in the user interface during an assessment session. |
| A-35 | Pronunciation Lexicon Specification (PLS) | For candidates who have indicated the spoken support in their PNP, any PLS files provided in the qti-external-supplemental-accessibility node are used to inform the text-to-speech engine how to pronounce words. |
| A-36 | sign-language | When content contains sign-language within a catalog, that content is presented to candidates who have the support listed in their PNP, in the sign language specified. In the context of a web-based presentation of content, sign language is generally presented as simultaneous content in a video alongside the default visually presented content (including text in a written language). The video playback must be controlled by the user. |
| A-37 | simplified-graphics | When content contains simplified-graphics within a catalog (usually as an alternative to a default image), that content is presented to candidates who have the support listed in their PNP. |
| A-38 | simplified-language-portions | When content contains simplified-language-portions within a catalog, that content is available for presentation to candidates who have the support listed in their PNP. |
| A-39 | Spell-checker (on-screen) | Candidates who have spell-checker in their PNP must have access to an on-screen spell-checking tool in the assessment user interface during the assessment session. |
| A-40a | Spoken (reading-type = screen-reader) | Candidates who have the spoken support in their PNP with a reading-type of “screen-reader” must be able to use a screen-reader during the assessment session, and content within the assessment is expected to be perceivable, understandable, and usable. |
| A-40b | Spoken (reading-type = computer-read-aloud) | Candidates who have the spoken support in their PNP with a reading-type of “computer-read-aloud” must have access to a computer interface that reads assessment content text aloud during the assessment session. |
| A-40c | Spoken Restriction Type | Candidates who have the spoken support in their PNP with a specific restriction-type must not have the restricted content read out loud to them during the assessment session. |
| A-40d | Spoken, user preferences for speech-rate, pitch, volume, and link-indication | For candidates who have supplied these optional spoken user preference parameters in their PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, though the candidate is still capable of modifying the settings during the assessment session. |
| A-40e | Spoken (typing-echo) | For candidates who have the spoken support in their PNP, the delivery system automatically sets the candidate preferences when the assessment session is initiated, though the candidate is still capable of modifying the settings during the assessment session. |
| A-40f | Text-to-Speech Suppression | For candidates who have the spoken support in their PNP, content that is within an element that contains the attribute “data-qti-suppress-tts” must not be read aloud to the candidate based on their reading-type and the vocabulary provided in the attribute. |
| A-41 | Tactile |
Term: An object that can be felt with the hands.
QTI 3.0: A tactile representation of the graphic information is made available outside of the user's computer. PNP: The term "tactile" indicates that the user must have access to tactile materials when available, and must be presented with any tactile tour content that describes the tactile materials. A tactile tour is text content that provides information that orients the user about the tactile object. When item content contains tactile qti-html-content within a catalog, that content is presented to candidates who have the support listed in their PNP. |
| A-42a | Text-appearance, text and background color | For candidates with text-appearance in their PNP, the delivery system allows the user to at least change the text color and the color of the text background. This feature may also be available to all candidates regardless of their PNP. |
| A-42b | Text-appearance, other | For candidates with text-appearance with any of the optional user preferences in their PNP, the delivery system presents the text content using the preferences supplied in the PNP. |
| A-43 | Thesaurus (on-screen) | A resource that allows the candidate to reference words of similar meaning. Candidates with thesaurus-on-screen in their PNP have access to an on-screen thesaurus tool. |
| A-44 | transcript | A nonsynchronous written alternative for time-based media content, which includes
dialogue (including the speaker name/role) and descriptions of sounds and/or important
information that is presented visually. When item content contains transcript content
within a catalog, that content is presented to candidates who have the transcript
support listed in their PNP.
PNP: The term "transcript" indicates that the user must have access to transcript content if available in a catalog resource. |
| A-45 | Visual-organizer (on-screen) | A tool that allows people to organize their thoughts by constructing a visual representation of their ideas. Candidates with thesaurus-on-screen in their PNP would access to an on-screen thesaurus tool. |
| A-46 | Activate-as-option | The term "activate-as-option-set" is a set of user preferences in a PNP. Assigning a feature or support to this set indicates that each feature must be made available to the user as an option to use, but the feature is not activated (the feature isn't actively in use) until requested by the user. A method for making the feature available to the user must be obvious, like (but not limited to) a button on a toolbar, an item in a drop down menu, or an option in an options window. |
| A-47 | Activate-at-initialization | The term "activate-at-initialization-set" is a set of user preferences in a PNP. Assigning a feature or support to this set indicates that each feature must be made available to the user as soon as the user initiates their working session (generally after logging into a content presentation system). |
A.6 QTI Metadata
| ID | Feature Name | Feature Description |
|---|---|---|
| M-1 | Item Template | An AssessmentItem that contains a TemplateProcessing section. The value is true if the item is actually an item template, in other words, the item changes its appearance based on some random or external factor. |
| M-2 | Time Dependent | A time dependent item takes the length of time taken for an attempt into consideration when scoring. |
| M-3 | Composite | Value is True if the item comprises more than one interaction, for example, an AssessmentItem that contains more than one interaction. |
| M-4 | Interaction Type | The interaction type(s) of the item. The vocabulary is comprised of the names, as defined in the information model, of the leaf classes derived from interaction. |
| M-5 | Feedback Type | Describes the type of feedback, if any, available in the item. If feedback is available then it is described as being non-adaptive or adaptive depending on whether the item is itself adaptive. A non-adaptive item generates feedback based on the responses submitted as part of (the last) attempt only. An adaptive item generates feedback that takes into consideration the path taken through the item, in other words, feedback based on the accumulation of all attempts and not just the last. |
| M-6 | Solution Available | Set to the value to “true” if a model solution is available for the item. For example, an AssessmentItem that provides correct responses for all declared response variables. |
| M-7 | Tool Name | The name of the tool used to author the evaluation object. |
| M-8 | Tool Version | The version of the tool used to author the evaluation object. |
| M-9 | Tool Vendor | The company which produced the tool used to author the evaluation object. |
| M-10 | Scoring Mode | Specifies the scoring mode: human, external machine, or response processing. |
A.7 Content Packaging
|
|
Feature Name | Feature Description |
|---|---|---|
| P-1 | Test Instances | The package contains test (assessment) level structures. |
| P-2 | Nested Section Instances | The package contains nested section level structures. |
| P-3 | Item Banks | Item bank structure. |
| P-4 | Item Instances | The package contains qti-assessment-item instances. |
| P-5 | Manifest Metadata (LOM) | Information about the package as a whole is based upon the 1EdTech binding of IEEE Learning Object Metadata (LOM). |
| P-6 | Resource Metadata (LOM) | Information about an Item, Section, Test, etc. is described as part of the resources specifically and is based upon the 1EdTech binding of IEEE Learning Object Metadata (LOM). |
| P-7 | Resource Metadata (QTI) | Information about an Item, Section, Test, etc. is described as part of the resources specifically and is based upon the 1EdTech QTI-specific metadata. |
| P-8 | Curriculum Standards Metadata | Learning Standards annotations for the resources and the manifest as a whole is supported using the 1EdTech Curriculum Standards Metadata (CSM) approach developed as part of 1EdTech Common Cartridge [CC-13]. These annotations are added in the corresponding metadata structures for the manifest and resource i.e. in the same manner as for the LOM entries. When using CSM the following XML should be added to the manifest element. |
| P-9 | External Metadata | Reference to a metadata file that is maintained and stored outside of the package. |
| P-10 | Learning Tools Interoperability v1.1 | The 1EdTech Learning Tools Interoperability (LTI) standard prescribes a way to easily and securely connect learning applications and tools with platforms, portals, learning environments, etc. LTI could be used to launch an assessment, some external tool or resource, or to specific points in assessment content. |
| P-11 | Stimulus Instances | The package contains a stimulus file, and the manifest references the stimulus. |
| P-12 | Variant | This is a feature in the 1EdTech CPv1.2 that is used to contain alternative resource information. A resource can have many variants (for example, an English variant, a Spanish variant, and a Chinese Variant). In QTI 3.0, the variant structure is used to contain the metadata that describes the conditions under which the alternative forms of an item are available to be used. Variants for an item are packaged together in a single item content package. |
| P-13 | CASE | Competencies and Academic Standards Exchange (1EdTech CASE v1.0). Used to associate item, section, test level learning and education competencies that can be exchanged with competency systems or frameworks. |
B. Revision History
This section is non-normative.
B.1 Version History
| Version No. | Release Date | Comments |
|---|---|---|
| Final Release 1.0 | 15 March 2022 | The first final release of this document as part of the QTI 3.0 specification set. |
| Final Release 2.0 | 1 June 2023 | Revisions made to account for changes in QTI 3.0 Certification, as approved by the Assessment Product Steering Committee in March 2023. |
C. References
C.1 Normative references
- [CC-13]
- IMS Global Common Cartridge v1.3. IMS Global Learning Consortium. July 2013. IMS Final Release. URL: http://www.imsglobal.org/cc/index.html
- [RFC2119]
- Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. IETF. March 1997. Best Current Practice. URL: https://www.rfc-editor.org/rfc/rfc2119
C.2 Informative references
- [QTI-OVIEW-30]
- Question & Test Interoperability (QTI) 3.0: Overview. Mark Hakkinen; Padraig O'hiceadha; Mike Powell; Tom Hoffman; Colin Smythe. IMS Global Learning Consortium. May 2022. IMS Final Release. URL: https://www.imsglobal.org/spec/qti/v3p0/oview/
D. List of Contributors
The following individuals contributed to the development of this document:
| Name | Organization | Role |
|---|---|---|
| Arjan Aarnink | Cito | |
| Vijay Ambati | ACT, Inc. | |
| Jérôme Bogaerts | Open Assessment Technologies | |
| Shiva Bojjawar | McGraw-Hill Education | |
| Catriona Buhayar | NWEA | |
| Jason Carlson | ACT, Inc. | |
| Jason Craft | Pearson | |
| Rich Dyck | Data Recognition Corporation | |
| Paul Grudnitski | Independent Invited Expert | |
| Mark Hakkinen | ETS | Co-chair |
| Susan Haught | 1EdTech Global | |
| Thomas Hoffmann | 1EdTech Global | Editor |
| Rob Howard | NWEA | |
| Stephen Kacsmark | Instructure | |
| Justin Marks | NWEA | |
| Amy Marrich | Open Assessment Technologies | |
| Mark McKell | Pearson | |
| Mark Molenaar | Apenutmize | |
| Padraig O'hiceadha | Houghton Mifflin Harcourt | Co-chair |
| Mike Powell | Pearson | Co-chair |
| Julien Sebire | Open Assessment Technologies | |
| Colin Smythe | 1EdTech Global | Editor |
| Tjeerd Hans Terpstra | Cito | |
| Travis Thompson | Data Recognition Corporation | |
| Wyatt Vanderstucken | ETS | |
| Jason White | ETS |