Primer on IMS Assessment Conformance and Certification
Date Issued: 1 May 2012
IPR and Distribution Notices
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.
IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS’s procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: /imsipr_policyFinal.pdf.
Copyright © 2012 IMS Global Learning Consortium. All Rights Reserved.
Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: /speclicense.html.
Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.
The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.
THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.
Public contributions, comments and questions can be posted here: APIP Forum.
IMS is establishing well-defined conformance certification programs for Question and Test Interoperability (QTI) v2.1 and Accessible Portable Item Protocol (APIP) v1.0 to ensure high levels of interoperability. The conformance certification programs test for the ability to support different levels of features.
The purpose of this document is to introduce the specific areas of conformance available to APIP and QTI and to describe the procedures for achieving certification. APIP and QTI are standards created and governed by IMS Global Learning Consortium (IMS Global), a non-profit learning standards organization. This document is intended to be used, for example, by vendors that wish to certify their systems as officially conforming to APIP v1.0 and QTI v2.1 as supported by IMS. Ultimately, certification of systems will help ensure that diverse end-users (students) receive access to computer-based assessments.
1.1 IMS APIP
IMS APIP is a technical standard for digital interchange of assessment content (items and tests). APIP also describes digital delivery options for a range of common accessibility needs. Use of APIP enables assessment materials to be exchanged digitally among a wide variety of products, such as item/test authoring products, item banks, and test delivery systems.
APIP addresses two long-standing needs in assessments. First, it allows for the transfer of content between vendors using a standardized XML format. Second, it supplies the necessary accessibility information in that content to support the needs of all students. Using IMS Access For All Personal Needs and Preferences (PNP) profile information, the content can be tailored to meet the access needs of each student.
The APIP standard defines an assessment file exchange that serves all students, not only students who have disabilities or are English language learners. In fact, the default content of the question (item) is based on a well established IMS QTI standard that has been used internationally for over a decade. APIP uses a new profile of QTI v2.1.
Figure 1.1 illustrates the basic idea of APIP. Authors create content, providing extra information when needed to make that content accessible to all students. An assessment rostering system can provide the specific needs of individual students; these are documented in the PNP profile. An APIP certified delivery system takes the needs of a specific student, and ensures that the content is provided to the student with all their needed access supports.
Figure 1.1 The Basic Concept of APIP.
The three main areas of APIP are:
· Content: The accessible content, which has supporting information for different kinds of accessibility needs
· PNP: The user Personal Needs & Preferences profile, which tells the delivery system what the needs are for specific examinees
· Delivery: The delivery system, which combines the user needs with the content to ensure the item is accessible for the examinee
1.2 IMS QTI
The QTI v2.1 specification describes a data model for the representation of question (assessmentItem) and test (assessmentTest) data and their corresponding results reports. The specification enables the exchange of item, test, and results data between authoring tools, item banks, test constructional tools, learning systems, and assessment delivery systems. The specification has been designed to support both interoperability and innovation through the provision of well-defined extension points.
1.3 IMS Assessment Community
To join the IMS developer and conformance certification community focused on APIP and QTI v2.1 please visit the IMS QTI2/APIP Alliance online here: /developers/apipalliance/index.cfm
Public contributions, comments and questions can be posted here: APIP Forum.
2 Conformance and Certification Overview
2.1 Relationship between APIP and QTI
APIP uses a precisely defined subset of IMS QTI v2.1 with additional features included in some other IMS specifications. The first step in implementing APIP is to implement the APIP defined subset of QTI v2.1. This will allow assessment suppliers to utilize QTI when accessibility is not a requirement and to receive a QTI conformance certification.
APIP and the profiles of QTI used in APIP will stay synchronized by IMS Global in conjunction with IMS member organizations through the QTI/APIP Accredited Profile Management Group (APMG). The QTI subset within APIP, such as interaction types (question types), is expected to grow over time as the marketplace for digital assessment evolves.
The additional features of APIP beyond QTI are accessibility metadata (based on IMS Access For All), accessible delivery options (developed on the original APIP grant program) and curriculum standards metadata (based on IMS Content Packaging/Common Cartridge). These features are defined in the APIP specification.
2.2 Conformance Certifications
IMS is establishing well-defined conformance certifications for QTI v2.1 and APIP v1.0 to ensure high levels of interoperability. The conformance certification process involves testing for the ability to support different levels of features. The conformance details are described in Section 3.
2.3 How Conformance Certification is Achieved and Recorded
Conformance certification is granted by passing a series of tests, submitting the results to IMS and agreeing to become part of the support community (the QTI/APIP Alliance /apip/index.html). IMS records all certifications on a public website: Conformance Certification Status. At the time conformance is certified, IMS issues a unique registration number designator for the product name and version for use in proposals. Buyers and users are encouraged to visit the certification website to find certified products and to log any issues with certified products.
The community aspect of conformance is critical to address any issues that may arise in the marketplace as well as the evolution to new versions. IMS requires suppliers to participate in resolving issues that may arise, as well as helping to guide and implement each incremental revision to the conformance certifications.
2.4 Achieving Conformance Certification
The tests required for certification are different depending on the particular QTI/APIP certification sought. The tests are also different for content/authoring tools versus delivery platforms. This is because content/authoring tools are tested to ensure that the content produced and exported is conformant within the scope of item types and tests that a particular tool can generate, while delivery systems are tested to ensure they can appropriately process a range of content alternatives. Conformant delivery platforms must import conformant content and provide conformant results. Conformant authoring tools must export conformant content and may or may not import conformant content, depending on the purpose of the tool.
3 Assessment Conformance Statements
The following conformance matrix lists the current profiles available for QTI and APIP certification as they relate to these categories:
QTI Delivery System & Authoring System
The QTI conformance criteria for Delivery Systems and Authoring Systems.
APIP Authoring System
The APIP conformance criteria for Authoring Systems and APIP packages.
APIP Delivery System
The APIP conformance criteria for Delivery Systems.
APIP Personal Needs & Preferences (PNP) Systems
The APIP PNP conformance criteria for systems supplying or processing PNP instances.
3.1 Conformance Profiles
Named profiles currently defined for conformance are described here as they appear in the matrix below.
QTIv2.1 Summative Entry
Conformance for Delivery Systems that process Tests used for 'Summative' assessment (this is equivalent to APIP Entry minus the accessibility features).
QTIv2.1 Formative Entry*
Conformance for Delivery Systems that process Tests based upon QTIv2.1 (this is equivalent to the Common Cartridge profile for QTIv2.1).
Conformance for Delivery Systems that process Tests based upon QTIv2.1 (this is equivalent to the Common Cartridge profile but with all of the prohibited features made Elective).
The entry level APIP conformance (this is equivalent to the Common Cartridge profile). Entry level conformance also permits the set of implementation Electives.
This is the level of conformance that IMS recommends should be adopted by APIP systems.
APIP PNP Entry
This is the set of entry level APIP PNP requirements for a Preferences System producing or processing PNP instances.
APIP PNP Core
This is the level of conformance to which IMS recommends that all APIP PNP systems should aspire.
Click here to go to the most up to date version of the QTI/APIP Conformance Matrix: QTI/APIP Conformance Matrix.
Elective access features are those features that could be supported by systems, but are not required for conformance. Elective features will be individually certified, and systems that use elective features must list the specific APIP Elective access features they support. Either Entry or Core certified systems are eligible to be certified in any elective feature. There is no Elective certification without Entry or Core certification.
The certification categories include a differentiation between the ability to import and export APIP Content or PNP files. Since APIP is primarily a transfer format, it will not certify how systems manage or modify data within the system itself. Delivery Systems are expected to combine the information provided by APIP content packages and PNP files. Although they do not need to natively use the APIP format during delivery, certified delivery systems must make use of the data/information supplied by APIP.
Systems that import APIP information must distinguish between valid APIP formats and invalid formats, and provide information about which packages are invalid. Systems will minimally support APIP Entry access features, and possibly support APIP Core and APIP Elective access features.
Systems that export APIP content or PNP files must be able to supply valid APIP formatted instances, minimally supporting APIP Entry access features, and possibly supporting APIP Core and APIP Elective access features.
4 Achieving Certification
Validation for APIP and QTI conformance will consist of the following strategies:
· Automated Validation. An electronic process to verify conformance using automated tools provided by the IMS APMG, and
· Test Case Based Validation. Using a standard set of APIP/QTI content packages and PNP files with known characteristics provided by the IMS APMG that can be used as part of a validation process and have known expected results that can be documented and/or demonstrated, and
· Documented or Demonstrable Validation. Documented statements and/or demonstrable capabilities showing conformance that can be reviewed by the IMS APMG or by the user.
Because the APIP standard is primarily a data transfer format, conformance and certification does not govern how applications manage or modify data within the system itself. For example, delivery systems must be able to combine the information provided by imported content and PNP files, but need not natively use the APIP format during delivery. There are limits to what can be verified in an automated fashion. In the categories listed in the prior section as an example, it makes a clear distinction between import and export. The reason for this is that export functions are much more easily verified using automation. The output from an export process must adhere to the APIP XML specifications and can be verified using automation. Import, on the other hand, is much more difficult to verify because the import process results in a proprietary system consuming the information and representing it in an internal format that does not need to adhere to the XML specification and cannot be verified using automation. Each of these approaches is discussed in detail below.
4.1 Automated Validation
Capabilities to be validated using automated tools:
· APIP/QTI Content Package export
· APIP PNP file export
For the export functional context, the APIPv1 conformance approach focuses on determining if the system can generate an object (PNP file or item) that is non-problematic. IMS provides an online tool available for individuals or organization to submit an object (PNP file, item) and then receive information about the existence (and sometimes nature) of problems with syntax and semantics of the object.
It is expected that the automated validation tool will:
· Accept Entry, Core, and Elective APIP/QTI conformance content elements identified in the conformance matrix.
· Accept custom (and profiled) extensions that are agreed to between exchanging parties (interdependent vendors).
· Identify which Core and Elective access features as well as the QTI v2.1 features are implemented in the supplied instance.
4.2 Test Case Based Validation
Capabilities to be validated using a standard set of test cases:
· APIP/QTI Content Package import
· APIP PNP file import
The QTI/APIP Alliance makes available sets of objects (PNP files, items) — some that are problematic and others that are non-problematic — for conducting import validation. Import functionality focuses on determining if the system can (a) flag as problematic objects that are known to have problems in syntax or semantics and (b) recognize as non-problematic the objects that are known to be without problems.
4.3 Documented or Demonstrable Validation
Capabilities to be documented and/or demonstrated by the solution provider:
- Render APIP content, supporting differences in PNP file information
As each test delivery platform may provide a unique user experience and use different technologies to deliver to content, providing automated validation of each implementation is virtually impossible. Therefore, the IMS APMG will rely on the solution provider to document that the required APIP content and PNP features have been implemented and can document and/or demonstrate them based on user expectations. The documents and demo systems can be submitted to the APMG for consideration and review but the APMG will not be able verify the exact system behaviors.
About this Document
Title: Primer on IMS Assessment Conformance and Certification
Editors: Colin Smythe (IMS Global) and Mark McKell (IMS Global)
Version Date: 1 May 2012
Release: Draft 1.0
Summary: This document introduces the different areas of conformance available to APIP and QTI and describes the procedures for achieving certification.
Revision Information: First release
Document Location: /apip/IMSAssessmentPrimerv1p0.html
IMS Global Learning Consortium, Inc. (“IMS Global”) is publishing the information contained in this document (“Specification”) for purposes of scientific, experimental, and scholarly collaboration only.
IMS Global makes no warranty or representation regarding the accuracy or completeness of the Specification.
This material is provided on an “As Is” and “As Available” basis.
The Specification is at all times subject to change and revision without notice.
It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.
IMS Global would appreciate receiving your comments and suggestions.
Please contact IMS Global through our website at imsglobal.org.
Please refer to Document Name: Primer on IMS Assessment Conformance and Certification v1.0
Date: 1 May 2012