1EdTech Question & Test Interoperability: An Overview Final Specification Version 1.2 |
Copyright © 2002 1EdTech Consortium, Inc. All Rights Reserved. The 1EdTech Logo is a trademark of 1EdTech Consortium, Inc. Document Name: 1EdTech Question & Test Interoperability: An Overview Date: 11 February 2002
|
IPR and Distribution Notices
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.
1EdTech takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on 1EdTech's procedures with respect to rights in 1EdTech specifications can be found at the 1EdTech Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.
Copyright © 2002 1EdTech Consortium. All Rights Reserved.
Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.
Use of this specification to develop products or services is governed by the license with 1EdTech found on the 1EdTech website: http://www.imsglobal.org/license.html.
The limited permissions granted above are perpetual and will not be revoked by 1EdTech or its successors or assigns.
THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.
Table of Contents
1. The Question & Test Interoperability Specification
1.1 Historical Perspective
1.2 The Requirements
1.3 Key Terminology
1.4 The Documents
1.4.1 Documentation Overview
1.4.2 QTI Overview
1.4.3 The ASI Information Model
1.4.4 The ASI XML Binding Document
1.4.5 The ASI Best Practices & Implementation Guide
1.4.6 The ASI Selection & Ordering Specification
1.4.7 The ASI Outcomes Processing Specification
1.4.8 The Results Reporting Information Model
1.4.9 The Results Reporting XML Binding Document
1.4.10 The Results Reporting Best Practices & Implementation Guide
1.4.11 The QTILite Specification
1.5 The QTI Structures
1.6 Using the 1EdTech QTI Specification
Bibliography
Appendix A - Glossary of Terms
About This Document
List of Contributors
1. The Question & Test Interoperability Specification
The 1EdTech Question & Test Interoperability (QTI) specification describes a basic structure for the representation of question (item) and test (assessment) data and their corresponding results reports. Therefore, the specification enables the exchange of this item, assessment and results data between Learning Management Systems, as well as content authors and, content libraries and collections. The 1EdTech QTI specification is defined in XML to promote the widest possible adoption. XML is a powerful, flexible, industry standard markup language used to encode data models for Internet-enabled and distributed applications. The 1EdTech QTI specification is extensible and customisable to permit immediate adoption, even in specialized or proprietary systems. Leading suppliers and consumers of learning products, services and content contributed time and expertise to produce this final specification. The 1EdTech QTI specification, like all 1EdTech specifications, does not limit product designs by specifying user interfaces, pedagogical paradigms, or establishing technology or policies that constrain innovation, interoperability, or reuse.
1.1 Historical Perspective
An initial V0.5 specification was released for discussion in March 1999 and the corresponding Base Document was agreed in November 1999. The first Public Draft Specification was released in February 2000 and the 1EdTech Question & Test Interoperability v1.0 specifications were released in their final form in May 2000. A version 1.01 update was released in August 2000. During the development of these specifications and their subsequent adoption by the community, several areas of further work were identified and the '1.x' versions of the 1EdTech QTI Specifications were scoped to address these issues1. Version 1.1 was released in March 2001 and contained the introduction of the QTILite specification. Results reporting was introduced in V1.2, released in January 2002, as was the concept of 'selection & ordering' and 'outcomes processing'. To-date, in excess of 6000 copies of the 1EdTech QTI V1.0/V1.01/V1.1/1.2 specifications have been downloaded from the 1EdTech web-site.
1.2 The Requirements
The 1EdTech QTI Working Group's work specifically relates to content providers (that is, question and test creators), virtual learning environment and tool vendors, and question/test users (that is, learners and teachers or trainers needing assessment tools). The targeted markets include primary and secondary education, community, junior and vocational colleges, higher education, and commercial and military training. The 1EdTech QTI specifications are intended to meet international needs as well. Therefore the 1EdTech QTI Working Group was focussed on enabling the following functionality:
- The ability to provide question/item banks to users regardless of virtual learning environment (VLE) deployed by the user;
- The ability to use question/item banks from various sources within a single VLE;
- Support for tools to develop new question/item banks in a consistent manner;
- The ability to report test results in a consistent manner.
Consequently, the following requirements have been suggested and are presented in order of priority:
- Definition of standardised attributes (question meta-data) for questions, choices, feedback/branch, scoring, meta-data along with identification of required and optional elements;
- Interoperability of question/item banks - definition for packaging and distribution;
- Extended schema for results reporting;
- Extended schema for assessment, tracking and presentation;
- APIs for dynamic interface into question retrieval and scoring/assessment engines.
It is also considered essential that the specification allows for extensibility and flexibility based on yet unidentified future needs and necessitated by specific customised implementations.
1.3 Key Terminology
Despite its name, the 1EdTech QTI specification details more than how to tag questions, tests and results. The standard Question types e.g. multiple choice, fill in the blank, or true/false choice, etc. can be constructed using a core set of presentation and response structures, and results of questions can be collected and scored by using a variety of methods. To represent these options, the 1EdTech QTI specification defines the 'Item'. Items contain all the necessary data elements required to compose, render, score and provide feedback from questions. Therefore, the key difference between a 'Question' and 'Item' is that an 'Item' contains the 'Question', layout rendering information, the associated response processing information, and the corresponding hints, solutions and feedback.
Similarly, the 'test' is an instance of an Assessment. Assessments are assembled from Items that are contained within a 'Section' to resemble a traditional test. Additionally, Assessments might be assembled from blocks of Items that are logically related. These groups are also defined as 'Sections' and so Assessments are composed of one or more Sections which themselves are composed of Items, or more Sections. Collectively, these three data objects are referred to as the ASI (Assessment, Section, Item) structures. These evaluation objects can be bundled together to create an object bank. This object bank can then be externally referenced and used as a single evaluation object. To avoid limitations associated with words like user, student, or learner the 1EdTech QTI working group adopted the term 'participant' to refer to the person interacting with an assessment. Thus, the key definitions are:
- Item - A combination of interrogatory, rendering, and scoring information;
- Section - A collection of zero or more items and/or other Sections;
- Assessment - A collection of one or more Sections;
- Object Bank - A group of Items and/or Sections that have been bundled e.g. to create an Item-bank;
- Participant - The user interacting with an assessment.
When constructing the results report for an Assessment, a Section or an Item, a similar structure is used. A results report can contain either a summary of the results themselves and/or the detailed set of results with respect to the assessment, section(s) and item(s). Each results report is contained within its own package that also describes the context for the evaluation e.g. participant identifier, etc.
1.4 The Documents
1.4.1 Documentation Overview
The QTI specification is comprised of ten separate documents. Different documents have particular relevance to the set of released version of the QTI specification, as shown in Table 1.1.
'*' denotes the documents that were produced for each version of the 1EdTech QTI specifications.
The technical structure of the 1EdTech QTI specification is based upon two components:
- The ASI components that are used to describe the actual evaluation objects to be presented to the participants;
- The results reporting objects that are used to contain the results to be reported for an evaluation undertaken by a participant.
It is important to stress that the two components of the 1EdTech QTI specification do not have to be used together. The results reporting part of the specification can be used to contain results obtained from an evaluation based upon QTI ASI but this is not a prerequisite. This means that either one or both of the parts of the QYI specification can be used as appropriate.
1.4.2 QTI Overview
The role of the QTI Overview (this document) is to provide a brief description of the full 1EdTech QTI specification. This is the only document that brings together all of the other documents to provide a perspective on the full specification.
1.4.3 The ASI Information Model
The QTI ASI Information Model document is comprised of several sections. The first section contains use cases in which the underlying usage, processing control, and core data structures of the QTI ASI specification are described. It also details the taxonomy of responses, as well as their relationship to questions type and the larger group of 'items'. The basic information model itself is outlined in conceptual terms by using a tabular layout of the Assessment, Section, and Item objects in terms of their elements, sub-elements and attributes. The Item, Section, and Assessment meta-data, which are used to catalogue these objects, are also described. In addition, the document contains a conformance statement to be used by vendors who plan to implement the specification; we have adopted a descriptive approach to conformance thereby enabling vendors to implement subsets of the full specification.
1.4.4 The ASI XML Binding Document
The XML Binding document describes the implementation of the ASI information model in XML. XML is introduced by outlining XML basics, including a conceptual discussion of the XML schema. The XML schema description of the QTI specification (ims_qtiasiv1p2.xsd and ims_qtiasiv1p2.dtd) defines the Assessment, Section, and Item as XML elements. An example schema for Assessments, Sections, and Items is included, along with details of the meta-data used to catalogue Assessment, Sections, and Items. Some of the XML Binding documents also include, as appendices, a copy of the uncommented XSD, as well as the uncommented DTD and XDR (XDR document is a Microsoft Corporation XML schema implementation)2.
1.4.5 The ASI Best Practices & Implementation Guide
This document is intended to provide vendors with an overall understanding of the ASI specification, the relationship of this specification with other 1EdTech specifications, and a best practices guide derived from experiences of those using the specification.3 Example Item types supported by the specification, examples of composite Item types, and a complete XML example for presenting an Assessment, Section, and Item is included. The Best Practices & Implementation Guide also includes a significant number of actual examples that describe how vendors can make the best use of the 1EdTech QTI specification. These examples, approximately eighty, are also useful as a starting template for each of the different forms of Assessment, Section and Item. Appendices provide the range of available DTDs, XDRs and XSDs (as appropriate), as well as a glossary of the XML structures used throughout the specification.
1.4.6 The ASI Selection & Ordering Specification
The 'Selection & Ordering' specification contains the description of how the order in which Sections and/or Items are presented can be controlled. The selection and ordering process is a two-stage operation in which the child objects are selected according to some defined criteria e.g. meta-data content, etc. and the order of their presentation is then determined. The selection and ordering process within an object is limited to the immediate children of the object and so complex requirements must be based upon the appropriate usage of Sections to contain the Section/Item hierarchies. This document contains the relevant information model, XML binding and best practices guidance but it should be read in the context of the core ASI documents.
1.4.7 The ASI Outcomes Processing Specification
The 'Outcomes Processing' specification contains the description of how the aggregated scores at the Assessment and Section levels can be derived. These scoring outcomes are based upon the child Sections and/or Items. Several scoring algorithms are supported (sum-of-scores, number correct, guessing penalty and best-K-ofN) through the usage of a predefined set of parameterised instructions; these avoid the realisation of the algorithms within the XML. This document contains the relevant information model, XML binding and best practices guidance but it should be read in the context of the core ASI documents. Each of the supported scoring algorithms in explained in an Appendix.
1.4.8 The Results Reporting Information Model
The 1EdTech QTI Results Reporting Information Model document is comprised of several sections. The first section contains use cases in which the underlying usage, processing control, and core data structures of the results to be reported are described. The basic information model itself is outlined in conceptual terms by using a tabular layout of the context, summary, Assessment results, Section results, and Item results objects in terms of their elements, sub-elements and attributes. The corresponding meta-data, which are used to catalogue these objects, are also described. In addition, the document contains a conformance statement to be used by vendors who plan to implement the specification; we have adopted a descriptive approach to conformance thereby enabling vendors to implement subsets of the full specification.
1.4.9 The Results Reporting XML Binding Document
The XML Binding document describes the implementation of the Results Reporting information model in XML. XML is introduced by outlining XML basics, including a conceptual discussion of the XML schema. The XML schema description of the QTI specification (ims_qtiresv1p2.xsd and ims_qtiresv1p2.dtd) defines the results report objects as XML elements. An example schema is included, along with details of the meta-data used to catalogue the results report.
1.4.10 The Results Reporting Best Practices & Implementation Guide
This document is intended to provide vendors with an overall understanding of the results reporting specification, the relationship of this specification with other 1EdTech specifications (including the QTI ASI), and a best practices guide derived from experiences of those using the specification. Example results reports are included along with a demonstration of how SCORMv1.2 results reporting can be supported. The Best Practices & Implementation Guide also includes a significant number of actual examples that describe how vendors can make the best use of the results reporting specification. These examples are also useful as a starting template for each of the different forms of results report. Appendices provide the range of available DTDs and XSDs (as appropriate), as well as a glossary of the XML structures used throughout the specification.
1.4.11 The QTILite Specification
This document describes the components that are required to construct the simplest form of an 1EdTech QTI-compliant system. QTILite supports multiple-choice questions (this includes the true/false questions) only and limits the rendering form to the classical one response from a set of choices. Multiple Items can be exchanged in a single QTI-XML instance but Assessments and Sections are not supported. The QTILite specification is a standalone document in that none of the others are required to understand and construct QTILite-compliant systems. All QTILite compliant Items are compliant with the full 1EdTech QTI V1.1 and V1.2 specifications but they are not backwards compatible with V1.0 or 1.01 of the specification. QTILite was introduced as an aide to understanding the QTI specification and is not intended for wide-spread long-term adoption.
1.5 The QTI Structures
The core data structures that can be exchanged using 1EdTech QTI are shown schematically in Figures 1.1 and 1.2. The four ASI structures are:
- Item(s) - one of more Items can be contained within a QTI-XML instance. The Item is the smallest independent unit that can be exchanged using 1EdTech QTI. An Item cannot be composed of other Items. An Item is more than a 'Question' in that it contains the 'Question', the presentation/rendering instructions, the response processing to be applied to the participant's response(s), the feedback that may be presented (including hints and solutions) and the meta-data describing the Item;
- Section(s) - one or more Sections can be contained within a QTI-XML instance. A Section can contain any mixture of Sections and/or Items. A Section is used to support two different needs:-
- Assessment - only one Assessment can be contained within a QTI-XML instance. It is not possible to define relationships between the Assessments. Each Assessment must contain at least one Section, thus it is not possible to have Items housed directly within an Assessment. The Assessment contains all of the necessary instructions to enable variable sequencing of the Items and the corresponding aggregated scoring for all of the Items to produce the final score;
- Object bank - the bundling together a collection of data objects (sections and/or Items) can be labelled as an object bank i.e. a group of Items can be grouped to produce an Item-bank.
While the definition of an Item and Assessment is well established it must be stressed that the 'Section' is merely a grouping construct. This allows any level of grouping of the Items and/or Sections. What these Sections actually mean in an assessment environment is dependent on the ways in which the contents are to be used.
The core Results Reporting data structures are:
- Result - the set of results relevant to an actual attempt of an assessment or some other form of evaluation. Multiple instances for a single participant can be contained within the 1EdTech QTI Results Reporting package;
- Context - the contextual information concerning the actual result being reported e.g. the name, of the participant, participant identifiers, etc;
- Summary_result - the summary information for a particular instance of the evaluation. Each result can contain only one set of summary information;
- Assessment_result - the detailed assessment information for a particular attempt at the assessment. Each result can contain information about one assessment only (this includes descriptions of any contained section and items);
- Section_result - the detailed information about the section(s) completed, or to being attempted. Each result can contain information about one section (this includes descriptions of any contained sections and/or items);
- Item_result - the detailed information about the item(s) completed, or to being attempted. Each result can contain information about one item.
The results reporting components are capable of containing the results from all of the components of an Assessment, Section and Item. This means that it is simple to use 1EdTech QTI Results Reporting XML binding to reports the results obtained from an evaluation based upon 1EdTech QTI ASI.
1.6 Using the 1EdTech QTI Specification
Users wishing to adopt the 1EdTech QTI specifications are advised to start with either the QTILite specification or the Best Practice & Implementation Guide documents. These documents contain extensive examples. All of these examples are available as part of the 1EdTech QTI toolkit and as such they make excellent templates. Several versions of the XSDs, DTDs and XDRs exist in terms of file structure (IBM, Unix, MacOS) and functional complexity (QTILite; Item-only; Items and Sections; core elements i.e. excluding the extension and V1.x/V2.0 features, the full uncommented; the full commented version). Beginners should focus on the QTILite and Item-only versions. The fully commented version should be avoided unless a documented version of the schema is required. There are core three 1EdTech QTI XML DTD/XSDs:
- 'ims_qtiasiv1p2.dtd' and 'ims_qtiasiv1p2.xsd';
- 'ims_qtiresv1p2.dtd' and 'ims_qtiresv1p2.xsd';
- 'ims_qtilitev1p2.dtd.
The 1EdTech QTI specification includes its own 1EdTech QTI-specific meta-data features but the 1EdTech Meta-data can also be used. The 1EdTech Content Packaging specification incorporates the 1EdTech QTI specification as a native structure i.e. the Assessment, Section and Item XML can be contained within a Content Package's XML. It is recommended that the 1EdTech Content Packaging specification mechanism is used for the physical exchange of the QTI-XML instances.
Bibliography
Appendix A - Glossary of Terms
About This Document
Title | 1EdTech Question & Test Interoperability: An Overview |
Editors | Colin Smythe, Eric Shepherd, Lane Brewer, and Steve Lay |
Version | 1.2 |
Version Date | 11 February 2002 |
Status | Final Specification |
Summary | This document describes the 1EdTech Question & Test Interoperability specification. It contains an overview of the full specification and explains the relationship between the different components of the specification. |
Revision Information | 22 January 2002 |
Purpose | To provide an overview of the 1EdTech Question & Test Interoperability specification. |
Document Location | http://www.imsglobal.org/question/v1p2/imsqti_oviewv1p2.html |
List of Contributors
The following individuals contributed to the development of this document:
1EdTech Consortium, Inc. ("1EdTech") is publishing the information contained in this 1EdTech Question & Test Interoperability: An Overview ("Specification") for purposes of scientific, experimental, and scholarly collaboration only.
1EdTech makes no warranty or representation regarding the accuracy or completeness of the Specification.
This material is provided on an "As Is" and "As Available" basis.
The Specification is at all times subject to change and revision without notice.
It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.
1EdTech would appreciate receiving your comments and suggestions.
Please contact 1EdTech through our website at http://www.imsglobal.org
Please refer to Document Name: 1EdTech Question & Test Interoperability: An Overview Date: 11 February 2002