IMS Logo IMS Question & Test Interoperability: Results Reporting Information Model
Final Specification Version 1.2
Copyright © 2002 IMS Global Learning Consortium, Inc. All Rights Reserved.
The IMS Logo is a trademark of IMS Global Learning Consortium, Inc.
Document Name: IMS Question & Test Interoperability: Results Reporting Information Model
Date: 11 February 2002

 

IPR and Distribution Notices

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS's procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Copyright © 2002 IMS Global Learning Consortium. All Rights Reserved.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/license.html.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.


 

Table of Contents


1. Introduction
     1.1 Question & Test Interoperability Overview
     1.2 Scope & Context
     1.3 Structure of this Document
     1.4 Nomenclature
     1.5 References

2. Specification Use-Cases
     2.1 Basic Architecture
     2.2 Results Aggregation System
     2.3 Processed Results for a Learning Management System
     2.4 Real-time Interactive
     2.5 Internal Interchange Data Language Representation

3. Basic Information Model

4. Conceptual Description of the Data Objects
     4.1 Underlying Structure of the Results Report Package
     4.2 Extensions & Extensibility
     4.3 QTI Results Report Package Tabular Description
           4.3.1 QTI Results Reporting Data Objects
           4.3.2 'summary_result' Data Objects
           4.3.3 'assessment_result' Data Objects
           4.3.4 'section_result' Data Objects
           4.3.5 'item_result' Data Objects
           4.3.6 Common Data Objects
           4.3.7 'extension' Definitions

5. IMS Supported QTI Vocabularies & Taxonomies

6. Meta-data Description
     6.1 IMS Meta-data
     6.2 QTI-Specific Meta-data

7. Conformance
     7.1 Valid Data Issues
     7.2 Conformance Summary
     7.3 Interoperability Statement

About This Document
     List of Contributors

Revision History

Index


1. Introduction

1.1 Question & Test Interoperability Overview

The Question & Test Interoperability (QTI) specification describes a basic structure for the representation of question (item) and test (assessment) data and their corresponding results reports [QTI, 02i]. Therefore, the specification enables the exchange of this item, assessment and results data between Learning Management Systems, as well as content authors and, content libraries and collections. The QTI specification is defined in XML to promote the widest possible adoption. XML is a powerful, flexible, industry standard markup language used to encode data models for Internet-enabled and distributed applications. The QTI specification is extensible and customizable to permit immediate adoption, even in specialized or proprietary systems. Leading suppliers and consumers of learning products, services and content contributed time and expertise to produce this final specification. The QTI specification, like all IMS specifications, does not limit product designs by specifying user interfaces, pedagogical paradigms, or establishing technology or policies that constrain innovation, interoperability, or reuse.

This document is the IMS QTI Results Reporting Information Model. The first section contains use cases in which the underlying usage, processing control, and core data structures of the results to be reported are described. The basic information model itself is outlined in conceptual terms by using a tabular layout of the context, summary, Assessment results, Section results, and Item results objects in terms of their elements, sub-elements and attributes. The corresponding meta-data, which are used to catalogue these objects, are also described. In addition, the document contains a conformance statement to be used by vendors who plan to implement the specification; we have adopted a descriptive approach to conformance thereby enabling vendors to implement subsets of the full specification.

1.2 Scope & Context

This document is the IMS Question & Test Interoperability: Results Reporting Information Model V1.2 Final Specification document. As such it will be used as the basis for the development of the following documents:

  • IMS QTI: Results Reporting XML Binding v1.2 [QTI, 02f];
  • IMS QTI: Results Reporting Best Practice & Implementation Guide v1.2 [QTI, 02g].

This requirement has been derived from the agreed IMS Q&TI V1.x Scoping document [QTI, 00]. This Information Model complements the IMS QTI: ASI Information model [QTI, 02a] but it can be applied to assessments that have not been represented using that specification.

1.3 Structure of this Document

The structure of this document is:

2. Specification Use-cases The definition and scoping of the results that are to be reported using this specification;
3. Basic Information Model The underlying QTI Results Reporting information model;
4. Conceptual Description of the Data Objects The detailed description of the QTI Results Reporting data objects in terms of their elements, sub-elements and attributes;
5. IMS Supported QTI Vocabularies & Taxonomies The definition and description of the vocabularies and taxonomies that are supported as default by IMS;
6. Meta-data Description The usage of IMS Meta-data and QTI Results Reporting specific meta-data;
7. Conformance The definition of the conformance statement to be used by vendors.

1.4 Nomenclature

AMS Assessment Management System
API Application Programming Interface
ASI Assessment, Section, Item
CBT Computer Based Training
DTD Document Type Definition
LMS Learning Management System
QTI Question & Test Interoperability
VLE Virtual Learning Environment
W3C World Wide Web Consortium
XML Extensible Mark-up Language
XSD XML Schema

1.5 References

[ETS, 99] A Sample Assessment Using the Four Process Framework, R.Almond, L.Steinberg and R.Mislevy, ETS Working Paper, October 1998.
[IMS, 01a] IMS Persistent Location-independent Resource Identifier Implementation Handbook, V1.0, M,McKell, IMS Specification, May, 2001.
[QTI, 00] IMS Question & Test Interoperability Version 1.x Scoping Statement, C.Smythe and E.Shepherd, Version 1.0, IMS, November 2000.
[QTI, 02a] IMS Question & Test Interoperability ASI Information Model Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02b] IMS Question & Test Interoperability: ASI XML Binding Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02c] IMS Question & Test Interoperability: ASI Best Practice & Implementation Guide, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02d] IMS Question & Test Interoperability: ASI Outcomes Processing Specification, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02e] IMS Question & Test Interoperability: ASI Selection & Ordering Specification, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02f] IMS Question & Test Interoperability: Results Reporting XML Binding, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02g] IMS Question & Test Interoperability: Results Reporting Best Practice & Implementation Guide, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02h] IMS Question & Test Interoperability: QTILite Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02i] IMS Question & Test Interoperability: Overview, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.

2. Specification Use-Cases

2.1 Basic Architecture

The basic architectural model for the QTI V1.2 specification is shown in Figure 2.1.

Basic architectural model for QTI V1.2

 

Figure 2.1 Basic architectural model for QTI V1.2.

In this architecture the ASI repository was introduced in QTI V1.0. This is still supported in QTI V1.2 but the Results repository is now added. The QTI Results Reporting specification is designed to support the following use-cases:

  • Permanent storage - the long-term storage of the results in a neutral format;
  • Results aggregation - the collation of a set of results from one or more assessment delivery systems and the transfer of those to a results analysis system. This may or may not include the packaging of results from many participants with one or more evaluations per participant. The key point is that the final results may be based upon the collation of date from many different assessment systems;
  • Processed results for a Learning Management System (LMS) - the exchange of results between an LMS and the assessment delivery system. This is usually on an individual assessment and a per participant basis. The exchange can also be interactive in that the LMS may respond to the results by launching further assessment and/or content;
  • Real-time interactive - the real-time exchange of information concerning the assessment delivery system and the assessment control engine (this may or may not be part of an LMS). The interactive nature means that the delivery system reports the results of a participant's responses etc. which the control engine then processes to determine the suitable reaction e.g. delivery of feedback, delivery of remedial content, etc;
  • Internal Interchange Data Language Representation - to act as the internal data representation for a system that consists of a set of communicating processes. This enables portability between different implementations of the processes.

2.2 Results Aggregation System

Assessments may be delivered by various systems such as local area networks, intranets, the Internet or even paper. The results from these assessments need to be collated to accurately compare the results between the various delivery methods. The overall architecture of such a system is shown in Figure 2.2.

Results aggregation use-case

 

Figure 2.2 Results aggregation use-case.

In such a system the QTI Results Reporting specification can be used to support the exchange of the information between the 'QTI Results Data' repositories and the 'Results Repository'. These results may be:

  • For a single assessment from a single participant - to be contained in a single QTI Results Report structure. The assessment information could be about a single topic, a full high-stake assessment, etc;
  • For several assessments for a single participant - to be contained in a single QTI Results Report structure;
  • For several assessments from more than one participant - to be contained in multiple QTI Results Report structures and consequently some form of corresponding packaging is required.

2.3 Processed Results for a Learning Management System

A common application is for an LMS to trigger another system to deliver content and collect results. Raw results are not very useful but a score, or accumulation of scores is useful. In this situation an Assessment Management System will determine the scores, and accumulation of scores, and potentially a grade ("A+", "A", "Pass", "Fail", etc). These scores and grades need to be passed back to the LMS. The architecture for such a system is shown in Figure 2.3.

Processed results for an LMS use-case

 

Figure 2.3 Processed results for an LMS use-case.

In such a system the QTI Results Reporting specification can be used to support the exchange of the information between the LMS and the AMS. It should be stressed that the QTI specification defines the interface data model for the exchange and not the behavior required on those models. The realization of this interface could involve the appropriate API.

2.4 Real-time Interactive

Certain systems in the marketplace require the raw answers from an item delivery system to allow it to analyze and provide a score, descriptive and/or prescriptive feedback. For example an essay grading system (results analysis system) might accept an essay from an item delivery system, grade the essay, generate feedback and pass this back to the item delivery system. Another example is several items are delivered by the item delivery system, on various topics, and the results analysis system generates specific topic scores and feedback. The architecture for such a system is shown in Figure 2.4.

Raw answers for a results analysis use-case

 

Figure 2.4 Raw answers for a results analysis use-case.

In such a system the IMS QTI Results Reporting specification can be used to support the exchange of the information between the 'Results Analysis System' and the 'Item Delivery System'. It should be stressed that the QTI specification defines the interface data model for the exchange and not the behavior required on those models. The realization of this interface could involve the appropriate API.

2.5 Internal Interchange Data Language Representation

In some cases the QTI specification may be adopted to support the internal representation of the results information. This representation is then used for the exchange of data between processes within a single system. An example of such an approach is shown in Figure 2.5 - this is based upon the four-process model created by ETS [ETS, 99].

Internal data language representation use-case

 

Figure 2.5 Internal data language representation use-case.

In such a system the QTI Results Reporting specification can be used to support the exchange of the information between the 'Item Response Processing' and Assessment/section Processing' processes. It could also be used for sub-process communication within each of these macro processes.

3. Basic Information Model

The underlying logical data structures for the QTI Results Reporting package are shown in Figure 3.1. Figure 3.1 shows the four core structures that can be exchanged using the QTI Results Reporting specification.

The principal QTI results reporting data structures

 

Figure 3.1 The principal QTI results reporting data structures.

The core data structures are:

  • result - the set of results relevant to an actual attempt of an assessment or some other form of evaluation. Multiple results for a single or multiple participants can be contained within the QTI Results Reporting package;
  • context - the contextual information concerning the actual evaluation being reported e.g. the name of the participant, participant identifiers, etc;
  • summary_result- the summary information for a particular instance of the evaluation. Each result can contain only one set of summary information;
  • assessment_result - the detailed assessment information for a particular attempt at the assessment. Each result can contain information about one assessment only (including any contained sections and items);
  • section_result - the detailed information about the section(s) completed, or to being attempted. Each result can contain information about one section (including any contained sections and/or items);
  • item_result - the detailed information about the item(s) completed, or to being attempted. Each result can contain information about one item.

This structure has been adopted so that:

  • Both summary or detailed results information can be exchanged depending on which is required. The aim is to minimize the overheads required to capture the key data. This means that the different use-cases are supported in an optimal manner;
  • The relationship between the IMS QTI ASI information model is immediately obvious. The Results Reporting specification can be used to support assessments etc. that have not be based upon the IMS QTI ASI model however this will require some form of mapping to the QTI representational model and/or the use of the appropriate extensions;
  • As a compromise between a representation that is neither too abstract nor too implementation specific. The underlying model supports extension mechanisms in a several areas.

4. Conceptual Description of the Data Objects

4.1 Underlying Structure of the Results Report Package

The primary elements of the learner information are shown in Figure 4.1:

The primary elements of the QTI results reporting data structures

 

Figure 4.1 The primary elements of the QTI results reporting data structures.

4.2 Extensions & Extensibility

A key requirement for the specification is its support, where appropriate, for extensions. These extensions take two forms:

  • Vocabulary extensions - extensions that allow the basic vocabularies to be extended. These vocabularies are assigned to many of the data objects and they are used to define the type of information being contained. The basic vocabularies are defined within a set of default files maintained by IMS. The vocabulary extensions may be included in a string list or referenced using a URI;
  • Functional extensions - extensions that are included to ensure that users of the specification can add functionality that is otherwise excluded from the specification (in the following tabular descriptions these are denoted by the 'extension' data structure). Further versions of the specification make full commitment to ensure backwards compatibility with these features. Within the XML binding each of these extensions will be given a unique element name.

The process by which the two forms of extension fit within the taxonomy is clearly denoted in the IMS QTI Results Reporting XML Binding [QTI, 02f].

4.3 QTI Results Report Package Tabular Description

The tables in this Section provide a conceptual, informative description of the elements in the data objects. The columns in these tables refer to:

No: The number of the data element. An element may be composed of sub-elements. The numbering scheme reflects these relationships.
Name: The descriptive name of the element.
Explanation: A brief functional description of the element.
Required: Indicates if the element is required:
  • M = Mandatory Element that must be included in the data object, if the element at the higher level is included;
  • C = Conditional Element. Existence is dependent on values of other Elements;
  • O = Optional Element.
Multi: Multiplicity of the element:
  • Blank = single instance;
  • Number = maximum number of times the element is repeatable;
  • n = multiple occurrences allowed, no limit;
  • Repeatability of an element implies that all sub-elements repeat with the same element.
Type: A description of formatting rules for the data element. Type includes the maximum length of the element:
  • ID = element used to uniquely identify an object;
  • Code = element value from a list of codes;
  • Description = descriptive element, human language
  • Flag = binary flag
  • Enumerated = list of predefined non-numeric options i.e. the definitive list of objects
  • The international character set specified by ISO 10646 will be used for all fields.

The type will also include a description of the set of valid values for the sub-element:

  • Coding schemes using numerical values;
  • The set of values as defined in the Domain i.e. making it closed. The list of values cannot be extended to include values not defined in the specification. If there is a need for values not included in the domain set of values then the extension should be done defining a new element under the Extension element that is a part of each data object definition.

Note: Additional descriptive information about the element.

In the following tables the data objects are organized as:

  • Table 4.1 - the QTI Results Reporting data structure i.e. the structure that constitutes the report package itself;
  • Table 4.2 - the 'summary_result' data structure;
  • Table 4.3 - the 'assessment_result' data structure;
  • Table 4.4 - the 'section_result' data structure;
  • Table 4.5 - the 'item_result' data structure;
  • Table 4.6 - the common data structures (these are used within more than one of the above data structures);
  • Table 4.7 - the functional extensions, <extension>, that are supported.

4.3.1 QTI Results Reporting Data Objects

Table 4.1 describes the data objects that are used in the construction of the QTI results reporting package itself.

 

Table 4.1 QTI results reporting data objects detailed description.

 
No
Name
Explanation
Reqd
Mult
Type
Note
1.1 qti_comment A comment about the full QTI results report structure. As per structure 6.1 (Table 4.6).
1.2 result The container for the a set of results for a particular participant in a particular context M n
 
Reports of multiple evaluations use multiple result instances.
1.2.1 qti_comment A comment about the context of the evaluation. As per structure 6.1 (Table 4.6).
1.2.2 context Contains the information about the participant that sets the context for the evaluation being reported. M
 

 
A context is mapped on a one-to-one relationship with an evaluation.
1.2.2.1 qti_comment A comment about the context of the evaluation. As per structure 6.1 (Table 4.6).
1.2.2.2 name The detailed formatted name of the participant. O
 
#PCDATA String.
1-256 chars.

 
1.2.2.3 generic_identifier The set of identifiers that are used to define the participant and their evaluation environment. Each identifier is vocabulary based. O n As per structure 6.4 (Table 4.6).
1.2.2.4 date The set of dates that are relevant to the context of the evaluation e.g. the creation date of the report. Each date is vocabulary based. O n As per structure 6.5 (Table 4.6).
1.2.2.5 extension The extension facility for the 'context' data structure. O
 
As per structure 7.1 (Table 4.7).
1.2.3 summary_result The set of summary information concerning a particular assessment completed by a participant. C
 
As per Table 4.2.
1.2.4 assessment_result The set of detailed results information for a particular assessment. An assessment is as defined within the QTI specifications.
 

 
As per Table 4.3.
1.2.5 sectionresult The set of detailed results information for the section. A section is as defined within the QTI specifications.
 

 
As per Table 4.4.
1.2.6 item_result The set of detailed results information for the item. An assessment is as defined within the QTI specifications.
 

 
As per Table 4.5.
1.2.7 extension The extension facility for the 'instance' data structure.
 

 
As per structure 7.2 (Table 4.7).

4.3.2 'summary_result' Data Objects

Table 4.1 describes the data objects that are used in the construction of the summary results information.

 

Table 4.2 'summary_result' data objects detailed description.

 
No
Name
Explanation
Reqd
Mult
Type
Note
2.1 asi_title The title of the evaluation being reported. This would be identical to that assigned under the QTI ASI spec. O
 
As per structure 6.6 (Table 4.6).
2.2 qti_comment As per structure 6.1 (Table 4.6).
2.3 type_label The type of summary results being reported e.g. an assessment, item, etc.
 

 
As per structure 6.2 (Table 4.6).
2.4 generic_identifier The generic identifiers that are allocated to different aspects of the summary information. O n As per structure 6.4 (Table 4.6).
2.5 date The dates that are relevant to the summary e.g. the date of creation. O n As per structure 6.5 (Table 4.6).
2.6 status The status of the evaluation e.g. completed, in-progress, etc. O
 
As per structure 6.11 (Table 4.6).
2.7 duration The time taken for the completion of the evaluation. O
 
As per structure 6.12 (Table 4.6).
2.8 score The score assigned to the evaluation using the default variable i.e. 'SCORE', 'CORRECT' or 'GRADE'. O
 
As per structure 6.13 (Table 4.6).
2.9 grade The grade that has been awarded to the participant e.g. 'Pass', 'Fail', etc. O
 
As per structure 6.14 (Table 4.6).
2.10 outcomes The data structure that contains the extended set of outcome results. O n As per structure 6.15 (Table 4.6).
2.11 extension The extension facility to the <summary> data structure. O
 
As per structure 7.3 (Table 4.7).

4.3.3 'assessment_result' Data Objects

Table 4.3 describes the data objects that are used in the construction of the detailed assessment results.

 

Table 4.3 'assessment_result' data objects detailed description.

 
No
Name
Explanation
Reqd
Mult
Type
Note
3.1 asi_title The title of the assessment being reported. This would be identical to that assigned under the QTI ASI spec. O
 
As per structure 6.6 (Table 4.6).
3.2 ident_ref The formal identifier associated with this assessment. This would be identical to that assigned under the QTI ASI spec. O
 
As per structure 6.7 (Table 4.6).
3.3 qti_comment As per structure 6.1 (Table 4.6).
3.4 asi_metadata The QTI- results reporting specific meta-data that is associated with this assessment. O n As per structure 6.8 (Table 4.6).
3.5 asi_description A brief description of the Assessment. The description can be contained in an external file. O
 
As per structure 6.22 (Table 4.6).
3.6 date The dates that are relevant to the assessment result record e.g. the date of creation. O n As per structure 6.5 (Table 4.6).
3.7 duration The time taken for the completion of the assessment. O
 
As per structure 6.12 (Table 4.6).
3.8 objective The set of objectives that are covered by this assessment for the type of participant(s). O n As per structure 6.9 (Table 4.6).
3.9 control The feedback mechanisms that were revealed to the participant(s) O
 
As per structure 6.10 (Table 4.6).
3.10 outcomes The data structure that contains all of the scores being reported as part of the assessment. O
 
As per structure 6.15 (Table 4.6).
3.11 feedback_displayed An indication of the type of feedback that was given to the participant O n As per structure 6.21 (Table 4.6).
3.12 num_items The accumulated number of Items that are related to this assessment O
 
As per structure 6.16 (Table 4.6).
3.13 num_sections The number of Sections that are directly related to the parent assessment O
 
As per structure 6.17 (Table 4.6).
3.14 num_items_presented The number of items presented to the participant during the assessment. O
 
As per structure 6.18 (Table 4.6).
3.15 num_items_attempted The number of items attempted by the participant during the assessment. O
 
As per structure 6.19 (Table 4.6).
3.16 num_sections_presented The number of sections presented to the participant during the assessment. O
 
As per structure 6.20 (Table 4.6).
3.17 section_result The set of detailed results information for each of the prescribed sections. A section is as defined within the QTI specifications. O n As per Table 4.3.
3.18 extension The extension facility for the assessment results. O
 
As per structure 7.4 (Table 4.7).

4.3.4 'section_result' Data Objects

Table 4.4 describes the data objects that are used in the construction of the detailed section results.

 

Table 4.4 'section_result' data objects detailed description.

 
No
Name
Explanation
Reqd
Mult
Type
Note
4.1 asi_title The title of the assessment being reported. This would be identical to that assigned under the QTI ASI specification. O
 
As per structure 6.6 (Table 4.6).
4.2 ident_ref The formal identifier associated with this section. This would be identical to that assigned under the QTI ASI specification. O
 
As per structure 6.7 (Table 4.6).
4.3 presented This is used to indicate whether or not the associated Section was presented to the participant. O
 
Enumerated:
Yes (default)
No
Unknown
This is required because the full structure may consist of nested Sections only some of which may be presented i.e. due any associated selection & ordering algorithm.
4.4 qti_comment As per structure 6.1 (Table 4.6).
4.5 asi_metadata The QTI- results reporting specific meta-data that is associated with this section. O n As per structure 6.8 (Table 4.6).
4.6 asi_description A brief description of the Section. The description can be contained in an external file. O
 
As per structure 6.22 (Table 4.6).
4.7 date The dates that are relevant to the section result record e.g. the date of creation. O n As per structure 6.5 (Table 4.6).
4.8 duration The time taken for the completion of the section. O
 
As per structure 6.12 (Table 4.6).
4.9 objective The set of objectives that are covered by this section for the type of participant(s). O n As per structure 6.9 (Table 4.6).
4.10 control The feedback mechanisms that were revealed to the participant(s) O
 
As per structure 6.10 (Table 4.6).
4.11 outcomes The data structure that contains all of the scores being reported as part of the section. O
 
As per structure 6.15 (Table 4.6).
4.12 feedback_displayed An indication of the type of feedback that was given to the participant O n As per structure 6.21 (Table 4.6).
4.13 num_items The accumulated number of Items that are related to this assessment O
 
As per structure 6.16 (Table 4.6).
4.14 num_sections The number of Sections that are directly related to the parent assessment O
 
As per structure 6.17 (Table 4.6).
4.15 num_items_presented The number of items presented to the participant during the section evaluation. O
 
As per structure 6.18 (Table 4.6).
4.16 num_items_attempted The number of items attempted by the participant during the section. O
 
As per structure 6.19 (Table 4.6).
4.17 num_sections_presented The number of sections presented to the participant during the assessment. O
 
As per structure 6.20 (Table 4.6).
4.18 section_result The set of detailed results information for each of the prescribed sections. A section is as defined within the QTI specifications. O n As per Table 4.3.
4.19 item_result The set of detailed results information for each of the prescribed items. An Item is as defined within the QTI specifications.
 

 
As per Table 4.4.
4.20 extension The extension facility for the section results. O
 
As per structure 7.5 (Table 4.7).

4.3.5 'item_result' Data Objects

Table 4.5 describes the data objects that are used in the construction of the QTI results reporting package itself.

 

Table 4.5 'item_result' data objects detailed description.

 
No
Name
Explanation
Reqd
Mult
Type
Note
5.1 asi_title The title of the item being reported. This would be identical to that assigned under the QTI ASI specification. O
 
As per structure 6.6 (Table 4.6).
5.2 ident_ref The formal identifier associated with this item. This would be identical to that assigned under the QTI ASI specification. O
 
As per structure 6.7 (Table 4.6).
5.3 presented This is used to indicate whether or not the associated Item was presented to the participant. O
 
Enumerated:
Yes (default)
No
Unknown
This is required because the full structure may consist of many Items only some of which may be presented i.e. due any associated selection & ordering algorithm.
5.3 qti_comment As per structure 6.1 (Table 4.6).
5.4 asi_metadata The QTI- results reporting specific meta-data that is associated with this item. O n As per structure 6.8 (Table 4.6).
5.5 asi_description A brief description of the Item. The description can be contained in an external file. O
 
As per structure 6.22 (Table 4.6).
5.6 date The dates that are relevant to the item result record e.g. the date of creation. O n As per structure 6.5 (Table 4.6).
5.7 duration The time taken for the completion of the section. O
 
As per structure 6.12 (Table 4.6).
5.8 objective The set of objectives that are covered by this item for the type of participant(s). O n As per structure 6.9 (Table 4.6).
5.9 control The feedback mechanisms that were revealed to the participant(s) O
 
As per structure 6.10 (Table 4.6).
5.10 response The response from the participant for a particular Item response block. M n
 

 
5.10.1 ident_ref The identifier for the response block as assigned by the 'ident' attribute under the QTI ASI specification. O
 
As per structure 6.7 (Table 4.6).
5.10.2 qti_comment As per structure 6.1 (Table 4.6).
5.10.3 responseform The information that describes the form of the response structure being presented. O
 

 

 
5.10.3.1 response_type The type of response that is expected from the participant. O
 
Enumerated list from:
'lid'
'xy',
'str',
'num',
'grp'
'extension'

 
5.10.3.2 cardinality The cardinality of the response block. O
 
Enumerated list from:
'single'
'multiple'
'ordered'
'extension'

 
5.10.3.3 render_type The type of rendering mechanism used. O
 
Enumerated list:
'choice'
'hotspot'
'slider'
'fib'
'extension'

 
5.10.3.4 timing The time-based dependence of the response. O
 
Enumerated list:
'Yes'
'No'

 
5.10.3.5 correct_response The correct answer for this response. This will take the form of the label for the identifier or the actual answer itself. O n #PCDATA String.
1-256 chars.

 
5.10.3.6 extension The extension facility for the description of the response structure. O
 
As per structure 7.7 (Table 4.7).
5.10.4 num_attempts The number of attempts made to answer the Item. O
 
#PCDATA String
1-2 chars.
The value '0' means that the question was not attempted.
5.10.5 response_value The sequenced responses given by the participant. O n #PCDATA String.
1-256 chars.
The order and number of responses matches those given by the participant. This information should match with the cardinality of the original Item.
5.10.5.1 response_time The time in seconds between the start of the evaluation event e.g. the start of the Item, and the entry of the response. O
 
#PCDATA String.
1-5 chars

 
5.10.5.2 response_status The status of the attempt to answer this Item. O
 
Enumerated list:
'Null'
'Valid'
'NA'
'Invalid'.
'Valid' is the default value.
If a valid attempt has been made to answer the Item then the status is 'Valid'. If no attempt was made then the status is 'NA'. If the no information is available then the status is 'Null'. An 'Invalid' status is returned to indicate a system failure condition.
5.10.5.3 uri The location of the object. C
 
CDATA String 256
As per RFC1630.
An alternative to this is the 'entityref' attribute cf. 5.10.5.4).
5.10.5.4 entityref The entity reference that is used to bind an external file to the XML instance.
 

 
CDATA string defining the link.
String 256.
This is an alternative to the 'uri' attribute (cf. 5.10.5.3).
5.10.5 extension The extension facility for the participant responses. O
 
As per structure 7.8 (Table 4.7).
5.11 outcomes The data structure of the scores being reported as part of the assessment. O
 
As per structure 6.15 (Table 4.6).
5.12 feedback_displayed The type of feedback that was presented to the participant. O n As per structure 6.21 (Table 4.6).
5.13 extension The extension facility for the item results. O
 
As per structure 7.6 (Table 4.7).

4.3.6 Common Data Objects

Table 4.6 describes the common data objects that are used to support the other data objects.

 

Table 4.6 Common data objects detailed description.

 
No
Name
Explanation
Reqd
Mult
Type
Note
6.1 qti_comment Comments of the QTI Results Reporting data objects. These comments are not removed by an XML parser. O
 
#PCDATA String
1-2048 chars.
These comments should be used to annotate the XML instances.
6.1.1 language_type The language that is being used for the information. As per structure 6.3.
6.2 type_label The label of the object that is being defined. O
 

 
The naming convention should reflect the usage of the name. See Section 5 for a further explanation of the vocabularies that must be supported.
6.2.1 source The source of the vocabulary. O
 
String
1-256 chars.
Contains the name of the external file containing vocabulary.
6.3 language_type The language that is being used for the information. This will be realized using 'xml:lang'. O
 
String
1-256 chars.
The language entries will be defined as per the ISO639 standard and RFC1766.
6.4 generic_identifier The generic identifier. O n
 
It is expected that this object will conform to the IMS Persistent Location-independent Resource Identifier handbook.
6.4.1 type_label The type of generic identifier as defined by the associated vocabulary. O
 
As per structure 6.2.
6.4.2 identifier_string A unique identifier for the data record assigned by the creating entity. M
 
#PCDATA
String
1-256 chars.
The unique identifier assigned by reference to the object. Can contain any character.
6.5 date The date and/or time to be recorded. O n
 

 
6.5.1 type_label The type of date as defined by the associated vocabulary. O
 
As per structure 6.2.
6.5.2 datetime The underlying string structure is: YYYY:MM:DDT
HH:MM:SS
M
 
#PCDATA String.
1-20 chars.
Defined as per ISO 8601.
6.6 asi_title The title of the evaluation being reported. This would be identical to that assigned under the QTI ASI specification. O
 
#PCDATA String.
1-256 chars.

 
6.7 ident_ref The formal identifier associated with this evaluation. This would be identical to that assigned under the QTI ASI specification. O
 
#PCDATA String.
1-256 chars.
This should conform to the IMS Persistent Location-independent Resource Identifier specification. Can contain any character.
6.8 asi_metadata The container for all of the vocabulary-based QTI-specific meta-data. O
 

 
Multiple vocabularies can be used for each set of meta-data fields.
6.8.1 vocabulary The vocabulary to be applied to the associated meta-data fields. O
 
#PCDATA
String 2048 chars.
If no file is used then the vocabulary can be contained as a comma delimited string.
6.8.1.1 uri The location of the object. C
 
CDATA
String 256 chars.
As per RFC1630.
An alternative to this is the 'entityref' attribute cf. 6.8.1.2).
6.8.1.2 entityref The entity reference that is used to bind an external file to the XML instance.
 

 
CDATA string defining the link.
String 256 chars.
This is an alternative to the 'uri' attribute (cf. 6.8.1.1)
6.8.1.3 vocab_type This is the type of vocabulary file that is identified in either the associated 'uri' or 'entityref' attributes. O
 
CDATA string type of file.
String 16 chars.

 
6.8.2 asi_metadatafield The QTI- results reporting specific meta-data that is associated with this evaluation. M n
 

 
6.8.2.1 language_type The language that is being used for the meta-data content. As per structure 6.3.
6.8.2.2 field_name The field label as taken from the possible vocabulary. M
 
#PCDATA
String 256 chars.

 
6.8.2.3 field_value The data for that field. M
 
#PCDATA
String 256 chars.
Its format will generally be either a text string or numerical.
6.9 objective The set of objectives that are covered by this evaluation for the type of participant(s). O
 

 

 
6.9.1 view The view to which the interpretation is applied. O n Enumerated:
All (default)
Administrator
AdminAuthority
Assessor
Author
Candidate
Invigilator
Proctor
Psychometrician
Scorer
Tutor
The 'All' view is the default value.
6.9.2 uri The location of the objectives file. C
 
CDATA
String 256 chars.
As per RFC1630.
An alternative to this is the 'entityref' attribute cf. 6.9.3).
6.9.3 entityref The entity reference that is used to bind an external file to the XML instance.
 

 
CDATA string defining the link.
String 256 chars.
This is an alternative to the 'uri' attribute (cf. 6.9.2)
6.10 control The feedback mechanisms that were revealed to the participant(s) O
 
#PCDATA
String 128 chars.
This will be a string that contains information describing the particular feedback given.
6.10.1 hint_switch Used to show if hints had been displayed to the participant. O
 
Enumerated:
Yes
No (default)

 
6.10.2 solution_switch Used to show if solutions had been displayed to the participant. O
 
Enumerated:
Yes
No (default)

 
6.10.3 feedback_switch Used to show if feedback had been displayed to the participant. O
 
Enumerated:
Yes
No (default)

 
6.11 status The status of the evaluation that is/has been undertaken. O
 

 

 
6.11.1 type_label The vocabulary for the field. O
 
As per structure 6.2.
6.11.2 datetime The date at which the status is recorded. O
 
As per structure 6.5.2.
6.11.3 status_value The status of the event. M
 
#PCDATA String.
1-64 chars.

 
6.12 duration The time taken for the completion of the section. O
 
#PCDATA String.
1-32 chars.
Defined as per ISO 8601. The underlying string structure is: PnYnMnDT
nH:nM:nS
6.13 score The score and associated descriptive information for the evaluation. O
 

 

 
6.13.1 varname The variable name being used to contain the score. O
 
#PCDATA String.
1-256 chars.

 
6.13.2 vartype The type of scoring variable. O
 
Enumerated:
String
Integer (Default)
Decimal
Scientific
Boolean
Set

 
6.13.3 status The status of the score being described. O
 
Enumerated:
Valid (default)
Noanswer
Error
The error value is assigned if an error was generated when undertaking the response processing or outcomes processing.
6.13.4 score_value The actual score value being reported. O
 
#PCDATA String.
1-32 chars.

 
6.13.5 score_interpretation A description of the way in which the scores should be interpreted. O
 
#PCDATA String.
1-2048 chars.

 
6.13.5.1 uri The location of the interpretation file. C
 
CDATA
String 256 chars.
As per RFC1630.
An alternative to this is the 'entityref' attribute cf. 6.13.5.2).
6.13.5.2 entityref The entity reference that is used to bind an external file to the XML instance.
 

 
CDATA string defining the link.
String 256 chars.
This is an alternative to the 'uri' attribute (cf. 6.13.5.1)
6.13.6 score_min The minimum score that could have been awarded to the evaluation. O
 
#PCDATA String real number.
32 chars
This is only valid for numeric variables.
6.13.7 score_max The maximum score that could have been awarded to the evaluation. O
 
#PCDATA String real number.
32 chars
This is only valid for numeric variables.
6.13.8 score_normalized The normalized value for the score i.e. a number in the range of 0-1. O
 
#PCDATA String real number.
32 chars
The number is normalized using the minimum and maximum values.
This is only valid for numeric variables.
6.13.9 score_average The average score that would be expected when undertaking the evaluation. O
 
#PCDATA String real number.
32 chars
This information is not derived from the data-set itself.
This is only valid for numeric variables.
6.13.10 score_std_error The standard error associated with the score. O
 
#PCDATA String real number.
32 chars
This information is not derived from the data-set itself.
This is only valid for numeric variables.
6.13.11 score_reliability A numerical indication of the reliability of the score. O
 
#PCDATA String real number.
32 chars
This information is not derived from the data-set itself.
6.13.12 score_cut The cut score which when achieved or surpassed denotes mastery of the topic. O
 
#PCDATA String 32 chars
 
6.13.13 extension The extension facility for the scores. O
 
As per structure 7.9 (Table 4.7).
6.14 grade The grading information assigned to the result of this evaluation. O
 

 

 
6.14.1 varname The variable name being used to contain the grade. O
 
#PCDATA String.
1-256 chars.

 
6.14.2 members The possible set of grades that could have been awarded. O
 
#PCDATA
String 1-2048.
This will be a comma separated list.
6.14.3 status The status of the grade being described. O
 
Enumerated:
Valid (default)
Noanswer
Error
The error value is assigned if an error was generated when undertaking the response processing or outcomes processing.
6.14.4 grade_value The grade that has been awarded. O
 
#PCDATA
String 1-64.

 
6.14.5 grade_cut The cut grade which when achieved or exceeded denotes mastery of the topic. O
 
#PCDATA
String 1-64.

 
6.14.6 extension The extension facility for the grades. O
 
As per structure 7.10 (Table 4.7).
6.15 outcomes The set of scores and/or grades that are to be reported for the evaluation. O
 

 

 
6.15.1 status The status of the report. O
 
As per structure 6.11.
6.15.2 score The details of a score assigned to the evaluation. O n As per structure 6.13.
6.15.3 grade The details of the grade assigned to the evaluation. O n As per structure 6.14.
6.16 num_items The number of accumulated Items that are contained in the evaluation. O
 
#PCDATA
String 1-32 chars.

 
6.17 num_sections The number of Sections that are directly contained in the parent. O
 
#PCDATA
String 1-32 chars.

 
6.18 num_items_presented The number of items that have been presented to the participant as part of the evaluation. O
 
#PCDATA
String 1-32 chars.

 
6.19 num_items_attempted The number of items that have been attempted by the participant during the evaluation. O
 
#PCDATA
String 1-32 chars.

 
6.20 num_sections_presented The number of sections that have been presented to the participant as part of the evaluation. O
 
#PCDATA
String 1-32 chars.

 
6.21 feedback_displayed The type of feedback that was presented to the participant. O n #PCDATA String.
1-10 chars.
This will take one of the values: 'Response', 'Hint' or 'Solution'.
6.21.1 ident_ref The formal identifier associated with this feedback. This would be identical to that assigned under the QTI ASI specification. O
 
As per structure 6.7 (Table 4.6).
6.21.2 asi_title The title of the feedback being reported. This would be identical to that assigned under the QTI ASI specification. O
 
As per structure 6.6 (Table 4.6).
6.21.3 uri The location of the feedback file. C
 
CDATA
String 256 chars.
As per RFC1630.
An alternative to this is the 'entityref' attribute cf. 6.20.4).
6.21.4 entityref The entity reference that is used to bind an external file to the XML instance.
 

 
CDATA string defining the link.
String 256 chars.
This is an alternative to the 'uri' attribute (cf. 6.20.3)
6.22 asi_description A description of the form of the evaluation object that was presented to the participant. O
 
#PCDATA String.
1-2048 chars.

 
6.22.1 uri The location of the description file. C
 
CDATA
String 256 chars.
As per RFC1630.
An alternative to this is the 'entityref' attribute cf. 6.21.2).
6.22.2 entityref The entity reference that is used to bind an external file to the XML instance.
 

 
CDATA string defining the link.
String 256 chars.
This is an alternative to the 'uri' attribute (cf. 6.21.1)

4.3.7 'extension' Definitions

Table 4.7 list the set of extension names that are used within the XML binding. These names are mapped to the 'extension' element used within Tables 4.1 to 4.6 of the Information Model.

 

Table 4.7 Common data objects detailed description.

 
No
Name
Source Element
Usage
7.1 extension_context context Extension for the 'Context' data structure.
7.2 extension _result result Extension for the 'Result' core data structure.
7.3 extension _summary_result summary Extension for the 'Summary' core data structure.
7.4 extension _assessment_result assessment_result Extension for the 'Assessment' core data structure.
7.5 extension _section_result section_result Extension for the 'Section' core data structure.
7.6 extension_item_result item_result Extension for the 'Item' core data structure.
7.7 extension _responseform responseform Extension for the 'Responseform' data structure. Supports any proprietary features that describe the response structure presented to the participant.
7.8 extension_response response Extension for the 'Response' data structure. Supports any proprietary features used in the original response block structure.
7.9 extension _score score Extension for the scoring information for a particular variable.
7.10 extension _grade grade Extension for the grading information.

5. IMS Supported QTI Vocabularies & Taxonomies

Within the QTI Results Reports there are many special vocabularies that are required to define the specific type of information being included. These vocabularies and their default IMS file name are listed in Table 5.1.

 

Table 5.1 'typename' list of vocabularies.

 
No
Source Element
File Name
Default Vocabulary
1 generic_identifier imsqtiv1p2_identifier.txt TBD.
2 date imsqtiv1p2_date.txt TBD.
3 status imsqtiv1p2_status.txt TBD.
4 summary_result imsqtiv1p2_summary.txt TBD.
5 asi_metadata imsqtiv1p2_metadata.txt See sub-section 6.2.

A request is made to those organizations adopting the IMS QTI results Reporting specification to inform the IMS QTI working-group of the vocabulary entries that they require.

6. Meta-data Description

6.1 IMS Meta-data

Support for the inclusion of the IMS Meta-data is through the usage of the IMS Content Packaging mechanism.

6.2 QTI-Specific Meta-data

The vocabulary for the QTI-specific meta-data is defined in Table 6.1. The vocabulary is supplied as a comma separated list in the text file "imsqtiv1p2_metadata.txt". In Table 6.1 and entry is either optional ('O') or not applicable ('N/A') for the results of a particular evaluation object (Assessment, section or Item result).

 

Table 6.1 Assessment, Section and Item meta-data list comparison.

 
Field
IMS
Class
Assess Result
Class
Section Result
Class
Item Result
Class
qmd_absolutescore_max The maximum score that the user may attain (Real number: 1-9999). O O O
qmd_absolutescore_min The minimum score that the user may attain (Real number: 1-9999). O O O
qmd_assessmenttype The type of assessment role:
Examination
Survey
Tutorial
Self-assessment.
O N/A N/A
qmd_computerscored Whether or not the item can be computer scored [Yes/No]. N/A N/A O
qmd_feedbackapermitted Whether or not the feedback is to be made available.
Value: [Yes/No] with default=Yes.
O O O
qmd_hintspermitted Whether or not the hints are to be made available [Yes/No] with default=Yes. O O O
qmd_itemselection Whether or not Section sequencing is available.
Value: [Yes/No] with default=Yes.
O O N/A
qmd_itemsequence Whether or not Item selection is available.
Value: [Yes/No] with default=Yes.
O O N/A
qmd_itemtype The type of Item (list including):
Logical identifier
XY co-ordinate
String
Numeric
Logical group
Composite
Proprietary
N/A N/A O
qmd_levelofdifficulty The level of difficulty of the Item (list including):
Pre-school
School
Higher/further education
Vocational
Professional Development
N/A N/A O
qmd_material Listing of the types of content supplied in the Item:
Text/basic
Text/rtf
Text/html
Text/xhtml
Image/gif
Image/jpeg
Audio/aicc
Audio/wav
Video/quicktime 3
Video/quicktime 4
Video/avi
Video/mpeg 1
Video/mpeg2
Video/mpeg4
Applet/java
Application
O O O
qmd_numberofitems The number of Items directly referenced within the Section. N/A O N/A
qmd_penaltyvalue The penalty value that is to be used with the GuessPenalty scoring algorithms. This is a number in the range -231-3 to 231-1. O O O
qmd_questiontype The type of question (list including):
True/false
Multiple-choice
Multiple-response
FIB-string
FIB-numeric
Image hot-spot
Drag-and-drop
Essay
N/A N/A O
qmd_renderingtype The type of rendering employed (list including):
Choice
Hot spot
Slider
Text entry
Proprietary
N/A N/A O
qmd_responsetype The class of responses required by the Item:
Single
Multiple
Ordered
Proprietary
N/A N/A O
qmd_scorereliability The reliability metric that has been assigned to the score of this evaluation object. O O O
qmd_scorestderr The standard error that has been assigned to the score for this evaluation object. O O O
qmd_scoretype The scoring classification:
Absolute
Percentage
Unscored
Multidimensional
O O N/A
qmd_scoringpermitted Whether or not scoring is enabled.
Value: [Yes/No] with default=Yes.
N/A N/A O
qmd_sectionselection Whether or not the solutions are available.
Value: [Yes/No] with default=Yes.
O O N/A
qmd_sectionsequence Whether or not Section selection is available.
Value: [Yes/No] with default=Yes.
O O N/A
qmd_sectionsincluded Whether or not other Sections are defined within the Section [Yes/No]. N/A O N/A
qmd_solutionspermitted Whether or not the solutions are to be made available.
Value: [Yes/No] with default=Yes.
O O O
qmd_status The status of the Item:
Experimental
Normal
Retired.
N/A N/A O
qmd_timedependence Whether or not the user responses are time dependent.
Value: [Yes/No] with default=No.
N/A N/A O
qmd_timelimit The number of minutes or an unlimited duration. O O O
qmd_toolvendor Name of the vendor of the tool creating the Assessments. O N/A O
qmd_topic A brief description of the topic covered by the Item. N/A O O
qmd_typeofsolution The type of solution supplied by the Item:
Complete
Incremental
Multilevel
Proprietary
N/A N/A O
qmd_versionnumber The version number that has been assigned to the object. The nomenclature for the versioning is outside the scope of this specification.
Value: String 1-256 chars.
O O O
qmd_weighting The weighting of the Item to be applied in scoring.
Real number: 0.01-10
N/A O O

The context for the IMS QTI-specific meta-data is taken from the object in which the meta-data is declared (the listed meta-data would refer to the results of the identified 'Item'):

  <qti_result_report>
    <context>
      ...
    </context>
    <item_result ident_ref="***...***">
      ...
      <asi_metadata>
        <vocabulary uri="imsqtiv1p2_metadata.txt" vocab_type="text/plain"/>
        <asi_metadatafield>
          <field_name>qmd_questiontype</field_name>
          <field_value>Multiple-choice</field_value>
        </asi_metadatafield>
        ...
      </asi_metadata>
      ...
    </item_result>
  </qti_result_report>

7. Conformance

The purpose of this statement is to provide a mechanism for customers to fairly compare vendors of assessment systems, tools and content. It is not mandatory for a vendor to support every feature of the QTI Results Reporting specification, but a vendor must detail their level of support with a "Conformance Statement". For example vendors may choose to accept or publish QTI data, but not choose to repackage QTI data. Compliance is represented by:

  • Conformance summary - this is a summary that shows, in colloquial terms, the capabilities of a particular implementation with respect to the IMS QTI Results Reporting specification;
  • Interoperability statement - this is a detailed technical checklist that identifies all of the feature capabilities of the implementation in terms of the IMS QTI Results Reporting specification functions.

7.1 Valid Data Issues

Vendors claiming conformance shall publish, accept, and/or repackage valid IMS QTI Results Reporting data as defined by the DTD including proprietary extensions where applicable. Vendors claiming their tools publish QTI Results Reports shall export valid QTI data. Vendors claiming their system tools accept QTI Results Reports data shall be able to parse and recognize valid QTI data. Vendors claiming their system tools repackage QTI Results Reports data shall be able "pass through" valid QTI Results Reports data whether the tool recognizes the optional elements or not. Vendors claiming their assessment content conforms to this specification shall provide valid QTI Results Reports data. Publishers claiming their content conforms to QTI Results Reports shall provide valid QTI Results Reports data.

7.2 Conformance Summary

Vendors claiming conformance must provide a "Conformance Summary", detailing their level of conformance, substantially similar to the information shown below, upon a reasonable request from a member of the IMS, or a prospective customer(s). It is expected that this table, a template of which is shown in Table 7.1, is a summary of the information given in the 'Interoperability statement'. The intention is for the 'Conformance Summary' to be informative in nature.

 

Table 7.1 QTI Results Report conformance summary.

 

QTI Results Report Conformance Summary (Version 1.2)

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
context
Y or N
Y or N
Y or N
summary result
Y or N
Y or N
Y or N
score
Y or N
Y or N
Y or N
grade
Y or N
Y or N
Y or N
outcomes
Y or N
Y or N
Y or N
assessment result
Y or N
Y or N
Y or N
outcomes
Y or N
Y or N
Y or N
section_result
Y or N
Y or N
Y or N
section result
Y or N
Y or N
Y or N
outcomes
Y or N
Y or N
Y or N
item_result
Y or N
Y or N
Y or N
item result
Y or N
Y or N
Y or N
response
Y or N
Y or N
Y or N
outcomes
Y or N
Y or N
Y or N

Completion of the three columns is intended to reflect:

  • Publish - this implies that the XML instance contains the identified elements. If such an element is not ticked then it will not occur within the exported QTI-XML instance(s);
  • Accept - it is assumed that the ability to accept the contents of an element is accompanied by the ability to use, and if appropriate, display that content. If this is not the case but the content of the material can be exported then the 'Repackage' column can still be ticked;
  • Repackage - this is the ability to import QTI-XML instances from one or more sources and to create a new instance that combines the imported information. It is not necessary for the repackaging system to be able to operate on the information supplied.

7.3 Interoperability Statement

An example of the detailed 'Interoperability Statement' is shown in Tables 7.2a, 7.2b, 7.2c, 7.2d and 7.2e (one for each of the core structures). Compliance to QTI means that at least one of the columns must be completed.

 

Table 7.2a Interoperability statement (context).

 
context
Version 1.2
Mandatory: This structure is mandatory i.e. it MUST be used in any IMS QTI Results Reporting XML instance.
Optional Fields: Optional fields are informative. Checking an optional field implies that all of the associated mandatory elements are supported.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
qti_comment
q
q
q
name
q
q
q
generic_identifier
q
q
q
date
q
q
q
Extension Fields: These features allow the data model to be extended.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
extension_context
q
q
q

 

Table 7.2b Interoperability statement (summary_result).

 
summary result
Version 1.2
Optional Fields: Optional fields are informative. Checking an optional field implies that all of the associated mandatory elements are supported.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
type_label
q
q
q
generic_identifier
q
q
q
asi_title
q
q
q
date
q
q
q
status
q
q
q
duration
q
q
q
score
q
q
q
grade
q
q
q
outcomes
q
q
q
status
q
q
q
score
q
q
q
grade
q
q
q
Extension Fields: These features allow the data model to be extended.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
extension_summary_result
q
q
q

 

Table 7.2c Interoperability statement (assessment_result).

 
assessment result
Version 1.2
Optional Fields: Optional fields are informative. Checking an optional field implies that all of the associated mandatory elements are supported.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
qti_comment
q
q
q
asi_title
q
q
q
ident_ref
q
q
q
asi_metadata
q
q
q
asi_description
q
q
q
date
q
q
q
duration
q
q
q
objective
q
q
q
control
q
q
q
outcomes
q
q
q
status
q
q
q
score
q
q
q
grade
q
q
q
feedback_displayed
q
q
q
num_items
q
q
q
num_sections
q
q
q
num_items_presented
q
q
q
num_items_attempted
q
q
q
num_sections_presented
q
q
q
section_result
q
q
q
Extension Fields: These features allow the data model to be extended.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
extension_assessment_result
q
q
q
extension_score
q
q
q
extension_grade
q
q
q

 

Table 7.2d Interoperability statement (section_result).

 
section result
Version 1.2
Optional Fields: Optional fields are informative. Checking an optional field implies that all of the associated mandatory elements are supported.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
qti_comment
q
q
q
asi_title
q
q
q
ident_ref
q
q
q
presented
q
q
q
asi_metadata
q
q
q
asi_description
q
q
q
date
q
q
q
duration
q
q
q
objective
q
q
q
control
q
q
q
outcomes
q
q
q
status
q
q
q
score
q
q
q
grade
q
q
q
feedback_displayed
q
q
q
num_items
q
q
q
num_sections
q
q
q
num_items_presented
q
q
q
num_items_attempted
q
q
q
num_sections_presented
q
q
q
section_result
q
q
q
item_result
q
q
q
Extension Fields: These features allow the data model to be extended.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
extension_section_result
q
q
q
extension _score
q
q
q
extension _grade
q
q
q

 

Table 7.2e Interoperability statement (item_result).

 
item result
Version 1.2
Optional Fields: Optional fields are informative. Checking an optional field implies that all of the associated mandatory elements are supported.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
qti_comment
q
q
q
asi_title
q
q
q
ident_ref
q
q
q
presented
q
q
q
asi_metadata
q
q
q
asi_description
q
q
q
date
q
q
q
duration
q
q
q
objective
q
q
q
control
q
q
q
response
q
q
q
responseform
q
q
q
num_attempts
q
q
q
response_value
q
q
q
outcomes
q
q
q
status
q
q
q
score
q
q
q
grade
q
q
q
feedback_displayed
q
q
q
Extension Fields: These features allow the data model to be extended.

 
Publish
(export, data)
Accept
(import, display)
Repackage Feature
extension_item_result
q
q
q
extension_responseform
q
q
q
extension_response
q
q
q
extension_score
q
q
q
extension_grade
q
q
q

Note that the 'Interoperability Statement' addresses support for the various elements within the binding. The set of attributes are not considered. Inclusion of conformance with respect to attributes will be considered in later versions of the specification.

It is important that the 'Interoperability Statement' is clear in showing what is and, perhaps more importantly, what is not supported. The usage of descriptive conformance approach has been adopted to encourage vendors to be as clear as possible when describing the capabilities of their QTI-compliant systems.

About This Document

 
Title IMS Question & Test Interoperability Results Reporting Information Model
Editors Colin Smythe, Lane Brewer, and Steve Lay
Version 1.2
Version Date 11 February 2002
Status Final Specification
Summary

This document presents the IMS QTI Results Reporting Information Model. This specification describes the data model for the exchange of results. These results may or may not have been derived from assessments that conform to the IMS QTI ASI specification.

Revision Information 22 January 2002
Purpose Defines the final specification for the results reporting part of the IMS QTI specifications.
Document Location http://www.imsglobal.org/question/v1p2/imsqti_res_infov1p2.html

List of Contributors

The following individuals contributed to the development of this document:

Russell Almond ETS, USA
Lane Brewer Galton Technologies Inc.
Todd Brewer Galton Technologies Inc.
Russell Grocott Can Studios Ltd.
Andy Heath CETIS/JISC, UK
Paul Hilton Can Studios Ltd.
John Kleeman Question Mark Computing Ltd.
Steven Lay University of Cambridge Local Examinations Syndicate, UK
Jez Lord Can Studios Ltd.
Paul Roberts Question Mark Computing Ltd.
Nial Sclater CETIS/University of Strathclyde, UK
Colin Smythe Dunelm Services Ltd.

Revision History

 
Version No.
Release Date
Comments
Base Document Version 1.2 7 June 2001 The first formal release of the IMS QTI: Results Reporting Information Model Base Document.
Public Draft
Version 1.2
30 September 2001 The formal public draft release of the IMS QTI: Results Reporting Information Model Public Draft Specification Document. This is the first release of this document for public review.
Final Version 1.2 11 February 2002 The first final release of the V1.2 IMS QTI Results Reporting Information Model.

Index

A
ASI 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     Assessment 1, 2, 3, 4, 5, 6, 7
     Item 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     Section 1, 2, 3, 4, 5, 6, 7, 8
Assessment 1, 2, 3, 4, 5, 6, 7
Assessment_result Elements
     assessment_result 1, 2, 3, 4, 5, 6
     extension_ assessment_result 1
Attributes
     asi_title 1, 2, 3, 4, 5, 6, 7, 8
     cardinality 1
     entityref 1, 2, 3, 4, 5, 6
     feedback_switch 1
     hint_switch 1
     ident_ref 1, 2, 3, 4, 5, 6, 7, 8, 9
     members 1
     presented 1, 2, 3, 4, 5, 6, 7, 8, 9
     render_type 1
     response_status 1
     response_time 1
     response_type 1
     solution_switch 1
     source 1
     status 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     timing 1
     uri 1, 2, 3, 4, 5, 6, 7
     varname 1, 2
     vartype 1
     view 1
     xmllang 1
 

C
Common Elements
     asi_metadata 1, 2, 3, 4, 5, 6, 7, 8, 9
     control 1, 2, 3, 4, 5, 6, 7, 8, 9
     date 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     duration 1, 2, 3, 4, 5, 6, 7, 8, 9
     feedback_displayed 1, 2, 3, 4, 5, 6, 7
     grade 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     grade_cut 1
     grade_value 1
     num_items 1, 2, 3, 4, 5
     num_items_attempted 1, 2, 3, 4, 5
     num_items_presented 1, 2, 3, 4, 5
     num_sections 1, 2, 3, 4, 5
     num_sections_presented 1, 2, 3, 4, 5
     other 1
     outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     qti_comment 1, 2, 3, 4, 5, 6, 7, 8, 9
     score 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
     score_average 1
     score_cut 1
     score_interpretation 1
     score_max 1
     score_min 1
     score_normalized 1
     score_reliability 1
     score_value 1
     status 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
Context Elements
     context 1, 2, 3, 4, 5, 6, 7
     date 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     extension_context 1, 2
     generic_identifier 1, 2, 3, 4, 5, 6
     name 1, 2, 3, 4, 5, 6, 7, 8, 9
 

D
DTD 1, 2

E
Elements
     asi_description 1, 2, 3, 4, 5, 6, 7
     asi_metadata 1, 2, 3, 4, 5, 6, 7, 8, 9
     asi_metadatafield 1, 2
     assessment_result 1, 2, 3, 4, 5, 6
     context 1, 2, 3, 4, 5, 6, 7
     control 1, 2, 3, 4, 5, 6, 7, 8, 9
     correct_response 1
     date 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     datetime 1, 2
     duration 1, 2, 3, 4, 5, 6, 7, 8, 9
     extension_assessment_result 1
     extension_context 1, 2
     extension_grade 1, 2
     extension_item_result 1, 2
     extension_response 1, 2
     extension_responseform 1
     extension_score 1, 2
     extension_section_result 1
     extension_summary_result 1
     feedback_display 1, 2, 3, 4, 5, 6, 7
     field_name 1, 2
     field_value 1, 2
     generic_identifier 1, 2, 3, 4, 5, 6
     grade 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     grade_cut 1
     grade_value 1
     identifier_string 1
     item_result 1, 2, 3, 4, 5, 6, 7, 8
     name 1, 2, 3, 4, 5, 6, 7, 8, 9
     num_attempts 1, 2
     num_items 1, 2, 3, 4, 5
     num_items_attempted 1, 2, 3, 4, 5
     num_items_presented 1, 2, 3, 4, 5
     num_sections_presented 1, 2, 3, 4, 5
     objective 1, 2, 3, 4, 5, 6, 7
     other 1
     outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     qti_comment 1, 2, 3, 4, 5, 6, 7, 8, 9
     qti_result_report 1
     response 1, 2, 3, 4, 5, 6, 7
     response_value 1, 2
     result 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     score 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
     score_average 1
     score_cut 1
     score_interpretation 1
     score_max 1
     score_min 1
     score_normalized 1
     score_reliability 1
     score_std_error 1
     score_value 1
     section_result 1, 2, 3, 4, 5, 6, 7, 8
     status 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     status_value 1
     summary_result 1, 2, 3, 4, 5, 6, 7
     type_label 1, 2, 3, 4, 5
     vocabulary 1, 2, 3, 4, 5, 6, 7, 8
Extension Elements
     extension 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     extension_assessment_result 1
     extension_context 1, 2
     extension_grade 1, 2
     extension_item_result 1, 2
     extension_response 1, 2
     extension_responseform 1
     extension_score 1, 2
     extension_section_result 1
     extension_summary_result 1
 

I
Item 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
Item_result Elements
     asi_metadata 1, 2, 3, 4, 5, 6, 7, 8, 9
     control 1, 2, 3, 4, 5, 6, 7, 8, 9
     correct_response 1
     date 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     duration 1, 2, 3, 4, 5, 6, 7, 8, 9
     extension_response 1, 2
     item_result 1, 2, 3, 4, 5, 6, 7, 8
     num_attempts 1, 2
     objective 1, 2, 3, 4, 5, 6, 7
     outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     qti_comment 1, 2, 3, 4, 5, 6, 7, 8, 9
     response 1, 2, 3, 4, 5, 6, 7
     response_value 1, 2
 

M
Meta-data 1, 2, 3, 4, 5, 6, 7
     Description 1
 

S
Section 1, 2, 3, 4, 5, 6, 7, 8
Section_result Elements
     extension_section_result 1
Summary_result Elements
     summary_result 1, 2, 3, 4, 5, 6, 7
 

V
Vocabularies 1, 2

X
XML 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     Schema
 

DTD 1, 2

XSD 1 XSD 1

 

 

 

IMS Global Learning Consortium, Inc. ("IMS") is publishing the information contained in this IMS Question & Test Interoperability: Results Reporting Information Model ("Specification") for purposes of scientific, experimental, and scholarly collaboration only.

IMS makes no warranty or representation regarding the accuracy or completeness of the Specification.
This material is provided on an "As Is" and "As Available" basis.

The Specification is at all times subject to change and revision without notice.

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

IMS would appreciate receiving your comments and suggestions.

Please contact IMS through our website at http://www.imsglobal.org

Please refer to Document Name:
IMS Question & Test Interoperability: Results Reporting Information Model Date: 11 February 2002