IMS Logo IMS Question & Test Interoperability:
ASI Outcomes Processing

Final Specification Version 1.2
Copyright © 2002 IMS Global Learning Consortium, Inc. All Rights Reserved.
The IMS Logo is a trademark of IMS Global Learning Consortium, Inc.
Document Name: IMS Question & Test Interoperability: ASI Outcomes Processing
Date: 11 February 2002

 

IPR and Distribution Notices

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS's procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Copyright © 2002 IMS Global Learning Consortium. All Rights Reserved.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/license.html.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.


 

Table of Contents


1. Introduction
     1.1 Question & Test Interoperability Overview
     1.2 Scope & Context
     1.3 Structure of this Document
     1.4 Nomenclature
     1.5 References

2. Information Model
     2.1 Exchanging ASI Objects
     2.2 Use Cases
     2.3 ASI Issues
     2.4 Outcomes Processing Solution
           2.4.1 The In-built Scoring Algorithms
           2.4.2 Defined Proprietary Algorithms
     2.5 Tabular Representation
           2.5.1 QTI Outcomes Processing Data Objects

3. XML Binding
     3.1 outcomes_processing> Elements
           3.1.1 <qticomment> Element
           3.1.2 <outcomes> Element
           3.1.3 <objects_condition> Element
           3.1.4 <processing_parameter> Element
           3.1.5 <map_output> Element
           3.1.6 <outcomes_feedback_test> Element
     3.2 <outcomes> Element
           3.2.1 <qticomment> Element
           3.2.2 <decvar> Element
           3.2.3 <interpretvar> Element
     3.3 <objects_condition>
           3.3.1 <qticomment> Element
           3.3.2 <outcomes_metadata> Element
           3.3.3 <and_out> Element
           3.3.4 <or_out> Element
           3.3.5 <not_out> Element
           3.3.6 <objects_parameter> Element
           3.3.7 <map_input> Element
           3.3.8 <objectscond_extension> Element
     3.4 <outcomes_feedback_test> Element
           3.4.1 <test_variable> Elements
           3.4.2 <displayfeedback> Element
     3.5 <outcomes_metadata> Element
     3.6 <and_objects> Element
           3.6.1 <outcomes_metadata> Element
           3.6.2 <and_objects> Element
           3.6.3 <or_objects> Element
           3.6.4 <not_objects> Element
     3.7 <or_objects> Element
           3.7.1 <outcomes_metadata> Element
           3.7.2 <and_objects> Element
           3.7.3 <or_objects> Element
           3.7.4 <not_objects> Element
     3.8 <not_objects> Element
           3.8.1 <outcomes_metadata> Element
           3.8.2 <and_objects> Element
           3.8.3 <or_objects> Element
           3.8.4 <not_objects> Element
     3.9 <test_variable> Element
           3.9.1 <variable_test> Element
           3.9.2 <and_test> Element
           3.9.3 <or_test> Element
           3.9.4 <not_test> Element
     3.10 <and_test> Element
           3.10.1 <variable_test> Element
           3.10.2 <and_test> Element
           3.10.3 <or_test> Element
           3.10.4 <not_test> Element
     3.11 <or_test> Element
           3.11.1 <variable_test> Element
           3.11.2 <and_test> Element
           3.11.3 <or_test> Element
           3.11.4 <not_test> Element
     3.12 <not_test> Element
           3.12.1 <variable_test> Element
           3.12.2 <and_test> Element
           3.12.3 <or_test> Element
           3.12.4 <not_test> Element

4. Best Practice & Implementation Guide
     4.1 Overall Data Model
     4.2 Relationship to the Other IMS Specifications
     4.3 Basic Example XML Instances
           4.3.1 Example ('Number Correct' and 'Number Correct Attempted')
           4.3.2 Example ('Weighted Number Correct', 'Weighted Number Correct Attempted',
'Parameter Weighted Number Correct' and 'Parameter Weighted Number
Correct Attempted')

           4.3.3 Example ('SumofScores' and 'SumofScoresAttempted')
           4.3.4 Example ('Weighted Sum of Scores', 'Weighted Sum of Scores Attempted', 'Parameter Weighted Sum of Scores' and 'Parameter Weighted Sum of Scores Attempted')
           4.3.5 Example ('Best K from N')
           4.3.6 Example ('Guessing Penalty' and 'WeightedGuessingPenalty')
           4.3.7 Example (Remapped Output 'Sum of Scores')
           4.3.8 Example (Remapped Input 'Sum of Scores')
           4.3.9 Example (Parameterized 'Sum of Scores')
     4.4 Advanced Example XML Instances
           4.4.1 Example (Multiple-choice Quiz)
           4.4.2 Example (True/false Quiz)
     4.5 Implementation Guidance
           4.5.1 The In-built Scoring Algorithms
           4.5.2 Defining Proprietary Scoring Algorithms
           4.5.3 Using Meta-data
           4.5.4 Consideration of Selection & Ordering
     4.6 Example XML Instances
     4.7 Proprietary Extensions
     4.8 V1.x/V2.0 Issues & Compatibility
     4.9 IMS Harmonization

Appendix A - Glossary of Terms
     A1 - General Terms
     A2 - Elements & Attributes

Appendix B - In-Built Scoring Algorithms
     B1 - 'Number Correct' Scoring Algorithm
     B3 - 'Weighted Number Correct' Scoring Algorithm
     B4 - 'Weighted Number Correct (Attempted)' Scoring Algorithm
     B5 - 'Parameter Weighted Number Correct' Scoring Algorithm
     B6 - 'Parameter Weighted Number Correct (Attempted)' Scoring Algorithm
     B7 - 'Sum of Scores' Scoring Algorithm
     B8 - 'Sum of Scores (Attempted)' Scoring Algorithm
     B9 - 'Weighted Sum of Scores' Scoring Algorithm
     B10 - 'Weighted Sum of Scores (Attempted)' Scoring Algorithm
     B11 - 'Parameter Weighted Sum of Scores' Scoring Algorithm
     B12 - 'Parameter Weighted Sum of Scores (Attempted)' Scoring Algorithm
     B13 - 'Best K from N' Scoring Algorithm
     B14 - 'Guessing Penalty' Scoring Algorithm
     B15 - 'Weighted Guessing Penalty' Scoring Algorithm

Appendix C - Logic Rules

About This Document
     List of Contributors

Revision History

Index


1. Introduction

1.1 Question & Test Interoperability Overview

The IMS Question & Test Interoperability (QTI) specification describes a basic structure for the representation of question (item) and test (assessment) data and their corresponding results reports [QTI, 02i]. Therefore, the specification enables the exchange of this item, assessment and results data between Learning Management Systems, as well as content authors and, content libraries and collections. The IMS QTI specification is defined in XML to promote the widest possible adoption. XML is a powerful, flexible, industry standard mark-up language used to encode data models for Internet-enabled and distributed applications. The QTI specification is extensible and customizable to permit immediate adoption, even in specialized or proprietary systems. Leading suppliers and consumers of learning products, services and content contributed time and expertise to produce this final specification. The IMS QTI specification, like all IMS specifications, does not limit product designs by specifying user interfaces, pedagogical paradigms, or establishing technology or policies that constrain innovation, interoperability, or reuse.

The 'Outcomes Processing' specification contains the description of how the aggregated scores at the Assessment and Section levels can be derived. These scoring outcomes are based upon the child Sections and/or Items. A wide range of scoring algorithms is supported through the usage of a predefined set of parameterized instructions; these avoid the realization of the algorithms within the XML. This document contains the relevant information model, XML binding and best practices guidance but it should be read in the context of the core ASI documents.

1.2 Scope & Context

This document is the IMS Question & Test Interoperability ASI Outcomes Processing Specification and must be read in conjunction with the core documents:

  • IMS QTI: ASI Information Model [QTI, 02a];
  • IMS QTI: ASI XML Binding [QTI, 02b];
  • IMS QTI: ASI Best Practice & Implementation Guide [QTI, 02c].

It defines the outcomes processing features that are to be applied to the Section and/or Assessments. These new Assessment and Section features are not backwards compatible with the other versions of the IMS QTI specifications. These new features have no effect on the IMS QTILite specification [QTI, 02h] because that refers only to the Item data structure.

1.3 Structure of this Document

The structure of this document is:

2. Information Model The description of the information model of the outcomes processing component for the full IMS QTI ASI;
3. XML Binding The description of the XML binding of the outcomes processing component for the full IMS QTI ASI;
4. Best Practice & Implementation Guide The description of the best practices and implementation guide for the outcomes processing component of the full IMS QTI ASI;
Appendix A - Glossary of Terms A brief description of all of the elements and attributes that have been used to support outcomes processing;
Appendix B - In-Built Scoring Algorithms A description of the operation of each of the default scoring algorithms;
Appendix C - Logic Rules The logic rules that are supported by the or_object, and_object and not_object elements.

1.4 Nomenclature

API Application Programming Interface
ASI Assessment, Section, Item
CBT Computer Based Training
DTD Document Type Definition
QTI Question & Test Interoperability
VLE Virtual Learning Environment
W3C World Wide Web Consortium
XML Extensible Mark-up Language

1.5 References

[IMS, 01] IMS Persistent, Location-Independent Resource Identifier Implementation Handbook, M.McKell, Version 1.0, IMS, April 2001.
[QTI, 02a] IMS Question & Test Interoperability: ASI Information Model Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02b] IMS Question & Test Interoperability: ASI XML Binding Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02c] IMS Question & Test Interoperability: ASI Best Practice & Implementation Guide, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02d] IMS Question & Test Interoperability: ASI Selection & Ordering Specification, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02e] IMS Question & Test Interoperability: Results Reporting Information Model, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02f] IMS Question & Test Interoperability: Results Reporting XML Binding, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02g] IMS Question & Test Interoperability: Results Reporting Best Practice & Implementation Guide, C.Smythe, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02h] IMS Question & Test Interoperability: QTILite Specification, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.
[QTI, 02i] IMS Question & Test Interoperability: Overview, C.Smythe, E.Shepherd, L.Brewer and S.Lay, Final Specification, Version 1.2, IMS, February 2002.

2. Information Model

2.1 Exchanging ASI Objects

The possible advanced Assessment structures that can be exchanged using QTI are shown in Figure 2.1:

  • The Assessment could contain a single Section block (Figure 2.1a);
  • The Assessment could consist of a reference to an external Section block (Figure 2.1b);
  • The Assessment could consist of a mixture of contained and externally referenced Section blocks (Figure 2.1c). There is no preferential order in the blocks and references.

 

Possible <assessment> structure (a)

(a)

Possible <assessment> structure (c)

(c)

Possible <assessment> structure (b)

(b)


 

 

Figure 2.1 Possible <assessment> structures.

In the case of Sections, some of the possible data structures that can be exchanged are shown in Figure 2.2:

  • The Section could contain one or more Items (Figure 2.2a);
  • The Section could contain one or more references to external Items (Figure 2.2b);
  • The Section could contain one or more Sections (Figure 2.2c);
  • The Section could contain one or more references to external Sections (Figure 2.2d);
  • The Section could contain a mixture of blocks and references to Section and Items. There is no constraint on the order in which the Sections and Items are declared/referenced i.e. interleaving is supported.

 

Possible <section> structure (a)

(a)

Possible <section> structure (b)

(b)

Possible <section> structure (c)

(c)

Possible <section> structure (d)

(d)

 

Possible <section> structure (e)

(e)

 

Figure 2.2 Possible <section> structures.

The wide range of different data structures that can be exchanged is especially significant when developing mechanisms that support 'Outcomes Processing'. 'Outcomes Processing' is responsible for:

  • The aggregation of scores assigned to the Items and/or Sections to create a score or set of scores for the parent Section. The ways in which this roll-up is achieved reflects the different scoring algorithms;
  • The aggregation of scores assigned to Sections to create a score or set of scores for the parent Assessment. The ways in which this roll-up is achieved reflects the different scoring algorithms;
  • Any assignment of scores based upon more than one Item and/or Section.

It is important to note that the type of scoring algorithm that is to be used is not mandated by the IMS specification. Instead, the specification is capable of supporting most of the available scoring algorithms. Also, the algorithms themselves are not encoded within the specification itself. Instead the type of algorithm is specified and the necessary parametric information supplied to enable the assessment engine to execute the corresponding algorithm.

2.2 Use Cases

The following representative use-cases have been identified for support within QTI ASI outcomes processing V1.2:

  1. Multiple-choice quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete (single response) questions whose outcomes are a dichotomous variable e.g. multiple-choice questions. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "CORRECT" for each Item. This takes on the value "True" if the response was correct and "False' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the percentage of total score obtained by a candidate and (3) the percentage of items actually attempted which were correct;
  2. True/false quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete true/false. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "isCorrect" for each Item. This takes on the value '1' if the response was correct and "-1' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the weighted total score;
  3. Multiple-response quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are multiple-response questions (two answers for each) with partial scores being awarded when at only one correct answer is given. For the sake of being definite we assume there are 5 Items and that we have a defined outcome, "SCORE" for each Item. This takes on the value "+1" for each correct response "-1' for each incorrect response. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the percentage of items actually attempted which were correct;
  4. End of chapter test - this is the same as use-case (a) except that all of the items have different weights. We wish to see (1) the number of correct items, (2) the percentage of total weight answered correctly by the candidate and (3) the "weighted" percentage of questions attempted correctly of those questions answered;
  5. English comprehension/composition test with essay - this is a placement test designed to assess English language ability. The test consists of two sections: (1) Short answer items (multiple-choice) and (2) the Essay. Short answer items come in two kinds. The first are 15 discrete vocabulary, grammar and usage Items that all produce a single dichotomous outcome "isCorrect". The second are the reading comprehension Items (although they may actually test other skills than just reading) that consist of a stimulus followed by five discrete questions. We assume that these are done as Items with five responses and hence five outcomes: these are Boolean variables whose meanings are as above. All of these Items can be given different weights (as can the 5 outcomes of the two reading comprehension items). The essay question is graded "A,B,C,D,E,F" by either a human or computer rater. The divisions do not necessarily represent equal spacings. The essay represents 1/3 of the total grade. Out of ten points, the mapping of the grades is as follows A=10, B=8, C=7, D=5, E=2, F=0. The total score for the assessment is required;
  6. Diagnostic scoring - this is an additional score report which has been retrofitted onto the 'English comprehension test ((e) above). In addition to the overall score, "diagnostic" scores are going to be given in four different sub-areas: "Reading", "Writing", "Vocabulary" and "Grammar" (the dependencies among these sub-scores and the overall score are ignored for computational simplicity). The 15 discrete items and each of the 10 outcomes from the reading items all load onto these sub-scores differently. Some outcomes are completely irrelevant to some of the sub-scores. An Item/outcome can have a high "vocabulary" load and a low "grammar" load, or vice-versa. The essay is also added onto each of the sub-scores, but with a different loading;
  7. Diagnostics complex response biology lite - this is a classroom assessment designed to assess the students understanding of the scientific method as it applies to Biology. The assessment returns not a single score but three scores reflecting "Domain Knowledge" (knowledge of the biological subjects in the assessment), "Methodological Knowledge" (understanding of the scientific method) and "Integrated Knowledge" (the ability to apply the scientific method to a given problem). The items are a series of complex "tasks". Each task gives a piece of scenario of a scientific investigation and has the students work through several steps of the process, answering questions about what they are doing at each step. Each task has many complex responses, and hence many observable outcomes. The outcomes are related to the number of steps in the task. Each is a variable given one of the values "Low", "Medium", or "High" depending on how the student did on one part of one step. The outcomes are labelled "OUT***" where '***' is a sequence number unique to the item. For more definite, assume that there are five tasks: Task1 has 7 outcomes, Task2 has 5, Task3 has 8, Task4 has 6 and Task5 has 13. For each of the reporting variables, the scoring model should add an appropriate "weight" if the outcome has reached an appropriate level (either "Medium" and above or "High"). The weight and the level will, in general, be different for each of the reporting variables. Not all outcomes are used for all of the reporting variables;

2.3 ASI Issues

The core features within the ASI structures that are related to the process of selection and ordering are shown in Figure 2.3.

An ASI outcomes processing structure

 

Figure 2.3 An ASI outcomes processing structure.

The relationship of these features to outcomes processing are:

  • <qtimetadata> - contains the QTI-specific meta-data about the object. This meta-data may be used to decide which objects are selected to support a particular aggregation mechanism;
  • <outcomes_processing> - the set of aggregation processing instructions that are to be applied to the child objects. The variables to contain the aggregated scores are declared using the <outcomes> element;
  • <feedback> - the feedback that is to be presented if the conditions defined within the <outcomes_processing> occur;
  • <selection_ordering> - the selection and ordering rules that are to be applied to the set of child objects contained within the parent. Only those objects that are presented to the participant can contribute to the scoring algorithm but these may or may not be attempted by the participant;
  • Internal and externally referenced Section and Item objects - the set of contained and referenced objects which can be selected and ordered;
  • External metadata - the external meta-data descriptions that are linked to the object and which conform to the IMS Meta-data specification.

An ASI outcomes processing exampl

 

Figure 2.4 An ASI outcomes processing example.

The scoping of the scoring algorithms that are supported is summarized using the example Section/Item combination shown in Figure 2.4. The scope of the scoring algorithm is:

  • For Section 'Section Ident_1' the set of children objects is - 'Item Ident_1', 'Section Ident_2', 'Section Ident-3' and 'Item Ident_2'. This means that the scoring algorithm identified in Section 'Section Ident_1' is applied to the default variables associated with each child object;
  • For Section 'Section Ident_2' the set of children objects is - 'Item Ident_2.1', 'Item Ident_2.2' and 'Item Ident_2.3'. The Section-level default variables are formed from the aggregation of these child objects;
  • For Section 'Section Ident_3' the set of children objects is - 'Item Ident_3.1', 'Item Ident_3.2' and 'Item Ident_3.3'. The Section-level default variables are formed from the aggregation of these child objects;
  • If for all of these three Sections only some of the objects are selected then the corresponding aggregation is based upon those objects presented.

It is the responsibility of the scoring engine that is implementing the scoring algorithm to maintain the scoping of the variables. If the default variable names are used then all of the variables will be named 'SCORE'. For instance, the scoping should ensure that:

Section_Ident_2.SCORE = Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE + Item_Ident_2.1.SCORE

At the level of the 'Section_Ident_1' then the variables 'Item_Ident_2.1.SCORE' etc. are only perceived through the variable 'Section_Ident_2.SCORE'.

2.4 Outcomes Processing Solution

The outcomes processing capability is based upon two complimentary mechanisms:

  • In-built scoring algorithms - in which the appropriate scoring algorithm is just named and so the corresponding assessment engine is responsible for applying the algorithm to the default input and output variables;
  • Defined proprietary algorithms - in which the relationship between the input and output scoring variables is defined using a parameterized approach.

2.4.1 The In-built Scoring Algorithms

The algorithms that are to be supported are:

  1. Number of right answers (including multivariate responses) - this is identified by the algorithm name "NumberCorrect" and has associated with it the default integer variable 'COUNT'. This is the store for the number of correctly answered objects. An object is defined as correctly answered if the value assigned to the default Boolean variable 'COUNT' for the object is 'True'. The number correct algorithm can only operate on Boolean scoring variables. The default integer scoring variables COUNT.max (equal to the number of objects attempted), COUNT.min (set to zero) and COUNT.normalized (a value of between zero and one) are also available. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  2. Number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'NumberCorrectAttempted';
  3. Weighted number right (including multivariate responses) - this is similar to the 'number of rights answers' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedNumberCorrect". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of COUNT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  4. The weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedNumberCorrectAttempted';
  5. Parameterized weighted number correct - name 'ParameterWeightedNumberCorrect' this is similar to the 'WeightedNumberCorrect' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of CORRECT is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  6. The parameterized weighted number of right answers normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedNumberCorrectAttempted';
  7. Percentage correct (including multivariate responses) - this score is derived from both the "NumberCorrect" and "NumberCorrectAttempted" algorithms in which the percentage correct is defined by the equation-

  1. Sum-of-scores - this is identified by the algorithm name "SumofScores" and has associated with it four default integer variables SCORE, SCORE.min, SCORE.max and SCORE.normalized. These variables store the sum of the scores, the minimum possible score (set to the sum of the 'minvalue' attributes for all of the corresponding objects), the maximum possible score (set to the sum of the 'maxvalue' attributes for all of the corresponding objects) and the normalized score (a value of between zero and one) respectively for the set of objects (the minimum, maximum and normalized values are only valid if they have been explicitly defined for all of the corresponding objects). The value of 'SCORE' is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  2. Sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'SumofScoresAttempted';
  3. Weighted sum of scores (including multivariate responses) - this is similar to the 'sum of scores' with the addition of the weighting of the individual components and is identified by the algorithm name "WeightedSumofScores". The same set of variables are used but their values reflect the object weighting as defined in the associated meta-data field 'qmd_weighting'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  4. The weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'WeightedSumofScoresAttempted';
  5. Parameterized weighted sum of scores - name 'ParameterWeightedSumofScores' this is similar to the 'WeightedSumofScores' algorithm but the weighting value is passed using the <objects_parameter> element in preference to the meta-data field. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  6. The parameterized weighted sum of scores normalized with respect to those objects that have been attempted and not just selected and presented. This algorithm is named 'ParameterWeightedSumofScoresAttempted';
  7. Best 'K' out of 'N' (including multivariate responses) - this is identified by the algorithm name "BestKofN" and is used to calculate the outcome by using the highest 'K' scores from the set of presented objects generically defined as 'N' objects. The values are placed in the default variables 'SCORE', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized'. The value of SCORE is determined with respect to the number of objects that have been selected and presented to the participant irrespective if an attempt is made to provide and answer;
  8. Negative scores (including multivariate responses) - called 'GuessingPenalty'. Each object has three associated variables for the values of the number of correct answers ('COUNT.correct'), the number of incorrect answers ('COUNT.incorrect') and the number of unattempted answers ('COUNT.unattempted');
  9. Item Response Theory (IRT) - supported using the processing parameter mechanism;
  10. Time remaining - for further study in V2.0;
  11. Par scores - for further study in V2.0;
  12. Cut scores (pass/fail, right/wrong and including multivariate responses) - for further study in V2.0;
  13. Multiple forms/equating (table based) - for further study in V2.0;
  14. Partial credit/Graded response - for further study;
  15. Critical Item - for further study in V2.0;
  16. Critical Outcome - for further study in V2.0;
  17. Factor analysis/ Multivariate IRT (research) -supported through extensions. For further study in V2.0;
  18. Bayes net (research) - supported through extensions. For further study in V2.0;
  19. Computer Adaptive Testing - for further study in V2.0.

2.4.2 Defined Proprietary Algorithms

Proprietary algorithms can be used but these must be characterized using the parameter definition mechanisms supported in the specification i.e. the <objects_parameter> and <processing_parameter> elements. Any number of parameters can be passed using these elements and there is no restriction on the naming convention used for these parameters - all parameters are specific to an algorithm. The names of the proprietary algorithms (passed in the scoremodel attribute) should not clash with the intrinsic names and some form of naming convention based upon the creating organization should be adopted.

2.5 Tabular Representation

The tables in this Section provide a conceptual, informative description of the elements in the data objects. The columns in these tables refer to:

No: The number of the data element. An element may be composed of sub-elements. The numbering scheme reflects these relationships.
Name: The descriptive name of the element.
Explanation: A brief functional description of the element.
Required: Indicates if the element is required:
  • M = Mandatory Element that must be included in the data object, if the element at the higher level is included;
  • C = Conditional Element. Existence is dependent on values of other Elements;
  • O = Optional Element.
Multi: Multiplicity of the element:
  • Blank = single instance;
  • Number = maximum number of times the element is repeatable;
  • n = multiple occurrences allowed, no limit;
  • Repeatability of an element implies that all sub-elements repeat with the same element.
Type: A description of formatting rules for the data element. Type includes the maximum length of the element:
  • ID = element used to uniquely identify an object;
  • Code = element value from a list of codes;
  • Description = descriptive element, human language
  • Flag = binary flag
  • Enumerated = list of predefined non-numeric options i.e. the definitive list of objects
  • The international character set specified by ISO 10646 will be used for all fields.

The type will also include a description of the set of valid values for the sub-element:

  • Coding schemes using numerical values;
  • The set of values as defined in the Domain i.e. making it closed. The list of values cannot be extended to include values not defined in the specification. If there is a need for values not included in the domain set of values then the extension should be done defining a new element under the Extension element that is a part of each data object definition.

Note: Additional descriptive information about the element.

2.5.1 QTI Outcomes Processing Data Objects

Table 2.1 describes the data objects that are used in the construction of the QTI outcomes processing elements.

 

Table 2.1 QTI outcomes processing data objects detailed description.

 
No
Name
Explanation
Reqd
Mult
Type
Note
1 outcomes_processing Accumulated outcomes processing and feedback applied within Sections and Assessments. O n
 

 
1.1 scoremodel The type of scoring model being adopted. O
 
CDATA string describing the model.
String 32 chars.
Default string is "SumofScores".
1.2 qticomment The comments used to annotate the XML file. O
 

 
Comments should be used to aid human readability of the XML file itself.
1.2.1 xml:lang The language that is being used for the information. O
 
String 32 chars. The language entries will be defined as per the ISO639 and ISO3166 standards.
1.3 outcomes To create the variables required for the assessment accumulated scores. M
 

 
The assessment accumulated processing variables group.
1.3.1 qticomment As per structure 1.2.
1.3.2 decvar Declaration of a variable to be used for scoring. M n
 
Each type of variable must be declared before it is used.
1.3.2.1 varname The name of the variable to be declared. M
 
String
256 chars.
Default is set as 'SCORE".
1.3.2.2 vartype The type of variable. M
 
Enumerated:
String
Integer (default)
Decimal
Scientific
Boolean
Enumerated
Set
Default is set to 'Integer'.
1.3.2.3 defaultval The default value for the variable. O
 
Numerical 32 chars
String 32 chars
True/False
Can be set to any value. Default is set to '0'.
1.3.2.4 minvalue The minimum value permitted for a numeric score. O
 
String 32chars. Applies to the value of the score after all of the item processing has been completed.
1.3.2.5 maxvalue The maximum score permitted for a numeric score. O
 
String 32chars. Applies to the value of the score after all of the item processing has been completed.
1.3.2.6 cutvalue The value above which the participant will have been defined to have mastery of the subject. O
 
String 32chars. The type of this cut value is set by the variable type.
1.3.2.7 members The set of enumerated values. O
 
String 1024chars. This is a comma separated list without enclosing parentheses.
1.3.3 interpretvar The interpretation to be applied to the variable in terms relevant to an actor. O n
 
At present this element will be a comment string however it will be further developed in version 1.2.
1.3.3.1 varname The name of the input variable being described. O
 
As per structure 1.3.2.1.
1.3.3.2 view The view to which the interpretation is applied. O n Enumerated:
All (default)
Administrator
AdminAuthority
Assessor
Author
Candidate
Invigilator
Proctor
Psychometrician
Scorer
Tutor
The 'All' view is the default value.
1.4 objects_condition This contains the conditions that are applied to define the ways in which the outcomes variables are combined to create the aggregated value. O n
 

 
1.4.1 qticomment As per structure 1.2.
1.4.2 outcomes_metadata Contains the rules that are applied to the IMS QTI-specific and/or IMS Meta-data meta-data fields of the object to decide if the object scoring is to be aggregated. C
 
String
1-256 chars.

 
1.4.2.1 mdname Identifies the IMS QTI-specific or IMS Meta-data field that is to be used for the aggregation rule. M
 
String
1-64 chars.

 
1.4.2.2 mdoperator Identifies the nature of the meta-data field comparison that is to be applied. M
 
Enumerated list:
EQ
NEQ
LT
LTE
GT
GTE

 
1.4.3 and_objects Contains the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'. C
 

 
See Appendix B for the logic rules.
1.4.3.1 outcomes_metadata As per structure 1.4.2
1.4.3.2 and_objects As per structure 1.4.3
1.4.3.3 or_objects As per structure 1.4.4
1.4.3.4 not_objects As per structure 1.4.5
1.4.4 or_objects Contains the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'. C
 

 
See Appendix B for the logic rules.
1.4.4.1 outcomes_metadata As per structure 1.4.2
1.4.4.2 and_objects As per structure 1.4.3
1.4.4.3 or_objects As per structure 1.4.4
1.4.4.4 not_objects As per structure 1.4.5
1.4.5 not_objects Contains the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'. C
 

 
This element contains only ONE of the sub-elements.
See Appendix B for the logic rules.
1.4.5.1 outcomes_metadata As per structure 1.4.2
1.4.5.2 and_objects As per structure 1.4.3
1.4.5.3 or_objects As per structure 1.4.4
1.4.5.4 not_objects As per structure 1.4.5
1.4.6 objects_parameter Contains the value of a particular parameter that is to be used by the corresponding scoring algorithm variable selection. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters. O n
 
These parameters only apply to the objects within the containing <objects_condition> element.
1.4.6.1 pname The name of the parameter that is being defined. M
 
String
256 chars.
Parameters of the same name may have different significance to different algorithms.
1.4.7 map_input This element is used to re-map the default input variable to another variable. O n
 

 
1.4.7.1 varname The name of the input variable whose name is to be remapped. O
 
As per structure 1.3.2.1.
1.4.8 objectscond_extension The objects condition extension facility. O n ANY
 
1.5 processing_parameter The value of a parameter that is passed to the scoring algorithm that is being used. O n String
128 chars.
These parameters apply to the algorithm as a whole and not to specific objects operated on by the algorithm.
1.5.1 pname The name of the parameter that is being defined. M
 
String
256 chars.
Parameters of the same name may have different significance to different algorithms.
1.6 map_output This is used to remap the result of the scoring algorithm from the default/named variable to another named variable. O n String
256 chars.
The new variable name should have been declared in the <outcomes> element of the parent object. This remapping operates for all of the associated variable names e.g. the *.min , *.max and *.normalized variables.
1.6.1 varname The name of the output variable whose name is to be remapped. O
 
As per structure 1.3.2.1.
1.7 outcomes_feedback_test Contains the tests to be applied to determine if any and the type of feedback to be presented. O n
 

 
1.7.1 title The title of the feedback test. O
 
String
256 chars.

 
1.7.2 test_variable The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied. M
 

 

 
1.7.2.1 variable_test The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied. C
 

 

 
1.7.2.1.1 varname The name of the variable whose state is to be tested. O
 
As per structure 1.3.2.1.
1.7.2.1.2 testoperator Identifies the nature of the variable comparison that is to be applied. M
 
Enumerated list:
EQ
NEQ
LT
LTE
GT
GTE

 
1.7.2.2 and_test This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator. C
 

 

 
1.7.2.2.1 variable_test As per structure 1.7.2.1.
1.7.2.2.2 and_test As per structure 1.7.2.2.
1.7.2.2.3 or_test As per structure 1.7.2.3.
1.7.2.2.4 not_test As per structure 1.7.2.4.
1.7.2.3 or_test This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator. C
 

 

 
1.7.2.3.1 variable_test As per structure 1.7.2.1.
1.7.2.3.2 and_test As per structure 1.7.2.2.
1.7.2.3.3 or_test As per structure 1.7.2.3.
1.7.2.3.4 not_test As per structure 1.7.2.4.
1.7.2.4 not_test This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator. C
 

 
Only one of the contained sub-elements can be used within each usage of the NOT operator.
1.7.2.4.1 variable_test As per structure 1.7.2.1.
1.7.2.4.2 and_test As per structure 1.7.2.2.
1.7.2.4.3 or_test As per structure 1.7.2.3.
1.7.2.4.4 not_test As per structure 1.7.2.4.
1.7.3 displayfeedback The trigger for displaying feedback. M n
 

 
1.7.3.1 feedbacktype The type of feedback to be displayed. M
 
Enumerated:
Response (default)
Solution
Hint
The default value is 'Response'.
1.7.3.2 linkrefid The identifier of the material to be referenced. M
 
String 32 chars. Consistency checking is beyond the scope of this specification. Usage rules are given in the Q&TI Best Practice Guide.

3. XML Binding

3.1 outcomes_processing> Elements

Description: The <outcomes_processing> element is the container for all of the outcomes processing instructions for Assessments and Sections. Multiple outcomes processing containers can be use when multiple scoring algorithms are to be applied to produce the aggregated outcomes. If multiple outcomes_processing elements are supplied, it is the intention that all should be run and the outcomes from all of them together should be reported as the outcomes of the enclosing Section or Assessment. The outcome variables defined by each <outcomes_processing> element should be unique across all outcomes_processing elements defined by a section. In particular, it is an error for multiple <outcomes_processing> elements to set the same outcome variable and the results will be undefined.

The <outcomes_processing> element structure

 

Figure 3.1 The <outcomes_processing> element structure.

Multiplicity: The <outcomes_processing> occurs zero or more times within the <assessment> and <section> elements.

Attributes:

  • scoremodel (optional. Default='SumofScores'). The type of the scoring algorithm that is to be used.
    Data-type = String (max of 32 chars).

Elements:

  • qticomment
  • outcomes
  • objects_condition
  • processing_parameter
  • map_outcome
  • outcomes_feedback_test

Example:

<outcomes_processing scoremodel="SumofScores">
  <outcomes>
    <decvar/>
  </outcomes>
</outcomes_processing>

3.1.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes processing structure as a whole.

Multiplicity: Occurs zero or once within the <outcomes_processing> element.

Attributes:

  • xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
    Data-type = string.

3.1.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0).

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: See sub-section 3.2.

3.1.3 <objects_condition> Element

Description: This element contains the conditions that are applied to define the ways in which the outcomes variables are combined to create the aggregated value using the in-built set of algorithm definitions.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes: See sub-section 3.3.

3.1.4 <processing_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters.

Multiplicity: Occurs zero or more times within the <outcomes_processing> element.

Attributes:

  • pname (mandatory). The name of the proprietary parameterized value that is to be used by the scoring algorithm.
    Data-type = string (1-64 chars).

3.1.5 <map_output> Element

Description: This element is used to re-map the named variable to another named variable (given in the body of the element). The target variable name must have been declared using <decvar> in the <outcomes> element of the enclosing <outcomes_processing> element. When a variable is remapped, all of its derived variables are to be remapped as well. Thus if remapping 'SCORE' to 'myScore', 'SCORE.min', 'SCORE.max' and 'SCORE.normalized' would be remapped to 'myScore.min', 'myScore.max' and 'myScore.normalized' respectively.
Data-type = string (1-256 chars).

Multiplicity: This occurs zero or many times within the <outcomes_condition> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable that is normally used by the scoring algorithm e.g. 'SCORE', 'CORRECT', etc. The default name is 'SCORE' but the variables SCORE.min, SCORE.max and SCORE.normalized are also to be remapped.
    Data-type = String (max of 256 chars).

3.1.6 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: See sub-section 3.4.

3.2 <outcomes> Element

Description: The <outcomes> element contains all of the variable declarations that are to be made available to the scoring algorithm. Each variable is declared using the <decvar> element apart from the default variable called 'SCORE' that is an integer and has a default value of zero (0). The declaration of each variable is accompanied by the implicit declaration of the <vname>.min, <vname>.max and <vname>.normalized variables. These are used to store the minimum value, maximum value and normalized value (in the range zero to one) of the scoring variable.

<outcomes> elements

 

Figure 3.2 <outcomes> elements.

Multiplicity: This occurs once within the <outcomes_processing> element.

Attributes: None.

Elements:

  • qticomment
  • decvar
  • interpretvar

3.2.1 <qticomment> Element

Description: This element contains the comments that are relevant to the outcomes.

Multiplicity: Occurs zero or once within the <outcomes> element.

Attributes:

  • xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
    Data-type = string (1-2048 chars).

3.2.2 <decvar> Element

Description: The <decvar> element permits the declaration of the scoring variables. The default name is 'SCORE' but the variables SCORE.min, SCORE.max and SCORE.normalized are also declared and made available from the scoring algorithms. A <decvar> with no attributes is assumed to define the integer variable 'SCORE' with a default value of zero.

Multiplicity: This occurs one or more times within the <outcomes> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable that is to be declared.
    Data-type = String (max of 256 chars).
  • vartype (required. Enumerated list: String, Decimal, Scientific, Boolean, Integer, Enumerated, Set. Default=Integer). The type of the variable declared.
    Data-type = Enumerated list.
  • defaultval (optional). The default value to which the variable is to be initialized.
    Data-type = String (max of 16 chars).
  • cutvalue (optional). The value/grade above which the subject is considered to have been mastered.
    Data-type = String (max of 16 chars).
  • minvalue (optional). The minimum value permitted for a numeric score.
    Data-type = String (max of 32 chars).
  • maxvalue (optional). The maximum value permitted for a numeric score.
    Data-type = String (max of 32 chars).
  • members (optional). The set of enumerated values that constitute the member.
    Data-type = String (max of 1024 chars).

3.2.3 <interpretvar> Element

Description: The <interpretvar> element is used to provide statistical interpretation information about the associated variables.

Multiplicity: This occurs zero or more times within the <outcomes> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable whose interpretation details are to be described. The default name is 'SCORE'.
    Data-type = String (max of 256 chars).
  • view (optional with selection from the enumerated list of: All, Administrator, AdminAuthority, Assessor, Author, Candidate, InvigilatorProctor, Psychometrician, Scorer, Tutor. Default=All). The view defines the scope for the display of the associated information i.e. to whom the material can be presented.
    Data-type = Enumerated list.

Another description of this element and its sub-components is given in IMS QTI: ASI XML Binding V1.2
[QTI, 02b].

3.3 <objects_condition>

Description: Each <objects_condition> element defines a subset of the objects (Items and Sections) selected by the selection algorithm that are to be used in this score. This allows for the construction of subscores. If no <outcomes_metadata> element is present within the <objects_conditions> element to select a subset of objects, then the <objects_conditions> applies to all objects selected by the selection algorithm. If multiple <objects_conditions> are given within a single <outcomes_processing> element, then the algorithm is applied to the union of all objects selected. The <objects_parameter>, <map_input> and <objectscond_extension> elements within the <objects_condition> element apply only to those objects selected in the condition. If an object is selected by more than one <objects_condition> element, it should receive parameter assignments and input mapping as if it were the first element so selected. These conditions include the identification and mapping of the variables for input.

<cke:objects_condition> elements

 

Figure 3.3 <objects_condition> elements.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes: None.

Elements:

  • qticomment
  • outcomes_metadata
  • and_objects
  • or_objects
  • not_objects
  • objects_parameter
  • map_input
  • objectscond_extension

3.3.1 <qticomment> Element

Description: This element contains the comments that are relevant to the presentation of the Item.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes:

  • xml:lang (optional). Identifies the language that is to be used within the instance. The vocabulary is defined as per the XML W3C specification.
    Data-type = string (1-2048 chars).

3.3.2 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the QTI-specific and/or IMS Meta-data meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule. This data is a string of up to 64 characters length.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.5.

Elements: None.

3.3.3 <and_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.6.

3.3.4 <or_out> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.7.

3.3.5 <not_out> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or once within the <objects_condition> element.

Attributes: See Sub-section 3.8.

3.3.6 <objects_parameter> Element

Description: This element contains the value of a particular parameter that is to be used by the corresponding scoring algorithm variable selection. Each parameter has a particular meaning to each scoring algorithm i.e. there is no established vocabulary for these parameters. These parameters are applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outomes_condition> select the same object, the parameters are taken from the first <outcomes_condition> that selects the element.

Multiplicity: Occurs zero or more times within the <objects_condition> element.

Attributes:

  • pname (mandatory). The name of the proprietary parameterized value that is to be used by the scoring algorithm.
    Data-type = string (1-64 chars).

3.3.7 <map_input> Element

Description: This element is used to re-map the named input variable to another variable. The default variable names are derived from the type of scoring algorithm identified in <outcomes_processing> element. The target variable name must have been declared in the evaluation objects that undergoing aggregation and must be of the same type as the default variable. When a variable is remapped, all of its derived variables are to be remapped as well. The input mapping is applied only to the objects selected by the enclosing <outcomes_condition> element. If multiple <outcomes_condition> select the same object, the input mapping used is the one contained in the first <outcomes_condition> that selects the element.
Data-type = String (1-256 chars).

Multiplicity: This occurs zero or many times within the <objects_condition> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable that is normally used in the scoring algorithm e.g. 'SCORE', 'COUNT', etc.
    Data-type = String (max of 256 chars).

3.3.8 <objectscond_extension> Element

Description: This element contains the proprietary extensions that can be used to extend the functionally capabilities of the <outcomes_condition> element.

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes: None.

3.4 <outcomes_feedback_test> Element

Description: The <outcomes_feedback> element contains the tests to be applied to determine if any and the type of feedback to be presented. This feedback could include information about passing the assessment etc.

<outcomes_feedback_test> elements

 

Figure 3.4 <outcomes_feedback_test> elements.

Multiplicity: This occurs zero or many times within the <outcomes_processing> element.

Attributes:

  • title (optional). The title of the feedback test.
    Data-type = String (max of 256 chars).

Elements:

  • test_variable
  • displayfeedback

3.4.1 <test_variable> Elements

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs once within the <outcomes_feedback_test> element.

Attributes: See Sub-section 3.9.

3.4.2 <displayfeedback> Element

Description: The <displayfeedback> element is responsible for assigning an associated feedback to the aggregated scoring if the 'True' state results.

Multiplicity: This occurs zero or more times within the <outcomes_feedback_test> element.

Attributes:

  • feedbacktype (optional - enumerated list: Response, Solution, Hint. Default = Response). The type of feedback that has been triggered by the associated outcomes condition. Within the Assessment and Section feedback elements only the 'Response' type of feedback is supported.
    Data-type = Enumerated list.
  • linkrefid (required). The identifier of the associated feedback. An <assessfeedback> or <sectionfeedback> element must exist with this identifier as defined by the 'ident' attribute.
    Data-type = String (max of 256 chars).

3.5 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the IMS QTI-specific and/or IMS Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = String (1-64 chars).

Multiplicity: Occurs zero or once within the <objects_condition> element.

Attributes:

  • mdname (mandatory). Identifies the QTI-specific or IMS Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
    Data-type = string (1-256 chars).
  • mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
    Data-type = string (1-16 chars).

Elements: None.

Example:

<objects_condition>
  <outcomes_metadata mdname="qmd_topic" mdoperator="EQ">Trigonometry</outcomes_metadata>
</objects_condition>

3.6 <and_objects> Element

Description: The <and_objects> element supports the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'. The logic rules for this element are described in Appendix B.

<and_objects> elements

 

Figure 3.5 <and_objects> elements.

Multiplicity: This occurs zero or once within the <objects_condition>, and zero or many times within the <and_objects>, <or_objects> and <not_objects> elements.

Attributes: None.

Elements:

  • outcomes_metadata
  • and_objects
  • or_objects
  • not_objects

3.6.1 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the IMS QTI-specific and/or IMS Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = string (1-64 chars).

Multiplicity: Occurs zero or many times within the <and_objects> element.

Attributes:

  • mdname (mandatory). Identifies the IMS QTI-specific or IMS Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
    Data-type = string (1-64 chars).
  • mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
    Data-type = string (1-16 chars).

Elements: None.

3.6.2 <and_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or many times within the <and_objects> element.

Attributes: See Sub-section 3.6.

3.6.3 <or_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or many times within the <and_objects> element.

Attributes: See Sub-section 3.7.

3.6.4 <not_objects> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or many times within the <and_objects> element.

Attributes: See Sub-section 3.8.

3.7 <or_objects> Element

Description: The <or_objects> element supports the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'. The logic rules are described in Appendix B.

Multiplicity: This occurs zero or once within the <objects_condition>, and zero or more times within the <and_objects>, <or_objects> and <not_objects> element.

Attributes: None.

Elements:

  • outcomes_metadata
  • and_objects
  • or_objects
  • not_objects

<or_objects> elements

 

Figure 3.6 <or_objects> elements.

3.7.1 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the IMS QTI-specific and/or IMS Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = string (1-64 chars).

Multiplicity: Occurs zero or many times within the <or_objects> element.

Attributes:

  • mdname (mandatory). Identifies the QTI-specific or IMS Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
    Data-type = string (1-64 chars).
  • mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
    Data-type = string (1-16 chars).

Elements: None.

3.7.2 <and_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or many times within the <or_objects> element.

Attributes: See Sub-section 3.6.

3.7.3 <or_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or many times within the <or_objects> element.

Attributes: See Sub-section 3.7.

3.7.4 <not_objects> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or many times within the <or_objects> element.

Attributes: See Sub-section 3.8.

3.8 <not_objects> Element

Description: The <not_objects> element supports the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'. The logic rules are listed in Appendix B.

<not_objects> elements.

 

Figure 3.7 <not_objects> elements.

Multiplicity: This occurs zero or once within the <objects_condition>, and zero or many times within the <and_objects>, <or_objects> and <not_objects> element.

Attributes: None.

Elements:

  • outcomes_metadata
  • and_objects
  • or_objects
  • not_objects

3.8.1 <outcomes_metadata> Element

Description: This element defines the rule that is applied to the QTI-specific and/or IMS Meta-data fields of the object to decide if the object scoring is to be aggregated. The content contains the value of the meta-data field that is being tested for within the rule.
Data-type = string (1-64 chars).

Multiplicity: Occurs zero or once within the <not_objects> element.

Attributes:

  • mdname (mandatory). Identifies the IMS QTI-specific or IMS Meta-data field that is to be used for the aggregation rule. No validation check is made in the instance on the existence or otherwise of this field.
    Data-type = string (1-64 chars).
  • mdoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the meta-data field comparison that is to be applied.
    Data-type = string (1-16 chars).

Elements: None.

3.8.2 <and_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'AND' operator. The object is selected for aggregation if all of the contained rules are 'True'.

Multiplicity: This occurs zero or once within the <not_objects> element.

Attributes: See Sub-section 3.6.

3.8.3 <or_objects> Element

Description: This element allows the construction of complex score condition rules to be built based upon the logical 'OR' operator. The object is selected for aggregation if at least one of the contained rules is 'True'.

Multiplicity: This occurs zero or many times within the <not_objects> element.

Attributes: See Sub-section 3.7.

3.8.4 <not_objects> Element

Description: This element allows the construction of complex rules to be built based upon the logical 'NOT' operator. The object is selected for aggregation if the contained rule(s) is 'False'.

Multiplicity: This occurs zero or many times within the <not_objects> element.

Attributes: See Sub-section 3.8.

3.9 <test_variable> Element

Description: The <test_variable> element contains the conditions that are applied to a defined set of input scoring variables to determine if feedback is to be presented. Complex test structures can be constructed using the associated logic elements.

<test_variable> elements

 

Figure 3.8 <test_variable> elements.

Multiplicity: This occurs once within the <outcomes_feedback_test> element.

Attributes: None.

Elements:

  • variable_test
  • and_test
  • or_test
  • not_test

3.9.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or once within the <test_variable> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable whose condition is to be tested. The default name is 'SCORE'.
    Data-type = String (max of 256 chars).
  • testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
    Data-type = string (1-16 chars).

3.9.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or once within the <test_variable> element.

Attributes: See Sub-section 3.10.

3.9.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or many times within the <test_variable> element.

Attributes: See Sub-section 3.11.

3.9.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or once within the <test_variable> element.

Attributes: See Sub-section 3.12.

3.10 <and_test> Element

Description: The <and_test> element is used to define variable test conditions all of which must be defined as 'True' to result in the output variables being assigned the corresponding values.

<and_test> elements

 

Figure 3.9 <and_test> elements.

Multiplicity: This occurs zero or once within the <test_variable> element, and zero or more times in the <and_test>, <or_test> and <not_test> elements.

Attributes: None.

Elements:

  • variable_test
  • and_test
  • or_test
  • not_test

3.10.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or more times within the <and_test> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable that is to be tested. The default name is 'SCORE'.
    Data-type = String (max of 256 chars).
  • testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
    Data-type = string (1-16 chars).

3.10.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or many times within the <and_test> element.

Attributes: See Sub-section 3.10.

3.10.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or many times within the <and_test> element.

Attributes: See Sub-section 3.11.

3.10.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or many times within the <and_test> element.

Attributes: See Sub-section 3.12.

3.11 <or_test> Element

Description: The <or_test> element is used to define variable test conditions at least one of which must be defined as 'True' to result in the output variables being assigned the corresponding values.

<or_test> elements

 

Figure 3.10 <or_test> elements.

Multiplicity: This occurs zero or more times within the <test_variable> element.

Attributes: None.

Elements:

  • variable_test
  • and_test
  • or_test
  • not_test

3.11.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or more times within the <or_test> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable that is to be tested. The default name is 'SCORE'.
    Data-type = String (max of 256 chars).
  • testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
    Data-type = string (1-16 chars).

3.11.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or many times within the <or_test> element.

Attributes: See Sub-section 3.10.

3.11.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or many times within the <or_test> element.

Attributes: See Sub-section 3.11.

3.11.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or many times within the <or_test> element.

Attributes: See Sub-section 3.12.

3.12 <not_test> Element

Description: The <not_test> element is used to define variable test conditions that result in the condition being defined as 'False' to result in the output variables being assigned the corresponding values.

<not_test> elements

 

Figure 3.11 <not_test> elements.

Multiplicity: This occurs zero or more times within the <test_var> element.

Attributes: None.

Elements:

  • variable_test
  • and_test
  • or_test
  • not_test

3.12.1 <variable_test> Element

Description: The conditional test that is to be applied to the aggregated score variables. A wide range of separate and combinatorial tests can be applied.

Multiplicity: This occurs zero or once within the <not_test> element.

Attributes:

  • varname (optional. Default = 'SCORE'). The name of the variable that is to be tested. The default name is 'SCORE'.
    Data-type = String (max of 256 chars).
  • testoperator (mandatory with selection from the enumerated list of: EQ, NEQ, LT, LTE, GT, GTE). Identifies the nature of the variable comparison that is to be applied.
    Data-type = string (1-16 chars).

3.12.2 <and_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'AND' operator.

Multiplicity: This occurs zero or once within the <not_test> element.

Attributes: See Sub-section 3.10.

3.12.3 <or_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'OR' operator.

Multiplicity: This occurs zero or once within the <not_test> element.

Attributes: See Sub-section 3.11.

3.12.4 <not_test> Element

Description: This element allows the construction of complex variable test rules to be built based upon the logical 'NOT' operator.

Multiplicity: This occurs zero or many times within the <not_test> element.

Attributes: See Sub-section 3.12.

4. Best Practice & Implementation Guide

4.1 Overall Data Model

The overall IMS QTI ASI Outcomes Processing data model is shown in Figure 4.1.

Overall ASI outcomes processing data model

 

Figure 4.1 Overall ASI outcomes processing data model.

4.2 Relationship to the Other IMS Specifications

The relationship of this 'Outcomes Processing' specification to the other parts of the IMS QTI specifications is:

  • The 'Outcomes Processing' root element is a part of the Assessment and Section core data objects in the IMS QTI ASI: Information Model and IMS QTI ASI: XML Binding. The <response_processing> element (defined within the QTI ASI Information model and XML Binding documents) defines the variables that are used as inputs to the response processing when the algorithm operates on items. Similarly, the <outcomes_processing> element (defined in this document) defines the variables that are used as inputs when the outcomes processing operates on sections. Note that in predefined algorithms, these operate on either an integer variable 'SCORE' or a boolean variable 'CORRECT'. To be used with the maximum number of outcomes processing algorithms an item's <response_processing> element should set both variables. Also, some algorithms treat items that were not attempted or not presented differently from those which were attempted so there must be some way of querying the items attempted or presented status. Finally, other algorithms assume that a default value for the outcome variable (usually a null score) is available even if the item was never presented. All of these data can be extracted from a IMS QTI results report, however, most implementations will probably use equivalent internal data structures;
  • The 'Selection & Ordering' component of the IMS QTI specification [QTI, 02d] interacts with the 'Outcomes Processing' part. This interaction is a result of the fact that the aggregated score for a Section and/or Assessment depends upon which Items and Section were selected for presentation to the participant;
  • The IMS QTI Results Reporting specification contains the description of how the results are recorded for a participant that has undertaken an evaluation. These results will depend upon the aggregated scores derived from the outcomes processing mechanism. The QTI Results Reporting specification describes how the results from any IMS QTI-compliant Assessment, Section and Item can be reported.

4.3 Basic Example XML Instances

The following set of basic examples demonstrates how the different use-cases are supported. Table 4.1 lists the examples used to show the operation of each of the outcomes processing algorithms.

 

Table 4.1 The examples demonstrating each outcomes processing algorithm.

 
Algorithm
Example

 
1
2
3
4
5
6
7
8
9
Number Correct
*
 
 
 
 
 
 
 
 
Number Correct Attempted
*
 
 
 
 
 
 
 
 
Weighted Number Correct
 
*
 
 
 
 
 
 
 
Weighted Number Correct Attempted
 
*
 
 
 
 
 
 
 
Parameter Weighted Number Correct
 
*
 
 
 
 
 
 
 
Parameter Weighted Number Correct Attempted
 
*
 
 
 
 
 
 
 
Sum of Scores
 
 
*
 
 
 
*
*
*
Sum of Scores Attempted
 
 
*
 
 
 
 
 
 
Weighted Sum of Scores
 
 
 
*
 
 
 
 
 
Weighted Sum of Scores Attempted
 
 
 
*
 
 
 
 
 
Parameter Weighted Sum of Scores
 
 
 
*
 
 
 
 
 
Parameter Weighted Sum of Scores Attempted
 
 
 
*
 
 
 
 
 
Best K from N
 
 
 
 
*
 
 
 
 
Guessing Penalty
 
 
 
 
 
*
 
 
 

4.3.1 Example ('Number Correct' and 'Number Correct Attempted')

A Section consists of 10 Items each of which has a 'score' assigned to its default scoring boolean variable (CORRECT) that is used to denote if the Item has been answered correctly i.e. 'True'. The number of correct responses is assigned to the default score variable for the Section (COUNT). The outcomes processing algorithms for calculating the number of correct responses both in terms of the Items presented and the Items presented and attempted are given below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S01" ident = "IMS_QTIV1p2_S_OUT_01">
    <outcomes_processing scoremodel = "NumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
    </outcomes_processing>
    <outcomes_processing scoremodel = "NumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_Attempted" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_Attempted</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
111
112
113
114
115
116
117
118
119
120
121
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>


The 'NumberCorrect' scoring algorithm that is supported for the Section is:

  COUNT = 0
  COUNT.min = 0
  COUNT.max = 0
  COUNT.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(CORRECT) = True
    then
      begin
        COUNT.max = COUNT.max + 1
        if item.CORRECT = True
        then COUNT = COUNT + 1
      endif
    enddo
  COUNT.normalized = COUNT / COUNT.max

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT = 7

COUNT.min = 0

COUNT.max = 10

COUNT.normalized = 0.7

The 'NumberCorrectAttempted' scoring algorithm that is supported for the Section is (the variable 'COUNT_Attempted' must be used to ensure that there is no scoping clash when two algorithms attempt to operate on the COUNT variable):

  COUNT_Attempted = 0
  COUNT_Attempted.min = 0
  COUNT_Attempted.max = 0
  COUNT_Attempted.normalized = 0
  for ALL_ATTEMPTED_ITEMS do
    if ITEM_HAS_VARIABLE(CORRECT) = True
    then
      begin
        COUNT_Attempted.max = COUNT_Attempted.max + 1
        if item.CORRECT = True
        then COUNT_Attempted = COUNT_Attempted + 1
      endif
    enddo
  COUNT_Attempted.normalized = COUNT_Attempted / COUNT_Attempted.max

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT_Attempted = 7

COUNT_Attempted.min = 0

COUNT_Attempted.max = 8

COUNT_Attempted.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample01.xml'. All more complete description of the 'Number Correct' and 'NumberCorrectAttempted' scoring algorithms is given in Appendices B1 and B2.

4.3.2 Example ('Weighted Number Correct', 'Weighted Number Correct Attempted', 'Parameter Weighted Number Correct' and 'Parameter Weighted Number Correct Attempted')

A Section consists of 10 Items (a mixture of implicit and referenced) and each Item has a 'score' assigned to its default scoring boolean variable (CORRECT) that is used to denote if the Item has been answered correctly i.e. 'True'. The four different weighting algorithms are presented using the appropriate weighting calculation mechanism. The number of correct responses is assigned to the default score variable for the Section (COUNT).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S02" ident = "IMS_QTIV1p2_S_OUT_02">
    <outcomes_processing scoremodel = "WeightedNumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_WNC" vartype = "Integer" 
             minvalue = "0" maxvalue = "15" cutvalue = "10"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_WNC</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "WeightedNumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_WNCA" vartype = "Integer" 
             minvalue = "0" maxvalue = "15" cutvalue = "10"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_WNCA</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedNumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_PWNC" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="COUNT">COUNT_PWNC</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedNumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_PWNC" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="COUNT">COUNT_PWNC</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>


The 'WeightedNumberCorrect' scoring algorithm that is supported for the Section is:

  COUNT_WNC = 0
  COUNT_WNC.min = 0
  COUNT_WNC.max = 0
  COUNT_WNC.normalized = 0
  for ALL_SELECTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = item(qmd_weighting)
        COUNT_WNC.max = COUNT_WNC.max + item.weighting
        if item.CORRECT = True
        then COUNT_WNC = COUNT_WNC + item.weighting
      endif
    enddo
  COUNT_WNC.normalized = COUNT_WNC / COUNT_WNC.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_WNC = 10

COUNT_WNC.min = 0

COUNT_WNC.max = 15

COUNT_WNC.normalized = 0.667

The 'WeightedNumberCorrectAttempted' scoring algorithm that is supported for the Section is:

  COUNT_WNCA = 0
  COUNT_WNCA.min = 0
  COUNT_WNCA.max = 0
  COUNT_WNCA.normalized = 0
  for ALL_ATTEMPTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = item(qmd_weighting)
        COUNT_WNCA.max = COUNT_WNCA.max + item.weighting
        if item.CORRECT = True
        then COUNT_WNCA = COUNT_WNCA + item.weighting
      endif
    enddo
  COUNT_WNCA.normalized = COUNT_WNCA / COUNT_WNCA.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_WCNA = 10

COUNT_WCNA.min = 0

COUNT_WCNA.max = 12

COUNT_WCNA.normalized = 0.833

The 'ParameterWeightedNumberCorrect' scoring algorithm that is supported for the Section is:

  COUNT_PWNC = 0
  COUNT_PWNC.min = 0
  COUNT_PWNC.max = 0
  COUNT_PWNC.normalized = 0
  for ALL_SELECTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = objects_parameter(qmd_weighting)
        COUNT_PWNC.max = COUNT.max + item.weighting
        if item.CORRECT = True
        then COUNT_PWNC = COUNT_PWNC + item.weighting
      endif
    enddo
  COUNT_PWNC.normalized = COUNT_PWNC / COUNT_PWNC.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_PWNC = 14

COUNT_PWNC.min = 0

COUNT_PWNC.max = 20

COUNT_PWNC.normalized = 0.7

The 'ParameterWeightedNumberCorrectAttempted' scoring algorithm that is supported for the Section is:

  COUNT_PWNCA = 0
  COUNT_PWNCA.min = 0
  COUNT_PWNCA.max = 0
  COUNT_PWNCA.normalized = 0
  for ALL_ATTEMPTED_ITEMS do 
    if ITEM_HAS_VARIABLE(COUNT) = True
    then
      begin
        item.weighting = objects_parameter(qmd_weighting)
        COUNT_PWNCA.max = COUNT_PWNCA.max + item.weighting
        if item.CORRECT = True
        then COUNT_PWNCA = COUNT_PWNCA + item.weighting
      endif
    enddo
  COUNT_PWNCA.normalized = COUNT_PWNCA / COUNT_PWNCA.max

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

COUNT_PWNCA = 14

COUNT_PWNCA.min = 0

COUNT_PWNCA.max = 16

COUNT_PWNCA.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample02.xml'. A more complete description of the weighted number correct' scoring algorithms is given in Appendices B3, B4, B5 and B6.

4.3.3 Example ('SumofScores' and 'SumofScoresAttempted')

A Section consists of 10 Items each of which has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and 0 for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE). The outcomes processing algorithms for calculating the sum of scores both in terms of the Items presented and the Items presented and attempted are given below.

1
2
3
4
5
6
7
8
9
10
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S03" ident = "IMS_QTIV1p2_S_OUT_03">
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
    </outcomes_processing>
    <outcomes_processing scoremodel = "SumofScoresAttempted">
      <outcomes>
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="SCORE">SCORE_Attempted</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>                                    </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'SumOfScores' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
    then
      begin
        SCORE.min = SCORE.min + item.SCORE.minvalue
        SCORE.max = SCORE.max + item.SCORE.maxvalue
        SCORE = SCORE + item.SCORE
      endif
    enddo
  SCORE.normalized = (SCORE-SCORE.min) / (SCORE.max-SCORE.min)

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

SCORE = 7

SCORE.min = 0

SCORE.max = 10

SCORE.normalized = 0.7

The 'SumofScoresAttempted' scoring algorithm that is supported for the Section is (the default output variable name has been remapped to avoid overloading the 'SCORE' variable during the concurrent application of the two scoring algorithms):

  SCORE_Attempted = 0
  SCORE_Attempted.min = 0
  SCORE_Attempted.max = 0
  SCORE_Attempted.normalized = 0.0
  for ALL_ATTEMPTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
    then
      begin
        SCORE_Attempted.min = SCORE_Attempted.min + item.SCORE.minvalue
        SCORE_Attempted.max = SCORE_Attempted.max + item.SCORE.maxvalue
        SCORE_Attempted = SCORE_Attempted + item.SCORE
      endif
    enddo
  SCORE_Attempted.normalized = (SCORE_Attempted - SCORE_Attempted.min) / 
                      (SCORE_Attempted.max - SCORE_Attempted.min)

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

SCORE_Attempted = 7

SCORE_Attempted.min = 0

SCORE_Attempted.max = 8

SCORE_Attempted.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample03.xml'. All more complete description of the 'Sum of Scores' and 'Sum of Scores Attempted' scoring algorithms are given in Appendix B7 and B8.

4.3.4 Example ('Weighted Sum of Scores', 'Weighted Sum of Scores Attempted', 'Parameter Weighted Sum of Scores' and 'Parameter Weighted Sum of Scores Attempted')

A Section consists of 10 Items each of which has a score assigned to its default scoring integer variable (SCORE). A score of '1' is given for a correct answer and '0' for an incorrect answer. The four different weighting algorithms are presented using the appropriate weighting calculation mechanism. The sum of these scores is assigned to the default score variable for the Section (SCORE).

1
2
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S04" ident = "IMS_QTIV1p2_S_OUT_04">
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
    <outcomes_processing scoremodel = "WeightedSumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "19" cutvalue = "12"/>
      </outcomes>
      <map_output varname="SCORE">SCORE_WSOS</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "WeightedSumofScoresAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "15" cutvalue = "10"/>
      </outcomes>
      <map_output varname="SCORE">SCORE_WSOSA</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedSumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="SCORE">SCORE_PWSOS</map_output>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterWeightedSumofScoresAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "20" cutvalue = "10"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2<objects_parameter>
      </objects_condition>
      <map_output varname="SCORE">SCORE_PWSOSA</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>3</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>6</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
201
203
204
205
206
207
208
209
210
211
212
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>


The 'WeightedSumOfScores' scoring algorithm that is supported for the Section is:

  SCORE_WSOS = 0
  SCORE_WSOS.min = 0
  SCORE_WSOS.max = 0
  SCORE_WSOS.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_WSOS.min = SCORE_WSOS.min + (item.SCORE.minvalue * item(qmd_weighting))
        SCORE_WSOS.max = SCORE_WSOS.max + (item.SCORE.maxvalue * item(qmd_weighting))
        SCORE_WSOS = SCORE_WSOS + (item.SCORE * item.qmd_weighting)
      endif
    enddo
  SCORE_WSOS.normalized = (SCORE_WSOS - SCORE_WSOS.min) / (SCORE_WSOS.max - SCORE_WSOS.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_WSOS = 16

SCORE_WSOS.min = 0

SCORE_WSOS.max = 19

SCORE_WSOS.normalized = 0.842

The 'WeightedSumOfScoresAttempted' scoring algorithm that is supported for the Section is:

  SCORE_WSOSA = 0
  SCORE_WSOSA.min = 0
  SCORE_WSOSA.max = 0
  SCORE_WSOSA.normalized = 0.0
  for ALL_ATTEMPTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_WSOSA.min = SCORE_WSOSA.min + (item.SCORE.minvalue * item(qmd_weighting))
        SCORE_WSOSA.max = SCORE_WSOSA.max + (item.SCORE.maxvalue * item(qmd_weighting))
        SCORE_WSOSA = SCORE_WSOSA + (item.SCORE * item(qmd_weighting))
      endif
    enddo
  SCORE_WSOSA.normalized = (SCORE_WSOSA-SCORE_WSOSA.min)/(SCORE_WSOSA.max-SCORE_WSOSA.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_WSOSA = 16

SCORE_WSOSA.min = 0

SCORE_WSOSA.max = 17

SCORE_WSOSA.normalized = 0.941

The 'ParameterWeightedSumOfScores' scoring algorithm that is supported for the Section is:

  SCORE_PWSOS = 0
  SCORE_PWSOS.min = 0
  SCORE_PWSOS.max = 0
  SCORE_PWSOS.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_PWSOS.min = SCORE_PWSOS.min + 
                  (item.SCORE.minvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOS.max = SCORE_PWSOS.max + 
                  (item.SCORE.maxvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOS = SCORE_PWSOS + (item.SCORE * objects_parameter(qmd_weighting))
      endif
    enddo
  SCORE_PWSOS.normalized = (SCORE_PWSOS-SCORE_PWSOS.min)/(SCORE_PWSOS.max-SCORE_PWSOS.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_PWSOS = 14

SCORE_PWSOS.min = 0

SCORE_PWSOS.max = 20

SCORE_PWSOS.normalized = 0.7

The 'ParameterWeightedSumOfScoresAttempted' scoring algorithm that is supported for the Section is:

  SCORE_PWSOSA = 0
  SCORE_PWSOSA.min = 0
  SCORE_PWSOSA.max = 0
  SCORE_PWSOSA.normalized = 0.0
  for ALL_PRESENTED_ITEMS do
    if ITEM_HAS_VARIABLE(SCORE) = True
      begin
        SCORE_PWSOSA.min = SCORE_PWSOSA.min + 
                  (item.SCORE.minvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOSA.max= SCORE_PWSOSA.max + 
                  (item.SCORE.maxvalue * objects_parameter(qmd_weighting))
        SCORE_PWSOSA = SCORE_PWSOSA + (item.SCORE * objects_parameter(qmd_weighting))
      endif
    enddo
  SCORE_PWSOSA.normalized =               (SCORE_PWSOSA-SCORE_PWSOSA.min) /
                  (SCORE_PWSOSA.max-SCORE_PWSOSA.min)

In the case where the first eight Items have been attempted and the first seven have been answered correctly then the final set of Section variable states will be:

SCORE_PWSOSA = 14

SCORE_PWSOSA.min = 0

SCORE_PWSOSA.max = 16

SCORE_PWSOSA.normalized = 0.875

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample04.xml'. All more complete description of the 'Sum of Scores' scoring algorithm is given in Appendices B9, B10, B11 and B12.

4.3.5 Example ('Best K from N')

A Section consists of 10 Items and each has a score assigned to its default scoring integer variable (SCORE). The 'maxvalue' score is given for a correct answer and the 'minvalue' for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE). The aggregated score is set as the best seven scores.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S05" ident = "IMS_QTIV1p2_S_OUT_05">
    <outcomes_processing scoremodel = "BestKofN">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <processing_parameter pname = "BestK">7</processing_parameter>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>                                    </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>


The 'BestKfromN' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  For HIGHEST_SCORES_FROM_PRESENTED_ITEMS(7) and ITEM_HAS_VARIABLE(SCORE) do
    begin
      SCORE.min = SCORE.min + item.SCORE.minvalue
      SCORE.max = SCORE.max + item.SCORE.maxvalue
      SCORE = SCORE + item.SCORE
    Enddo
  SCORE.normalized = (SCORE - SCORE.min) / (SCORE.max - SCORE.min)

In the case where all ten Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

SCORE = 7

SCORE.min = 0

SCORE.max = 7

SCORE.normalized = 1

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample05.xml'. A more detailed description of these scoring algorithms are presented in Appendices B13.

4.3.6 Example ('Guessing Penalty' and 'WeightedGuessingPenalty')

A Section consists of 10 Items each of which has a 'score' assigned to its default scoring boolean variable (CORRECT) that is used to denote if the Item has been answered correctly i.e. 'True'. The number of correct responses is assigned to the default score variable for the Section (COUNT). The outcomes processing algorithms for calculating the number of correct responses both in terms of the Items presented and the Items presented and attempted are given below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S01" ident = "IMS_QTIV1p2_S_OUT_01">
    <outcomes_processing scoremodel = "GuessingPenalty">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Decimal" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
    </outcomes_processing>
    <outcomes_processing scoremodel = "WeightedGuessingPenalty">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT_Weighted" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_Weighted</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
142
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
        </qtimetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_penaltyvalue</fieldlabel>
            <fieldentry>0.2</fieldentry>
        </qtimetadata>
      </itemmetadata>
231
232
233
234
235
236
237
238
239
240
241
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="False" vartype="Boolean" varname="CORRECT"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>


The 'GuessingPenalty' scoring algorithm that is supported for the Section is:

  COUNT = 0
  COUNT.correct = 0
  COUNT.incorrect = 0
  COUNT.unattempted = 0
  for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
    begin
      if child.object(ATTEMPTED) = True 
      then
        begin
          if childobject.CORRECT = True
          then 
            begin
              parentobject.COUNT.correct = parentobject.COUNT.correct + 1
              parentobject.COUNT = parentobject.COUNT + 1
            end
          else 
            begin
              parentobject.COUNT.incorrect = parentobject.COUNT.incorrect + 1
              parentobject.COUNT = parentobject.COUNT - item(qmd_penaltyvalue)
            end
        endif
      else childobject.COUNT.unattempted = childobject.COUNT.unattempted + 1
    enddo

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT = 6.8

COUNT.correct = 7

COUNT.correct = 7

COUNT.incorrect = 1

COUNT.unattempted = 2

The 'WeightedGuessingPenalty' scoring algorithm that is supported for the Section is:

  COUNT_Weighted = 0
  COUNT_Weighted.correct = 0
  COUNT_Weighted.incorrect = 0
  COUNT_Weighted.unattempted = 0
  for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
    begin
      if child.object(ATTEMPTED) = True 
      then
        begin
          if childobject.CORRECT = True
          then 
            begin
              parentobject.COUNT_Weighted.correct = 
                parentobject.COUNT_Weighted.correct + 1
              parentobject.COUNT_Weighted = parentobject.COUNT_Weighted + 1
            end
          else 
            begin
              parentobject.COUNT_Weighted.incorrect = 
                parentobject.COUNT_Weighted.incorrect + 1
              parentobject.COUNT_Weighted = 
              (parentobject.COUNT_Weighted * item(qmd_weighting) - 
              item(qmd_penaltyvalue)
            end
        endif
      else childobject.COUNT_Weighted.unattempted = 
        childobject.COUNT_Weighted.unattempted + 1
    enddo

In the case where eight Items have been attempted and seven have been answered correctly then the final set of Section variable states will be:

COUNT = 13.8

COUNT.correct = 7

COUNT.correct = 7

COUNT.incorrect = 1

COUNT.unattempted = 2

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample06.xml'. All more complete description of these scoring algorithm is given in Appendices B14 and B15.

4.3.7 Example (Remapped Output 'Sum of Scores')

A Section consists of 5 Items (a mixture of implicit and referenced) and each Item has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and -1 for an incorrect answer. The sum of these scores is assigned to the new score variable for the Section (Section_06_SCORE).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S07" ident = "IMS_QTIV1p2_S_OUT_07">
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval= "0" varname= "Section_07_SCORE" vartype= "Integer" 
             minvalue= "0" maxvalue= "10" cutvalue= "6"/>
      </outcomes>
      <map_output varname = "SCORE">Section_07_SCORE</map_output>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The output remapped 'SumofScores' scoring algorithm that is supported for the Section is:

  Section_07_SCORE = 0
  Section_07_SCORE.min = 0
  Section_07_SCORE.max = 0
  Section_07_SCORE.normalized = 0.0
  For ALL_ATTEMPTED_ITEMS and ITEM_HAS_VARIABLE(SCORE) do
    begin
      Section_07_SCORE.min = Section_07_SCORE.min + item.SCORE.minvalue
      Section_07_SCORE.max = Section_07_SCORE.max + item.SCORE.maxvalue
      Section_07_SCORE = Section_07_SCORE + item.SCORE
    Enddo
  Section_07_SCORE.normalized = 
  (Section_07_SCORE - Section_07_SCORE.min)/(Section_07_SCORE.max - Section_07_SCORE.min)

In the case where all ten Items have been attempted and eight have been answered correctly then the final set of output remapped Section variable states will be:

Section_07_SCORE = 8

Section_07_SCORE.min = 0

Section_07_SCORE.max = 10

Section_07_SCORE.normalized = 0.8

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample07.xml'.

4.3.8 Example (Remapped Input 'Sum of Scores')

A Section consists of 5 Items (a mixture of implicit and referenced) and each Item has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and -1 for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S06" ident = "IMS_QTIV1p2_S_OUT_06">
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval= "0" varname= "SCORE" vartype= "Integer" 
             minvalue= "0" maxvalue= "10" cutvalue= "6"/>
      </outcomes>
      <objects_condition>
        <map_input varname = "SCORE">item_SCORE</map_input>
      </objects_condition>
    </outcomes_processing>
    <selection_ordering>
      <selection seltype/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1" varname="item_SCORE"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The input remapped 'SumofScores' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  For ALL_ATTEMPTED_ITEMS and ITEM_HAS_VARIABLE(item_SCORE) do
    begin
      SCORE.min = SCORE.min + item.item_SCORE.minvalue
      SCORE.max = SCORE.max + item.item_SCORE.maxvalue
      SCORE = SCORE + item.item_Score
    Enddo
  SCORE.normalized = (SCORE - SCORE.min)/(SCORE.max - SCORE.min)

In the case where all ten Items have been attempted and eight have been answered correctly then the final set of Section variable states will be:

SCORE = 8

SCORE.min = 0

SCORE.max = 10

SCORE.normalized = 0.8

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample08.xml'.

4.3.9 Example (Parameterized 'Sum of Scores')

A Section consists of 10 Items and each has a score assigned to its default scoring integer variable (SCORE). A score of 1 is given for a correct answer and -1 for an incorrect answer. The sum of these scores is assigned to the default score variable for the Section (SCORE). The intention is to produce a result score for all those Items on the topic that are 'Advanced' and 'Basic'.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S08" ident = "IMS_QTIV1p2_S_OUT_08">
    <outcomes_processing scoremodel = "SumOfScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "6"/>
      </outcomes>
      <objects_condition>
        <or_objects>
          <outcomes_metadata  mdname="qmd_levelofdifficulty" 
                      mdoperator= "EQ">basic</outcomes_metadata>
          <outcomes_metadata  mdname="qmd_levelofdifficulty" 
                      mdoperator="EQ">advanced</outcomes_metadata>
        </or_objects>
      </objects_condition>
    </outcomes_processing>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>3</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_06" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>6</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_07" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>2</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_08" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>basic</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval = "0" minvalue = "0" maxvalue = "1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_09" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_10" maxattempts = "2">
      <itemmetadata>
        <qtimetadata>
          <qtimetadatafield>
            <fieldlabel>qmd_weighting</fieldlabel>
            <fieldentry>1</fieldentry>
          </qtimetadatafield>
        </qtimetadata>
        <qmd_levelofdifficulty>advanced</qmd_levelofdifficulty>
        <qmd_topic>qtiv1p2test</qmd_topic>
      </itemmetadata>
      ...
      <resprocessing>
        <outcomes>
          <decvar defaultval="0" minvalue="0" maxvalue="1"/>
        </outcomes>
        ...
      </resprocessing>
      ...
    </item>
  </section>
</questestinterop>

The 'SumOfScores' scoring algorithm that is supported for the Section is:

  SCORE = 0
  SCORE.min = 0
  SCORE.max = 0
  SCORE.normalized = 0.0
  For ALL_PRESENTED_ITEMS            and ITEM_HAS_VARIABLE(SCORE)
                and 
    (item.metadata.difficulty = basic or item.metadata.difficulty = advanced) 
  do
    begin
      SCORE.min = SCORE.min + item.SCORE.minvalue
      SCORE.max = SCORE.max + item.SCORE.maxvalue
      SCORE = SCORE + item.SCORE
    Enddo
  SCORE.normalized = (SCORE - SCORE.min) / (SCORE.max - SCORE.min)

In the case where all ten Items have been attempted and seven have been answered correctly then the final set of Section variable states will be (the meta-data selections ensure that all Items are selected to contribute to the score aggregation):

SCORE = 7

SCORE.min = 0

SCORE.max = 10

SCORE.normalized = 0.7

This example is available in the file: 'ims_qtiasiv1p2/outcome/basicoutexample09.xml'.

4.4 Advanced Example XML Instances

The following set of advanced examples demonstrates how the different use-cases are supported.

4.4.1 Example (Multiple-choice Quiz)

Multiple-choice quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete (single response) questions whose outcomes are a dichotomous variable e.g. multiple-choice questions. For the sake of being definite we assume there are 5 Items and that we have a defined outcome, "CORRECT" for each Item. This takes on the value "True" if the response was correct and "False' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the percentage of total score obtained by a candidate and (3) the percentage of items actually answered which were correct.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S01" ident = "IMS_QTIV1p2_S_OUTADV_01">
    <outcomes_processing scoremodel = "NumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "SCORE">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "SCORE">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "NumberCorrectAttempted">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
             minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <map_output varname="COUNT">COUNT_Attempted</map_output>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <sectionfeedback ident = "SectionMastery" view = "All">
      <material>
        <mattext>Well done.  You have passed this Topic.</mattext>
      </material>
    </sectionfeedback>
    <sectionfeedback ident = "SectionFail" view = "All">
      <material>
        <mattext>Sorry.  You have failed this topic.</mattext>
      </material>
    </sectionfeedback>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Which one of the following statements is true about the QTI 
                  specification ?</mattext>
          </material>
          <response_lid ident = "LID01" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes" minnumber = "1" maxnumber = "1">
              <flow_label>
                <response_label ident = "A">
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
11
112
113
114
115
116
117
118
119
120
121
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
                  <material>
                    <mattext>It is independent of pedagogy</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>It cannot be extended to support proprietary 
                      features</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>It contains many unnecesary features
                    </mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>It provides only one view for different 
                      types of participants</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <qticomment>To detect the correct answer.</qticomment>
          <conditionvar>
            <varequal respident = "LID01">A</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <qticomment>To detect if the incorrect answer is supplied.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "LID01" case = "No">A</varequal>
            </not>
          </conditionvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "InCorrect"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
        <material>
          <mattext>Correct answer is A.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext xml:lang="en-US" xml:space="default" texttype= "" charset= "">
            Answers B, C and D are incorrect.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>The QTI specification supports the exchange of which data 
              structure ?</mattext>
          </material>
          <response_lid ident = "LID02" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes" minnumber = "1" maxnumber = "1">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>Response</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>Question</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>Item</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>Test</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar varname = "CORRECT" vartype = "Boolean" defaultval = "False"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
281
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
          <qticomment>Correct response detection</qticomment>
          <conditionvar>
            <varequal respident = "LID02">C</varequal>
          </conditionvar>
          <setvar varname = "CORRECT" action = "Set">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
        <respcondition continue = "No">
          <qticomment>Processing for the incorrect answer.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "LID02">C</varequal>
            </not>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct.  Items.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>No.  The correct answer is Item.</mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>What is the maximum number of Assessments can be contained 
              in a single QTI ASI XML instance?</mattext>
          </material>
          <response_lid ident = "TF01" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>0</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>1</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>More than one</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <qticomment>Processing for the incorrect answer.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "TF01">B</varequal>
            </not>
          </conditionvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Incorrect"/>
        </respcondition>
        <respcondition continue = "No">
          <qticomment>Processing for the correct answer.</qticomment>
          <conditionvar>
            <varequal respident = "TF01" case = "No">B</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  A QTI-XML instance CAN contain, 
            at most, one Asssessment.</mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "1">
      <presentation>
        <flow>
          <material>
            <mattext>Which one of the following data structures is used to 
              provide hierarchically structured assessments ?</mattext>
          </material>
          <response_lid ident = "LID03" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>Assessment</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>Section</mattext>
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>Item</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>Question</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar varname = "CORRECT" vartype = "Boolean" defaultval = "False"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <qticomment>Correct response check</qticomment>
          <conditionvar>
            <varequal respident = "LID03" case = "No">B</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
        <respcondition>
          <conditionvar>
            <not>
              <varequal respident = "LID03" case = "No">B</varequal>
            </not>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>No.  Only Sections support hierachical Assessments.</mattext>
        </material>
      </itemfeedback>
    </item>    
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "1">
      <presentation>
        <flow>
          <material>
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
403
403
404
405
406
407
408
409
410
411
412
313
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
            <mattext>Which question types are supported by the QTILite 
              specification ?</mattext>
          </material>
          <response_lid ident = "LID03" rcardinality = "Multiple" rtiming = "No">
            <render_choice shuffle = "Yes" minnumber = "1" maxnumber = "4">
              <flow_label>
                <response_label ident = "A">
                  <material>
                    <mattext>Multiple choice</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "B">
                  <material>
                    <mattext>Multiple response</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "C">
                  <material>
                    <mattext>Fill-in-the-blank</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "D">
                  <material>
                    <mattext>Drag-and-drop</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <outcomes>
          <decvar varname = "CORRECT" vartype = "Boolean" defaultval = "False"/>
          <decvar defaultval = "0" vartype = "Integer" minvalue = "0" 
            maxvalue = "1" varname = "SCORE"/>
        </outcomes>
        <respcondition>
          <qticomment>Correct answer response processing.</qticomment>
          <conditionvar>
            <varequal respident = "LID03">A</varequal>
          </conditionvar>
          <setvar varname = "CORRECT" action = "Set">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
        <respcondition>
          <qticomment>Incorrect answer response processing.</qticomment>
          <conditionvar>
            <not>
              <varequal respident = "LID03">A</varequal>
            </not>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
443
444
445
446
447
448
448
459
460
461
462
463
464
465
466
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct.  Multple-choice only.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>No.  Only Multiple-choice questions are supported.</mattext>
        </material>
      </itemfeedback>
    </item>
  </section>
</questestinterop>

The actual percentage total score and percentage of items actually answered will be derived values from the SCORE.normalized (SumofScores) and COUNT_Attempted.normalized (NumberCorrectAttempted) variables. If it is assumed that the candidate attempts the first four questions and gets the first three of those correct, then the set of variables are:

SCORE = 3

SCORE.min = 0

SCORE.max = 5

SCORE.normalized = 0.6

PercentageTotalScore = 60%

COUNT = 3

COUNT.min = 0

COUNT.max = 5

COUNT.normalized = 0.6

COUNT_Attempted = 3

COUNT_Attempted.min = 0

COUNT_Attempted.max = 4

COUNT_Attempted.normalized = 0.75

PercentageCorrectAttempted = 75%

This example is given as 'ims_qtiasiv1p2/outcome/advoutexample01.xml'.

4.4.2 Example (True/false Quiz)

True/false quiz - this is a simple "end of chapter" quiz given by an LMS. The scores are reported back to the student for self-evaluation, so there is no particular need for high reliability. All of the questions are discrete true/false. For the sake of being definite we assume there are 10 Items and that we have a defined outcome, "isCorrect" for each Item. This takes on the value '1' if the response was correct and "-1' if the response was not correct. All items are given equal weight. We wish to know, (1) the number of correct items, (2) the total score and (3) the weighted total score.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
<questestinterop>
  <section title = "IMSQTI_Outcomes_Processing_S02" ident = "IMS_QTIV1p2_S_OUTADV_02">
    <outcomes_processing scoremodel = "NumberCorrect">
      <outcomes>
        <decvar defaultval = "0" varname = "COUNT" vartype = "Integer" 
          minvalue = "0" maxvalue = "10" cutvalue = "5"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">5</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "SumofScores">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
          minvalue = "-10" maxvalue = "10" cutvalue = "0"/>
      </outcomes>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
    <outcomes_processing scoremodel = "ParameterSumofScoresWeighted">
      <outcomes>
        <decvar defaultval = "0" varname = "SCORE" vartype = "Integer" 
          minvalue = "-20" maxvalue = "20" cutvalue = "0"/>
      </outcomes>
      <objects_condition>
        <objects_parameter pname="qmd_weighting">2</objects_parameter>
      </objects_condition>
      <map_output varname="SCORE">SCORE_PWSOS</map_output>
      <outcomes_feedback_test title = "Section Mastery Test">
        <test_variable>
          <variable_test testoperator = "GTE" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionMastery" feedbacktype = "Response"/>
      </outcomes_feedback_test>
      <outcomes_feedback_test title = "Section Fail Test">
        <test_variable>
          <variable_test testoperator = "LT" varname = "COUNT">0</variable_test>
        </test_variable>
        <displayfeedback linkrefid = "SectionFail" feedbacktype = "Response"/>
      </outcomes_feedback_test>
    </outcomes_processing>
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
11
112
113
114
115
116
117
118
119
120
    <sectionfeedback ident = "SectionMastery" view = "All">
      <material>
        <mattext>Well done.  You have passed this Topic.</mattext>
      </material>
    </sectionfeedback>
    <sectionfeedback ident = "SectionFail" view = "All">
      <material>
        <mattext>Sorry.  You have failed this topic.</mattext>
      </material>
    </sectionfeedback>
    <selection_ordering>
      <selection/>
      <order order_type = "Sequential"/>
    </selection_ordering>
    <item ident = "IMS_QTIV1p2_I_OUT_01" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Multiple Assessments cannot be contained in a single QTI 
              ASI XML instance.
            </mattext>
          </material>
          <response_lid ident = "TF01" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the incorrect answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 
            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF01">T</varequal>
          </conditionvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Incorrect"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF01" case = "No">F</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
121
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback linkrefid = "Correct" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  A QTI-XML instance CAN contain multiple 
            Assessments.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_02" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Time-based responses are supported by the QTI 
              specification.
            </mattext>
          </material>
          <response_lid ident = "TF02" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
        <decvar defaultval = "-1" vartype = "Integer" minvalue="-1" 
          maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition>
          <conditionvar>
            <varequal respident = "TF02">T</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
281
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF02" case = "No">F</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  Time-based reponses ARE supported by 
            the specification.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_03" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>Items defined according to QTI V1.0 are not compatible with 
              Items from QTI V1.2.
            </mattext>
          </material>
          <response_lid ident = "TF03" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "True" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 
            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition>
          <conditionvar>
            <varequal respident = "TF03">F</varequal>
          </conditionvar>
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF03" case = "No">T</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  V1.0 Items ARE compatible with the V1.2 
            specification.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_04" maxattempts = "2">
      <presentation>
        <flow>
          <material>
            <mattext>The QTI specification is incompatible with the IMS Content 
              Packaging specification.
            </mattext>
          </material>
          <response_lid ident = "TF04" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label>
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label>
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 
            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition>
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
          <conditionvar>
            <varequal respident = "TF04">F</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF04" case = "No">T</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect">
        <material>
          <mattext>Sorry, you are wrong.  The IMS QTI and Content Packaging 
            specifications are compatible.
          </mattext>
        </material>
      </itemfeedback>
    </item>
    <item ident = "IMS_QTIV1p2_I_OUT_05" maxattempts = "2">
      <presentation>
        <flow class = "Block">
          <material>
            <mattext texttype = "text/plain" charset = "ascii-us">The QTI 
              specification is used within the ADL SCORM.
            </mattext>
          </material>
          <response_lid ident = "TF05" rcardinality = "Single" rtiming = "No">
            <render_choice shuffle = "Yes">
              <flow_label class = "Block">
                <response_label ident = "T">
                  <material>
                    <mattext>True</mattext>
                  </material>
                </response_label>
              </flow_label>
              <flow_label class = "Block">
                <response_label ident = "F">
                  <material>
                    <mattext>False</mattext>
                  </material>
                </response_label>
              </flow_label>
            </render_choice>
          </response_lid>
        </flow>
      </presentation>
      <resprocessing>
        <qticomment>Processing for the correct answer.</qticomment>
        <outcomes>
          <decvar defaultval = "False" vartype = "Boolean" varname = "CORRECT"/>
          <decvar defaultval = "-1" vartype = "Integer" 
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
            minvalue="-1" maxvalue="1" varname = "SCORE"/>
        </outcomes>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF05" case = "No">F</varequal>
          </conditionvar>
          <setvar action = "Set" varname = "CORRECT">True</setvar>
          <setvar action = "Set" varname = "SCORE">1</setvar>
          <displayfeedback feedbacktype = "Response" linkrefid = "Correct"/>
        </respcondition>
        <respcondition continue = "No">
          <conditionvar>
            <varequal respident = "TF05" case = "No">T</varequal>
          </conditionvar>
          <displayfeedback linkrefid = "Incorrect" feedbacktype = "Response"/>
        </respcondition>
      </resprocessing>
      <itemfeedback ident = "Correct" view = "All">
        <material>
          <mattext>Correct answer.</mattext>
        </material>
      </itemfeedback>
      <itemfeedback ident = "Incorrect" view = "All">
        <material>
          <mattext>Sorry, you are wrong.  The QTI specification is NOT used 
            within ADL SCORM.
          </mattext>
        </material>
      </itemfeedback>
    </item>
  </section>
</questestinterop>

If it is assumed that the candidate attempts the first four questions and gets the first three of those correct, then the set of variables are:

SCORE = 1

SCORE.min = -5

SCORE.max = 5

SCORE.normalized = 0.6

COUNT = 3

COUNT.min = 0

COUNT.max = 5

COUNT.normalized = 0.6

SCORE_Weighted = 2

SCORE_Weighted.min = -10

SCORE_Weighted.max = 10

SCORE_Weighted.normalized = 0.6

This example is given as 'ims_qtiasiv1p2/outcome/advoutexample01.xml'.

4.5 Implementation Guidance

4.5.1 The In-built Scoring Algorithms

Whenever possible the in-built scoring algorithms should be used to ensure interoperability between systems. It should be noted that it is possible to use multiple scoring algorithms on the same set of evaluation objects - this enables multivariate scoring mechanisms. At the current time the in-built scoring algorithms are:

  • Number Correct - see Appendix B1;
  • Number Correct Attempted - see Appendix B2;
  • Weighted Number Correct - see Appendix B3;
  • Weighted Number Correct Attempted - see Appendix B4;
  • Parameter Weighted Number Correct - see Appendix B5;
  • Parameter Weighted Number Correct Attempted - see Appendix B6;
  • Sum of Scores - see Appendix B7;
  • Sum of Scores Attempted - see Appendix B8;
  • Weighted Sum of Scores - see Appendix B9;
  • Weighted Sum of Scores Attempted - see Appendix B10;
  • Parameter Weighted Sum of Scores - see Appendix B11;
  • Parameter Weighted Sum of Scores Attempted - see Appendix B12;
  • Best K of N - see Appendix B13;
  • Guessing Penalty - see Appendix B14;
  • Weighted Guessing Penalty - see Appendix B15;

4.5.2 Defining Proprietary Scoring Algorithms

It is understood that not all of the required scoring algorithms will be supported intrinsically. Proprietary algorithms can be supported by:

  • Assigning an appropriate name to the new scoring algorithm and identifying this algorithm in the 'scoremodel' attribute for the <outcome_processing> element. Scoring engines that do not recognize the named scoring algorithm should indicate an error stating that it will not be possible to produce the associated set of aggregated scores;
  • Deciding how the default or specially named will be handled by the new scoring algorithm;
  • Assigning particular parameter names and passing these to the scoring algorithm using the <objects_parameter> and <processing_parameter> elements.

In some cases it will be necessary to coerce a numeric score into a count of whether or not the object has been answered correctly. The conversion from a SCORE to a CORRECT variable will be achieved using the cutscore value for the variable. The algorithm now becomes:

  if (object.SCORE >= object.SCORE.cutscore)
  then object.CORRECT = True
  else object.CORRECT = False

4.5.3 Using Meta-data

The meta-data of the objects is an important aggregation object selection mechanism. This selection must operate on the IMS QTI-specific meta-data and the IMS Meta-data definitions. There are three meta-data entry techniques used within IMS QTI:

  • For the explicitly identified IMS QTI-specific meta-data fields (the elements named using the 'qmd_**...***' scheme) the name of the meta-data element is used as the string name in the 'mdname' attribute of the <outcomes_metadata> element. The actual meta-data field value is that content of the <qmd_***...***> element. This mechanism is deprecated and should not be used;
  • For the IMS QTI-specific meta-data fields named using the generic vocabulary mechanism (using the element <qtimetadatafield>) the string name will be that identified using the <qtimetadatafield><fieldlabel> structure. The actual meta-data field value is the content of the structure <qtimetadatafield><fieldentry>;
  • For the IMS Meta-data entries the string naming convention will adopt the '.' mechanism to denote the concatenation of nested tags e.g. the string "metametadata.contribute.role.source" is used to denote the IMS Meta-data structure of '<metametadata><contribute><role><source>'. The 'lom:' namespace should be used at the front of the string.

4.5.4 Consideration of Selection & Ordering

It is important to realize that the actual score assigned can only be determined once the evaluation has been presented. Assessments and Sections can consist of a complex of child objects that are selected and ordered according to some set of rules defined within the <selection_ordering> element. The current outcomes processing mechanism is used to define the static rules that are used to create the aggregated scores that result from the selected, ordered, presented and attempted objects. Scores can only be assigned to objects that have been presented but some of the algorithms will provide normalized information that takes into account whether or not a presented object is attempted by the participant.

4.6 Example XML Instances

The full set of example files, as referred to in Sections 4.3 and 4.4 are available as part of the IMS QTI Resource Kit. The set of example XML instances are listed in Table 4.2.

 

Table 4.2 The QTI outcomes processing XML example files.

 
Filename
Nature
Description
basicoutexample01.xml S(1)I(10) The 'NumberCorrect' and 'NumberCorrectAttempted' in-built scoring algorithm using the standard input and output variable names.
basicoutexample02.xml S(1)I(10) The 'WeightedNumberCorrect', 'WeightedNumberCorrectAttempted', 'ParameterWeightedNumberCorrect' and 'ParameterWeightedNumberCorrectAttempted' in-built scoring algorithms using the standard input and output variable names.
basicoutexample03.xml S(1)I(10) The 'SumofScores' and 'SumofScoresAttempted' in-built scoring algorithms using the standard input and output variable names.
basicoutexample04.xml S(1)I(10) The 'WeightedSumofScores', 'WeightedSumofScoresAttempted', 'ParameterWeightedSumofScores' and 'ParameterWeightedSumofScoresAttempted' in-built scoring algorithm using the standard input and output variable names.
basicoutexample05.xml S(1)I(10) The 'BestKfromN' in-built scoring algorithm (selecting the best 7 scores) using the standard input and output variable names.
basicoutexample06.xml S(1)I(10) The 'GuessingPenalty' in-built scoring algorithm using the standard input and output variable names.
basicoutexample07.xml S(1)I(10) The 'SumofScores' in-built scoring algorithm using the standard input variable names but using remapped output variables.
basicoutexample08.xml S(1)I(10) The 'SumofScores' in-built scoring algorithm using remapped input variable names but with standard output variables.
basicoutexample09.xml S(1)I(10) The 'SumofScores' in-built scoring algorithm using the standard input and output variable names but selecting the objects according to their meta-data properties.
advoutexample01.xml A(1)S(1)I(10) Multiple-choice quiz with aggregated outcomes based upon the number correct, the percentage of total score obtained by a candidate and the percentage of items actually answered that were correct.
advoutexample02.xml A(1)S(1)I(10) True/false quiz with aggregated outcomes based upon the number correct, the total score and the weighted total score.

These files can be found in the directory: 'ims_qtiasiv1p2/outcome/'.

4.7 Proprietary Extensions

The proprietary extensions facilities listed in Table 4.3 are supported as elements within the specifications:

 

Table 4.3 List of proprietary extension elements.

 
Extension Element Name
Host Element
Description
objectscond_extension objects_condition Inclusion of alternative test mechanisms on the eligibility of an object to contribute to the scoring algorithm.

Note: These elements are only used if the suppliers of the ASIs require proprietary features that are not supported by the available range of elements. It is recommended that extensions are used sparingly. Backwards compatibility with proprietary extensions will NOT be guaranteed in future versions of these specifications.

4.8 V1.x/V2.0 Issues & Compatibility

The 'Outcomes Processing' functionality is not backwards compatible with early versions of the specification. These functions that have been replaced in earlier specifications are listed in Table 4.4.

 

Table 4.4 List of elements replaced in earlier specifications.

 
Previous Element Name
New Element Name
Description
sectionprocessing selection The original outcomes processing container for Sections.
assessmentprocessing assessment The original outcomes processing container for Assessments.


V1.0, V1.01 and V1.1 QTI-XML instances that do not use the elements listed within Table 4.4 are backwards compatible because the <outcomes_processing> element within V1.2 is optional.

4.9 IMS Harmonization

The issues of harmonization of this part of the IMS QTI specification with other IMS specifications is addressed by:

  • IMS Meta-data - the <outcomes_metadata> element should be capable of operating on any form of the entries defined within the IMS Meta-data specification.

Appendix A - Glossary of Terms

A1 - General Terms

attempted Several of the scoring algorithms differentiate between objects that were attempted and those that were not. Exact details of what constitutes an attempt will vary according to the exact details of the presentation environment, however, it is expected that the participants will know what actions on their parts constitute attempts. A typical example would be a submit button where pressing the button constitutes an attempt, where as moving to the next object using navigations controls to skip to the next item without pressing submit would not be considered an attempt.
object In the context of a scoring algorithm an object is an Item or Section that was selected for presentation to the participant and hence must (at least potentially) be operated on by the scoring algorithm. Item and section objects are nearly identical in that the both contain meta-data, which can be used for subset selection, and the both report outcome variables that are operated on by the items.
presented An Item is presented if it is displayed on whatever device is used for rendering items and the participant is given an opportunity to respond to it. A Section is presented if its selection rules have been used and at least one Item selected (directly or indirectly) by those rules has been presented.
reliability A term from psychometrics, the reliability of a score on an Assessment is defined as the correlation between the scores by the same participant taking the same Assessment twice (in an ideal world where the participant forgets all of the items and learns nothing new between administrations). As a correlation, this is a number between -1 and 1 (with negative results not seen in practice). Higher values are better. Practically, there are several methods for estimating reliability of a score and the method used should be described in the interpretvar for the score.
selected Selected objects appear in two contexts. First, the selection and ordering algorithm selects a set of sections and items for presentation as part of a section. These may or may not be presented, depending on what happens in the course of the Assessment (in particular, items may be selected but not presented due to time-outs). The selected objects and not just the presented ones are the ones operated on by the selection algorithm. Second, the outcomes_condition element can select a subset of the objects selected by selection and ordering for use with a particular scoring algorithm. This is particularly useful for defining sub-scores.
standard-error The "chance error" in the estimation of the score, i.e., the measurement error of the Assessment instrument. In an ideal experiment where a participant could take the same assessment multiple times, forgetting the items and not changing in between, this would be the standard deviations of the scores so obtained. Practically, there are several methods for estimating the reliability of a score and the interpretvar should provide information about which one is used.
sub-score A score defined on only a portion of the objects selected by the selection and ordering conditions.

A2 - Elements & Attributes

 

and_objects

The and_objects element is used to create complex logical ('AND') score processing rules. In this rule all of the clauses must be true before the object contributes to the score. This construct is commonly used to identify the set of meta-data values that must be true before the object can contribute to the outcomes aggregation. Appendix B contains a detailed description of the logic rules.

and_test

The and_test element is used to create a complex logical ('AND') test on score variables to determine what feedback will be presented. All of the contained clauses must be 'True' for this state to be 'True'. This mechanism is used to provide the participant with feedback on the Section or Assessment.

cutvalue

The cutvalue attribute is used to define the value for a score above which the participant is declared to have achieved mastery. This is an attribute of the decvar element.

decvar

The decvar element is used to declare the variables that are to be required to support the ASI scoring models. The integer variable 'SCORE' is the default declaration with a value of zero. The name, type and default value of the variables is determined by the varname, vartype and defaultval attributes. Other relevant attributes are minvalue, maxvalue and members.

defaultval

The defaultval attribute is used by the decvar element to define the start value for the variable. This value is bounded by the minvalue and maxvalue attributes, when used. The type of value is determined by the type of variable declared i.e. it should conform to the vartype attribute as used by the decvar element.

displayfeedback

The displayfeedback element is the trigger for the presentation of feedback to the users. The type of feedback to be displayed is determined by the feedbacktype attribute. The linkrefid attribute is used to identify the element containing the feedback.

feedbacktype

The feedbacktype attribute is used by the displayfeedback element to denote the type of feedback that is to be invoked. The default value for the feedbacktype attribute is 'Response'. This attribute is required because the element containing the feedback may have more than one type of feedback within its body. At the current time IMS QTI hints and solutions are not supported with the Assessment and Section structures.

interpretvar

The interpretvar element is used to describe statistical features about the associated variable. The variable that is associated is identified by the varname attribute. The primary usage for this element is to enable descriptions about the significance of the variables to be associated with their declaration - statistical parameters will be of particular importance.

linkrefid

The linkrefid attribute is used to associate the feedback trigger, displayfeedback, with the outcomes feedback in the Section and/or Assessment. This means that only a single copy of the data structure is required with multiple references to it.

map_input

The map_input element is used to change the algorithm specific default name of the input variable (e.g. 'SCORE', 'CORRECT') that is used as the source variable for the evaluation of objects (items and section) in a scoring algorithm. This enables the scoring algorithm to use different item and section level outcome variables in the scoring algorithm.

map_output

The map_output element is used to change the default name of the result variable reported out by an outcome_processing algorithm (e.g., 'SCORE', 'COUNT'). This enables the scoring algorithm to provide more descriptive output names and to use names to distinguish various subscores.

maxvalue

The maxvalue attribute is used with the decvar element to define the maximum value that can be assigned to the declared variable. This also limits the maximum value that can be assigned as part of the processing. This value does not take into account any weighting that may be applied to the object. This must be declared if the '*.maximum' and '*.normalized' variables are to be correctly defined.

mdname

The mdname operator is used with the outcomes_metadata element to identify the meta-data field that is to be used. The meta-data field may be contained within either the IMS QTI-specific fields or the externally linked IMS Meta-data field of the object. It is the responsibility of the host system to determine the existence or otherwise of the meta-data field.

mdoperator

The mdoperator is used with the outcomes_metadata element to identify the comparison operation to be applied on the meta-data field. The set of possible operators is defined as and enumerated set and consists of: EQ, NEQ, LT, LTE, GT, GTE. This set is derived from the operators normally supported by database access languages. Operations on strings should be limited to the EQ and NEQ otherwise the outcome is undefined.

members

The members attribute is used by the decvar element to define the members of the set being defined. The members of the set are defined as a comma separated list without any enclosing parentheses.

minvalue

The minvalue attribute is used with the decvar element to define the minimum value that can be assigned to the declared variable. This also limits the minimum value that can be assigned as part of the processing. This value does not take into account any weighting that may be applied to the object. This must be declared if the '*.minimum' and '*.normalized' variables are to be correctly defined.

not_objects

The not_objects element is used to create complex logical ('NOT') score processing rules. In this rule the contained clause must be false before the object contributes to the score. This construct is commonly used to identify the set of meta-data values that must be true before the object can contribute to the outcomes aggregation. Appendix B contains a detailed description of the logic rules.

not_test

The not_test element is used to create a complex logical ('NOT') test on score variables to determine the type of feedback that will be presented. The contained clauses must be 'False' for this state to be 'True'. This mechanism is used to provide the participant with feedback on the Section or Assessment.

objects_condition

This element is the container for all of the conditions that are to be applied to each of the aggregation objects (Sections and/or Items). The scoring variables for all of the objects that obey these conditions must be used by the scoring engine to produce the corresponding aggregated score.

objects_parameter

The objects_parameter element is used within the objects_condition element to pass key parameters to the system that is used to test the eligibility of the objects that are to contribute to the aggregated scoring. The name of the associated parameter is defined using the pname attribute.

objectscond_extension

This element is the extension facility within the objects_condition element. It should be used when the current conditions that can be applied to determine which objects are to contribute to the aggregated outcomes processing have insufficient functionality.

or_objects

The or_objects element is used to create complex logical ('OR') score processing rules. In this rule at least one of the clauses must be true before the object contributes to the score. This construct is commonly used to identify the set of meta-data values that must be true before the object can contribute to the outcomes aggregation. Appendix B contains a detailed description of the logic rules.

or_test

The or_test element is used to create a complex logical ('OR') test on score variables to determine the type of feedback that will be presented. At least one of the contained clauses must be 'True' for this state to be 'True'. This mechanism is used to provide the participant with feedback on the Section or Assessment.

outcomes

The outcomes element is used to contain the declaration of the variables for the outcomes processing. The outcomes element contains the decvar and interpretvar sub-elements to define the scoring variables.

outcomes_feedback_test

The outcomes_feedback_test element is used to contain the tests that are applied to the aggregated scores to determine the corresponding participant feedback. The actual feedback is contained in the sectionfeedback or assessfeedback elements as appropriate. The title attribute is used to contain the name of the feedback test set.

outcomes_metadata

The outcomes_metadata element is used to contain the rules that are applied to the meta-data fields of the object. This element uses the mdname and mdoperator attributes to define the meta-data label and the type of comparison respectively. Complex combinations of the meta-data values can be constructed using the and_object, or_object and not_object elements.

outcomes_processing

The outcomes_processing element is used to contain all of the aggregated outcomes processing instructions that are used by the parent Section or Assessment. The type of scoring algorithm that is assigned to the object is defined using the scoremodel attribute.

pname

The pname attribute is used by the objects_parameter and processing_parameter elements to identify the name of the parameter that is to be passed to the appropriate component of the scoring algorithm. There is no naming convention for these parameters and each parameter will, in general, be used differently by each scoring algorithm.

processing_parameter

The processing_parameter element is used within the outcomes_processing element to pass key parameters to the scoring algorithm itself e.g. this is used for the 'BestKofN' algorithm to define the value of 'K'. The name of the associated parameter is defined using the pname attribute.

qticomment

This is the commenting facility within the XML schemas. The comments can take any form supported as #PCDATA. The key difference between this comment style and the standard '<!-- *** -->' is that the former is passed through the XML parser to the host system.

scoremodel

The scoremodel attribute is used by the outcomes_processing element to identify that type of scoring model being used. The current implied default value is 'SumofScores'. This attribute allows vendors to clearly identify when they are using proprietary scoring models.

test_variable

The test_variable element is used within the outcomes_feedback_test element to contain the actual tests that are applied to the scoring variables. Complex test rules are constructed using the and_test, or_test and not_test elements.

testoperator

The testoperator attribute is used by the variable_test element to define the type of test being applied to the scoring variable. The possible tests are enumerated as: EQ, NEQ, LT, LTE, GT, GTE i.e. equal to, not equal to, less than, than or equal to, greater than, greater than or equal to.

title

The title attribute is an optional naming of the associated element. The title is used to help readability of the XML files. The title is usually 1-256 characters in length.

variable_test

The variable_test element is the actual test rule that is to be applied to an identified scoring variable. The variable is identified using the varname attribute and the type of test is defined using the testoperator attribute. Complex tests are constructed using this element within the and_test, or_test and not_test elements.

varname

The varname attribute is used to define the name of the variable. This attribute is used by the decvar and interpretvar elements. The name can consist of any lowercase and uppercase characters i.e. a-z, A-Z, 0-9 and underscore (0-9 should not be used for the first character). The default variable name is 'SCORE'.

vartype

The vartype attribute is used by the decvar element to define the type of variable being declared. The type of variable is: String, Boolean, Integer, Decimal, Scientific and Enumerated. The 'Enumerated' option enables the declaration of typed entries and the range of types is defined within the members attribute that must accompany the declaration.

view

The view attribute is used to define the 'actors' permitted to see the associated information e.g. feedback, objectives, etc. The supported actors are All (used to indicate access to all), Administrating Authority, Administrator, Assessor, Author, Candidate, Invigilator/Proctor, Psychometrician, Scorer and Tutor.

xml:lang

The xml:lang attribute is used wherever the language of the entry text can be varied. This attribute is used to define the language of the associated text. The format of the attribute shows that it is one of the core attributes provided by XML itself.

Appendix B - In-Built Scoring Algorithms

B1 - 'Number Correct' Scoring Algorithm

Name: NumberCorrect
Description: This is a count of the number of objects that have been presented and scored as correct. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables: Name Type Usage

 
COUNT Integer The total number of attempted objects with CORRECT=True.

 
COUNT.min Integer Set to 0 (zero).

 
COUNT.max Integer Set to the number of objects that were attempted.

 
COUNT.normalized Real The normalized value (range 0-1) for the number of correct answers.
Constraints: The variable 'CORRECT' must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max = parentobject.COUNT.max + 1
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT + 1
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.COUNT' and 'parentobject.COUNT.max'.
Derive Parameters: Percentage Correct = parentobject.COUNT.normalized * 100
(assuming parentobject.COUNT.max ?0)
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.
B2 - 'Number Correct (Attempted)' Scoring Algorithm

Name: NumberCorrectAttempted
Description: This is a count of the number of objects that have been attempted (this may be a subset of those selected by the system) and scored as correct. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables:

Name

Type

Usage


 
COUNT Integer The total number of attempted objects with CORRECT=True.

 
COUNT.min Integer Set to 0 (zero).

 
COUNT.max Integer Set to the number of objects that were attempted.

 
COUNT.normalized Real The normalized value (range 0-1) for the number of correct answers.
Constraints: The variable 'CORRECT' must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max = parentobject.COUNT.max + 1
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT + 1
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.COUNT' and 'parentobject.COUNT.max'.
Derive Parameters: Percentage Correct = parentobject.COUNT.normalized * 100
(assuming parentobject.COUNT.max ?0)
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.

B3 - 'Weighted Number Correct' Scoring Algorithm

Name: WeightedNumberCorrect
Description: This is a weighted count of the number of objects that have been presented and scored as correct. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables:

Name

Type

Usage


 
COUNT Integer The total weighted number of attempted objects with CORRECT=True.

 
COUNT.min Integer Set to 0 (zero).

 
COUNT.max Integer Set to the sum of the weights of the number of objects attempted.

 
COUNT.normalized Real The normalized value (range 0-1) for the number of correct answers.
Constraints: The variable 'CORRECT' must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
The child objects must have meta-data entries that contain the meta-data field 'qmd_weighting'. In the case where no weighting has been defined then it is assumed to be one (1).
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.COUNT' and 'parentobject.COUNT.max'.
Derive Parameters: Percentage Correct = parentobject.COUNT.normalized * 100
(assuming parentobject.COUNT.max ?0)
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.

B4 - 'Weighted Number Correct (Attempted)' Scoring Algorithm

Name: WeightedNumberCorrectAttempted
Description: This is a weighted count of the number of objects that have been presented and attempted, and scored as correct. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables:

Name

Type

Usage


 
COUNT Integer The total weighted number of attempted objects with CORRECT=True.

 
COUNT.min Integer Set to 0 (zero).

 
COUNT.max Integer Set to the sum of the weights of the number of objects attempted.

 
COUNT.normalized Real The normalized value (range 0-1) for the number of correct answers.
Constraints: The variable 'CORRECT' must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
The child objects must have meta-data entries that contain the meta-data field 'qmd_weighting'. In the case where no weighting has been defined then it is assumed to be one (1).
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.COUNT' and 'parentobject.COUNT.max'.
Derive Parameters: Percentage Correct = parentobject.COUNT.normalized * 100
(assuming parentobject.COUNT.max ?0)
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.

B5 - 'Parameter Weighted Number Correct' Scoring Algorithm

Name: ParameterWeightedNumberCorrect
Description: This is a weighted count of the number of objects that have been presented and scored as correct. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables:

Name

Type

Usage


 
COUNT Integer The total weighted number of attempted objects with CORRECT=True.

 
COUNT.min Integer Set to 0 (zero).

 
COUNT.max Integer Set to the sum of the weights of the number of objects attempted.

 
COUNT.normalized Real The normalized value (range 0-1) for the number of correct answers.
Constraints: The variable 'CORRECT' must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
The weighting that is to be applied is defined using the values assigned in the outcomes processing element statement <objects_parameter pname="qmd_weighting">. In the case where no weighting has been defined then it is assumed to be one (1).
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.COUNT' and 'parentobject.COUNT.max'.
Derive Parameters: Percentage Correct = parentobject.COUNT.normalized * 100
(assuming parentobject.COUNT.max ?0)
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.

B6 - 'Parameter Weighted Number Correct (Attempted)' Scoring Algorithm

Name: ParameterWeightedNumberCorrectAttempted
Description: This is a weighted count of the number of objects that have been presented, attempted and scored as correct. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables:

Name

Type

Usage


 
COUNT Integer The total weighted number of attempted objects with CORRECT=True.

 
COUNT.min Integer Set to 0 (zero).

 
COUNT.max Integer Set to the sum of the weights of the number of objects attempted.

 
COUNT.normalized Real The normalized value (range 0-1) for the number of correct answers.
Constraints: The variable 'CORRECT' must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
The weighting that is to be applied is defined using the values assigned in the outcomes processing element statement <objects_parameter pname="qmd_weighting">. In the case where no weighting has been defined then it is assumed to be one (1).
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.min = 0
parentobject.COUNT.max = 0
parentobject.COUNT.normalized=0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(CORRECT)) = True 
    then
      begin
        parentobject.COUNT.max               = parentobject.COUNT.max 
                      + childobject.weighting
        if childobject.CORRECT = True
        then parentobject.COUNT = parentobject.COUNT 
                      + childobject.weighting
      endif
  enddo
parentobject.COUNT.normalized = 
      parentobject.COUNT/parentobject.COUNT.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.COUNT' and 'parentobject.COUNT.max'.
Derive Parameters: Percentage Correct = parentobject.COUNT.normalized * 100
(assuming parentobject.COUNT.max ?0)
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.

B7 - 'Sum of Scores' Scoring Algorithm

Name: SumofScores
Description: This is the sum of the scores assigned to the individual child objects selected for presentation to the participant. An object is included within the scoring algorithm if it has a variable that has been declared as 'SCORE' and with type integer.
Default Variables:

Name

Type

Usage


 
SCORE Integer The sum of the individual 'SCORE; values assigned within each of the child objects.

 
SCORE.min Integer The sum of the individual 'minvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.max Integer The sum of the individual 'maxvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.normalized Real The normalized value (range 0-1) for the total score.
Constraints: The 'SCORE' integer variable must be used otherwise the <map_input> instruction must identify the new variable name.
The 'minvalue' attribute for the variable must have been defined. If undefined then the SCORE.min value is undefined.
The 'maxvalue' attribute for the variable must have been defined. If undefined then the SCORE.max value is undefined.
Algorithm:
parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + childobject.SCORE.minvalue
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + childobject.SCORE.maxvalue
        parentobject.SCORE             = parentobject.SCORE 
                    + childobject.SCORE
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.SCORE' and 'parentobject.SCORE.max'.
Derive Parameters: Percentage Score = parentobject.SCORE.normalized * 100
Notes: If this algorithm is used concurrently with another that operates on the 'SCORE' variables then the <map_output> element should be used to ensure that 'SCORE' variable is not overloaded and therefore provides unreliable results.

B8 - 'Sum of Scores (Attempted)' Scoring Algorithm

Name: SumofScoresAttempted
Description: This is the sum of the scores assigned to the individual attempted (this may be a subset of those selected) child objects. An object is included within the scoring algorithm if it has a variable that has been declared as 'SCORE' and with type integer.
Default Variables:

Name

Type

Usage


 
SCORE Integer The sum of the individual 'SCORE; values assigned within each of the child objects.

 
SCORE.min Integer The sum of the individual 'minvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.max Integer The sum of the individual 'maxvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.normalized Real The normalized value (range 0-1) for the total score.
Constraints: The 'SCORE' integer variable must be used otherwise the <map_input> instruction must identify the new variable name.
The 'minvalue' attribute for the variable must have been defined. If undefined then the SCORE.min value is undefined.
The 'maxvalue' attribute for the variable must have been defined. If undefined then the SCORE.max value is undefined.
Algorithm:
parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + childobject.SCORE.minvalue
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + childobject.SCORE.maxvalue
        parentobject.SCORE             = parentobject.SCORE 
                    + childobject.SCORE
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.SCORE' and 'parentobject.SCORE.max'.
Derive Parameters: Percentage Score = parentobject.SCORE.normalized * 100
Notes: If this algorithm is used concurrently with another that operates on the 'SCORE' variables then the <map_output> element should be used to ensure that 'SCORE' variable is not overloaded and therefore provides unreliable results.

B9 - 'Weighted Sum of Scores' Scoring Algorithm

Name:

WeightedSumofScores

Description:

This is the weighted sum of the scores assigned to the individual child objects that have been presented. An object is included within the scoring algorithm if it has a variable that has been declared as 'SCORE' and with type integer.

Default Variables:

Name

Type

Usage


 
SCORE Integer The weighted sum of the individual 'SCORE; values assigned within each of the child objects.

 
SCORE.min Integer The weighted sum of the individual 'minvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.max Integer The weighted sum of the individual 'maxvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.normalized Real The normalized value (range 0-1) for the total score.
Constraints: The 'SCORE' integer variable must be used otherwise the <map_input> instruction must identify the new variable name.
The 'minvalue' attribute for the variable must have been defined. If undefined then the SCORE.min value is undefined.
The 'maxvalue' attribute for the variable must have been defined. If undefined then the SCORE.max value is undefined.
The child objects must have meta-data entries that contain the meta-data field 'qmd_weighting'. Where no weighting has been defined then it is set to be one (1).
Algorithm:
parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.SCORE' and 'parentobject.SCORE.max'.
Derive Parameters: Percentage Score = parentobject.SCORE.normalized * 100
Notes: If this algorithm is used concurrently with another that operates on the 'SCORE' variables then the <map_output> element should be used to ensure that 'SCORE' variable is not overloaded and therefore provides unreliable results.

B10 - 'Weighted Sum of Scores (Attempted)' Scoring Algorithm

Name: WeightedSumofScoresAttempted
Description: This is the weighted sum of the scores assigned to the individual child objects that have been presented and attempted. An object is included within the scoring algorithm if it has a variable that has been declared as 'SCORE' and with type integer.
Default Variables:

Name

Type

Usage


 
SCORE Integer The weighted sum of the individual 'SCORE; values assigned within each of the child objects.

 
SCORE.min Integer The weighted sum of the individual 'minvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.max Integer The weighted sum of the individual 'maxvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.normalized Real The normalized value (range 0-1) for the total score.
Constraints: The 'SCORE' integer variable must be used otherwise the <map_input> instruction must identify the new variable name.
The 'minvalue' attribute for the variable must have been defined. If undefined then the SCORE.min value is undefined.
The 'maxvalue' attribute for the variable must have been defined. If undefined then the SCORE.max value is undefined.
The child objects must have meta-data entries that contain the meta-data field 'qmd_weighting'. Where no weighting has been defined then it is set to be one (1).
Algorithm:
parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.SCORE' and 'parentobject.SCORE.max'.
Derive Parameters: Percentage Score = parentobject.SCORE.normalized * 100
Notes: If this algorithm is used concurrently with another that operates on the 'SCORE' variables then the <map_output> element should be used to ensure that 'SCORE' variable is not overloaded and therefore provides unreliable results.

B11 - 'Parameter Weighted Sum of Scores' Scoring Algorithm

Name: ParameterWeightedSumofScores
Description: This is the weighted sum of the scores assigned to the individual child objects that have been presented. An object is included within the scoring algorithm if it has a variable that has been declared as 'SCORE' and with type integer.
Default Variables:

Name

Type

Usage


 
SCORE Integer The weighted sum of the individual 'SCORE; values assigned within each of the child objects.

 
SCORE.min Integer The weighted sum of the individual 'minvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.max Integer The weighted sum of the individual 'maxvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.normalized Real The normalized value (range 0-1) for the total score.
Constraints: The 'SCORE' integer variable must be used otherwise the <map_input> instruction must identify the new variable name.
The 'minvalue' attribute for the variable must have been defined. If undefined then the SCORE.min value is undefined.
The 'maxvalue' attribute for the variable must have been defined. If undefined then the SCORE.max value is undefined.
The weighting that is to be applied is defined using the values assigned in the outcomes processing element statement <objects_parameter pname="qmd_weighting">. In the case where no weighting has been defined then it is assumed to be one (1).
Algorithm:
parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_SELECTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.SCORE' and 'parentobject.SCORE.max'.
Derive Parameters: Percentage Score = parentobject.SCORE.normalized * 100
Notes: If this algorithm is used concurrently with another that operates on the 'SCORE' variables then the <map_output> element should be used to ensure that 'SCORE' variable is not overloaded and therefore provides unreliable results.

B12 - 'Parameter Weighted Sum of Scores (Attempted)' Scoring Algorithm

Name: ParameterWeightedSumofScoresAttempted
Description: This is the weighted sum of the scores assigned to the individual child objects that have been presented and attempted. An object is included within the scoring algorithm if it has a variable that has been declared as 'SCORE' and with type integer.
Default Variables:

Name

Type

Usage


 
SCORE Integer The weighted sum of the individual 'SCORE; values assigned within each of the child objects.

 
SCORE.min Integer The weighted sum of the individual 'minvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.max Integer The weighted sum of the individual 'maxvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.normalized Real The normalized value (range 0-1) for the total score.
Constraints: The 'SCORE' integer variable must be used otherwise the <map_input> instruction must identify the new variable name.
The 'minvalue' attribute for the variable must have been defined. If undefined then the SCORE.min value is undefined.
The 'maxvalue' attribute for the variable must have been defined. If undefined then the SCORE.max value is undefined.
The weighting that is to be applied is defined using the values assigned in the outcomes processing element statement <objects_parameter pname="qmd_weighting">. In the case where no weighting has been defined then it is assumed to be one (1).
Algorithm:
parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized = 0.0
for ALL_ATTEMPTED(childobjects) do
  begin
    if child.object(HAS_VARIABLE(SCORE)) = True 
    then
      begin
        parentobject.SCORE.min               = parentobject.SCORE.min 
                    + (childobject.SCORE.minvalue
                      * childobject.weighting)
        parentobject.SCORE.max               = parentobject.SCORE.max 
                    + (childobject.SCORE.maxvalue
                      * childobject.weighting)
        parentobject.SCORE             = parentobject.SCORE 
                  + (childobject.SCORE 
                    * childobject.weighting)
      endif
  enddo
parentobject.SCORE.normalized  =
  parentobject.SCORE-parentobject.SCORE.min/
  parentobject.SCORE.max-parentobject.SCORE.max
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.SCORE' and 'parentobject.SCORE.max'.
Derive Parameters: Percentage Score = parentobject.SCORE.normalized * 100
Notes: If this algorithm is used concurrently with another that operates on the 'SCORE' variables then the <map_output> element should be used to ensure that 'SCORE' variable is not overloaded and therefore provides unreliable results.

B13 - 'Best K from N' Scoring Algorithm

Name: BestKfromN
Description: This is the sum of the best 'K' scores assigned to the individual child objects that have been attempted. An object is included within the scoring algorithm if it has a variable that has been declared as 'SCORE' and with type integer. This algorithm is normalized using all objects that have been selected and presented.
Default Variables:

Name

Type

Usage


 
SCORE Integer The sum of the individual best 'K' 'SCORE' values assigned across the set of attempted child objects.

 
SCORE.min Integer The sum of the individual worst 'K' 'minvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.max Integer The sum of the individual best 'K' 'maxvalue' entries assigned to the 'SCORE' variable for each child object.

 
SCORE.normalized Real The normalized value (range 0-1) for the total score.
Constraints: The 'SCORE' integer variable must be used otherwise the <map_input> instruction must identify the new variable name.
The 'minvalue' attribute for the variable must have been defined. If undefined then the SCORE.min value is undefined. All of the objects must have the same value for 'minvalue'.
The 'maxvalue' attribute for the variable must have been defined. If undefined then the SCORE.max value is undefined. All of the objects must have the same value for 'maxvalue'.
The <processing_parameter> instruction must be used to assign an integer value to 'K'. In the absence of this instruction then 'K' is set to the number of objects attempted.
Algorithm:
parentobject.SCORE = 0
parentobject.SCORE.min = 0
parentobject.SCORE.max = 0
parentobject.SCORE.normalized=0.0
parentobject.SCORE.max               = value_of_K  * childobject.SCORE.maxvalue
parentobject.SCORE.min               = value_of_K  * childobject.SCORE.minvalue
SORT (ALLPRESENTED_OBJECTS and child.object(HAS_VARIABLE(SCORE) 
                and ORDERED_BY_ DECREASING_SCORE)
for TOP(value_of_K) do              
  begin
    parentobject.SCORE = 
      parentobject.SCORE  + childobject.SCORE
  enddo
parentobject.SCORE.normalized =
  (parentobject.SCORE- parentobject.SCORE.min) /
  (parentobject.SCORE.max- parentobject.SCORE.min)
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant. This affects the values of 'parentobject.SCORE' and 'parentobject.SCORE.max'.
Derive Parameters: Percentage Score = parentobject.SCORE.normalized * 100
Notes: If this algorithm is used concurrently with another that operates on the 'SCORE' variables then the <map_output> element should be used to ensure that 'SCORE' variable is not overloaded and therefore provides unreliable results.

B14 - 'Guessing Penalty' Scoring Algorithm

Name: GuessingPenalty
Description: This is the count of objects that have been answered correctly, incorrectly and which have not been attempted. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables:

Name

Type

Usage


 
COUNT Integer The score calculated taking into account the relevant penalty.

 
COUNT.correct Integer The number of objects answered correctly i.e. with CORRECT=True.

 
COUNT.incorrect Integer The number of objects answered incorrectly i.e. with CORRECT=False.

 
COUNT.unattempted Integer The number of objects not attempted
Constraints: The variable 'CORRECT must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
The child objects must have meta-data entries that contain the meta-data field 'qmd_penaltyvalue'. Where no penalty value has been defined then it is set to one (1).
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.correct = 0
parentobject.COUNT.incorrect = 0
parentobject.COUNT.unattempted = 0
for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
  begin
    if child.object(ATTEMPTED) = True 
    then
      begin
        if childobject.CORRECT = True
        then 
          begin
            parentobject.COUNT.correct = 
              parentobject.COUNT .correct + 1
            parentobject.COUNT = parentobject.COUNT + 1
          end
        else 
          begin
            parentobject.COUNT.incorrect = 
              parentobject.COUNT .incorrect + 1
            parentobject.COUNT = parentobject.COUNT - 
                childobject.penaltyvalue
          end
      endif
    else parentobject.COUNT.unattempted = 
      parentobject.COUNT.unattempted + 1
  enddo
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant.
Derive Parameters: None.
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.

B15 - 'Weighted Guessing Penalty' Scoring Algorithm

Name: WeightedGuessingPenalty
Description: This is the weighted count score for objects that have been answered correctly, incorrectly and which have not been attempted. An object is included within this scoring algorithm if it has a variable that has been declared as 'CORRECT' and with type Boolean. The object is defined as correct when CORRECT=True.
Default Variables:

Name

Type

Usage


 
COUNT Integer The score calculated taking into account the relevant penalty.

 
COUNT.correct Integer The number of objects answered correctly i.e. with CORRECT=True.

 
COUNT.incorrect Integer The number of objects answered incorrectly i.e. with CORRECT=False.

 
COUNT.unattempted Integer The number of objects not attempted
Constraints: The variable 'CORRECT must have been declared as a Boolean type otherwise the <map_input> instruction must identify the new variable name.
The child objects must have meta-data entries that contain the meta-data field 'qmd_penaltyvalue'. Where no penalty value has been defined then it is set to one (1).
The child objects must have meta-data entries that contain the meta-data field 'qmd_weighting'. Where no penalty value has been defined then it is set to one (1).
Algorithm:
parentobject.COUNT = 0
parentobject.COUNT.correct = 0
parentobject.COUNT.incorrect = 0
parentobject.COUNT.unattempted = 0
for ALL_PRESENTED(childobjects)and (HAS_VARIABLE(CORRECT)  do
  begin
    if child.object(ATTEMPTED) = True 
    then
      begin
        if childobject.CORRECT = True
        then 
          begin
            parentobject.COUNT.correct = 
              parentobject.COUNT .correct + 1
            parentobject.COUNT = parentobject.COUNT + 
              parentobject.weighting
          end
        else 
          begin
            parentobject.COUNT.incorrect = 
              parentobject.COUNT .incorrect + 1
            parentobject.COUNT = 
              parentobject.COUNT *childobject.weighting- 
              childobject.penaltyvalue
          end
      endif
    else parentobject.COUNT.unattempted = 
      parentobject.COUNT.unattempted + 1
  enddo
Sequencing: Selection and ordering will determine which objects are presented and thus may be attempted by the participant.
Derive Parameters: None.
Notes: If this algorithm is used concurrently with another that operates on the 'COUNT' variables then the <map_output> element should be used to ensure that 'COUNT' variable is not overloaded and therefore provides unreliable results.

Appendix C - Logic Rules

The elements or_objects, and_objects and not_objects are used within the objects_condition element to select the Items and/or Sections whose scoring variables are to be aggregated. An aggregation rule takes the form such as:

<outcomes_metadata mdname="qmd_topic" mdoperator="EQ">trigonometry</outcomes_metadata>

i.e. select all of the objects that have a meta-data entry of "qmd_topic" that is equal to 'trigonometry'.

The objects_condition element will contain a set of rules, i.e. R1, R2, R3...Rk. The logic operators between these rules are defined as:

  • ¬R1 means 'NOT R1' which is represented in QTIASI-XML as <not_objects>R1</not_objects>
  • R1 and R2 means 'R1 AND R2' which is represented in QTIASI-XML as <and_objects>R1, R2</and_objects>
  • R1 or R2 means 'R1 OR R2' which is represented in QTIASI-XML as <or_objects>R1, R2</or_objects>

Using this initial set of constructions gives rise to the following representations:

  • ¬ (R1 OR R2) which is represented in QTIASI-XML as
  <not_objects>
    <or_objects>
        R1
        R2
    </or_objects>
    </not_objects>

  • ¬ (R1 AND R2) which is represented in QTIASI-XML as
  <not_objects>
    <and_objects>
        R1
        R2
    </and_objects>
  </not_objects>

  • (¬R1 AND ¬R2) which is represented in QTIASI-XML as
  <and_objects>
    <not_objects>R1</not_objects>    
    <not_objects>R2</not_objects>
  </and_objects>

  • (¬R1 OR ¬R2) which is represented in QTIASI-XML as
  <or_objects>
    <not_objects>R1</not_objects>    
    <not_objects>R2</not_objects>
  </or_objects>

  • ((R1 OR R2) AND (R3 OR R4)) which is represented in QTIASI-XML as
  <and_objects>
    <or_objects>
        R1
        R2
    </or_objects>
    <or_objects>
        R3
        R4
    </or_objects>
  </and_objects>
  • ((R1 OR R2 OR R3) AND (R4 OR R5 OR (R6 AND R7 AND R8)) AND ¬(R9 OR R10 OR R11)) which is represented in QTIASI-XML as
  <and_objects>
    <or_objects>
        R1
        R2
        R3
    </or_objects>
    <or_objects>
        R4
        R5 
      <and_objects>
          R6
          R7
          R8
      </and_objects>
    </or_objects>
    <or_objects>
      <not_objects>
        <or_objects>
            R9
            R10
            R11
        </or_objects>
      </not_objects>
    </or_objects>
  </and_objects>

This set of rules gives rise to the following guidelines:

  • Only a single <and>, <or> or <not> structure is required at the top-most level of the logic statement;
  • Multiple occurrences of the <and>, <or> and <not> elements is permitted within <and> and <or> elements to allow the construction of complex logic statements;
  • The <not> element can only contain a single element the result of which is to invert the logic of the test.

About This Document

 
Title IMS Question & Test Interoperability ASI Outcomes Processing
Editors Colin Smythe, Lane Brewer, and Steve Lay
Version 1.2
Version Date 11 February 2002
Status Final Specification
Summary

This document presents the IMS QTI ASI Outcomes Processing Specification. This specification is one of the set of the IMS Question & Test Interoperability specifications. This Outcomes Processing specification is an extension to the IMS QTI ASI V1.2 specification.

Revision Information 22 January 2002
Purpose Defines the 'Assessment and Section Outcome Processing' features that have been added to IMS QTI V1.1 as part of the development of the IMS QTI V1.2 specification.
Document Location http://www.imsglobal.org/question/v1p2/imsqti_asi_outv1p2.html

List of Contributors

The following individuals contributed to the development of this document:

Russell Almond ETS, USA
Lane Brewer Galton Technologies Inc.
Todd Brewer Galton Technologies Inc.
Russell Grocott Can Studios Ltd.
Andy Heath CETIS/JISC, UK
Paul Hilton Can Studios Ltd.
Richard Johnson

Goal Design Inc.

John Kleeman Question Mark Computing Ltd.
Steven Lay University of Cambridge Local Examinations Syndicate, UK
Jez Lord Can Studios Ltd.
Paul Roberts Question Mark Computing Ltd.
Nial Sclater CETIS/University of Strathclyde, UK
Eric Shepherd

Question Mark Corporation

Colin Smythe Dunelm Services Ltd.

Revision History

 
Version No.
Release Date
Comments
Final Version 1.2 11 February 2002 The first release of the QTI ASI Outcomes Processing Final specification.

Index

A
Administrating Authority 1
Administrator 1, 2, 3
ASI 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
Assessment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
Assessment Elements
     assessfeedback 1, 2
     assessment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
Assessor 1, 2, 3
Attributes
     action 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     case 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
     charset 1, 2
     class 1
     columns 1
     continue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     cutvalue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     defaultval 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47
     feedbacktype 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     ident 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47
     label 1
     linkrefid 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     maxattempts 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38
     maxnumber 1, 2, 3
     maxvalue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50
     mdname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     mdoperator 1, 2, 3, 4, 5, 6, 7, 8, 9
     members 1, 2, 3, 4, 5
     minnumber 1, 2, 3
     minvalue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50
     order_type 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     pname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     rcardinality 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     respident 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     rtiming 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     scoremodel 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     shuffle 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     step 1
     testoperator 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     texttype 1, 2
     title 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
     varname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49
     vartype 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36
     view 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     xmllang 1, 2, 3, 4, 5, 6
     xmlspace 1
Author 1, 2, 3
 

B
Basic 1, 2

C
Candidate 1, 2, 3
Common Elements
     conditionvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     decvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53
     displayfeedback 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     fieldentry 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     interpretvar 1, 2, 3, 4, 5, 6, 7
     material 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     mattext 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
     objectives 1
     order 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     other 1, 2, 3, 4, 5, 6
     outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81
     outcomes_processing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25
     qticomment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     qtimetadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     qtimetadatafield 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     reference 1
     selection 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28
     selection_ordering 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     setvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     varequal 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     vocabulary 1, 2, 3, 4, 5, 6, 7
 

E
Elements
     and_object 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     and_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     assessfeedback 1, 2
     assessment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     conditionvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     decvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53
     displayfeedback 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     fieldentry 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     fieldlabel 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     flow 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     flow_label 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     interpretvar 1, 2, 3, 4, 5, 6, 7
     itemfeedback 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     itemmetadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
     map_input 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
     map_output 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
     material 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     mattext 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
     not_object 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     not_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     objectives 1
     objects_condition 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19
     objects_parameter 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
     objectscond_extension 1, 2, 3, 4, 5, 6
     or_objects 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     or_test 1, 2, 3, 4, 5, 6, 7, 8, 9
     order 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     other 1, 2, 3, 4, 5, 6
     outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81
     outcomes_feedback_test 1, 2, 3, 4, 5, 6, 7, 8, 9
     outcomes_metadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
     outcomes_processing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25
     presentation 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     processing_parameter 1, 2, 3, 4, 5, 6, 7, 8
     qmd_levelofdifficulty 1, 2, 3, 4
     qmd_topic 1, 2, 3, 4, 5, 6
     qmd_weighting 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32
     qticomment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     qtimetadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     qtimetadatafield 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
     questestinterop 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     reference 1
     render_choice 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     respcondition 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     response_label 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     response_lid 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     resprocessing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42
     section 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
     sectionfeedback 1, 2, 3, 4
     selection 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28
     selection_ordering 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     setvar 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     test_variable 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
     varequal 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     variable_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     vocabulary 1, 2, 3, 4, 5, 6, 7
Extension Elements
     objectscond_extension 1, 2, 3, 4, 5, 6
 

H
Harmonisation
     IMS
 

Meta-data 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

I
Interoperability structures
     Assessment 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
     Item 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
     Section 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41
Invigilator 1, 2
Item 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
Item Elements
     flow 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     flow_label 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     itemfeedback 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
     itemmetadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
     presentation 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     render_choice 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     respcondition 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
     response_label 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     response_lid 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
     resprocessing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42
 

M
Meta-data
     Description 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
     Elements
 

qmd_levelofdifficulty 1, 2, 3, 4

qmd_topic 1, 2, 3, 4, 5, 6, 7, 8

qmd_weighting 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32      Language 1
     Resource Identifier 1
     status 1
     Version 1, 2, 3
Multiple choice 1
Multiple response 1
 

O
Outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21
Outcomes processing
     Attributes
 

testoperator 1, 2, 3, 4, 5, 6, 7, 8, 9, 10      Elements
 

and_object 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

and_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

map_input 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

map_output 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26

not_object 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

not_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

objects_condition 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19

objects_parameter 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

objectscond_extension 1, 2, 3, 4, 5, 6

or_objects 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

or_test 1, 2, 3, 4, 5, 6, 7, 8, 9

outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81

outcomes_feedback_test 1, 2, 3, 4, 5, 6, 7, 8, 9

outcomes_metadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14

outcomes_processing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25

processing_parameter 1, 2, 3, 4, 5, 6, 7, 8

test_variable 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

variable_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

P
Participant
     Administering Authority 1, 2
     Administrator 1, 2, 3
     Assessor 1, 2, 3
     Author 1, 2, 3
     Candidate 1, 2, 3
     Invigilator 1, 2
     Proctor 1, 2
     Psychometrician 1, 2, 3
     Scorer 1, 2, 3
     Tutor 1, 2, 3
Proctor 1, 2
Psychometrician 1, 2, 3
 

Q
QTILite 1, 2, 3
Question 1, 2, 3, 4, 5, 6

R
Resource Identifier 1
Response 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19

S
Scoring algorithms
     BestKfromN 1, 2, 3
     GuessingPenalty 1, 2, 3, 4, 5
     NumberCorrect 1, 2, 3, 4, 5, 6, 7
     NumberCorrectAttempted 1, 2, 3, 4, 5, 6, 7, 8
     ParameterWeightedNumberCorrect 1, 2, 3, 4, 5
     ParameterWeightedNumberCorrectAttempted 1, 2, 3, 4, 5
     ParameterWeightedSumofScores 1, 2, 3, 4
     ParameterWeightedSumofScoresAttempted 1, 2, 3, 4
     SumofScores 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
     SumofScoresAttempted 1, 2, 3, 4, 5
     WeightedGuessingPenalty 1, 2, 3
     WeightedNumberCorrect 1, 2, 3, 4, 5
     WeightedNumberCorrectAttempted 1, 2, 3, 4, 5
     WeightedSumofScores 1, 2, 3, 4
     WeightedSumofScoresAttempted 1, 2, 3, 4
Section 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41
Section Elements
     sectionfeedback 1, 2, 3, 4
Selection & ordering 1, 2, 3
     Attributes
 

mdname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

mdoperator 1, 2, 3, 4, 5, 6, 7, 8, 9

order_type 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11

pname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13      Elements
 

order 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

selection 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28

selection_ordering 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 Solution 1, 2, 3

T
True/false 1, 2, 3
Tutor 1, 2, 3

U
Use Cases 1

V
Version 1.1 Additions
     Attributes
 

class 1

xmllang 1, 2, 3, 4, 5, 6

xmlspace 1      Elements
 

fieldentry 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

fieldlabel 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

flow 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

flow_label 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

qtimetadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

qtimetadatafield 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

vocabulary 1, 2, 3, 4, 5, 6, 7 Version 1.2 Additions
     Attributes
 

cutvalue 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

mdname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

mdoperator 1, 2, 3, 4, 5, 6, 7, 8, 9

order_type 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11

pname 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

testoperator 1, 2, 3, 4, 5, 6, 7, 8, 9, 10      Elements
 

and_object 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

and_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

map_input 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

map_output 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26

not_object 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

not_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

objects_condition 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19

objects_parameter 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

objectscond_extension 1, 2, 3, 4, 5, 6

or_objects 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

or_test 1, 2, 3, 4, 5, 6, 7, 8, 9

order 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

outcomes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81

outcomes_feedback_test 1, 2, 3, 4, 5, 6, 7, 8, 9

outcomes_metadata 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14

outcomes_processing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25

processing_parameter 1, 2, 3, 4, 5, 6, 7, 8

reference 1

selection 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28

selection_ordering 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

test_variable 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

variable_test 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

X
XML 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
XML Schema
     DTD 1
 

 

 

 

IMS Global Learning Consortium, Inc. ("IMS") is publishing the information contained in this IMS Question & Test Interoperability: ASI Outcomes Processing ("Specification") for purposes of scientific, experimental, and scholarly collaboration only.

IMS makes no warranty or representation regarding the accuracy or completeness of the Specification.
This material is provided on an "As Is" and "As Available" basis.

The Specification is at all times subject to change and revision without notice.

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

IMS would appreciate receiving your comments and suggestions.

Please contact IMS through our website at http://www.imsglobal.org

Please refer to Document Name:
IMS Question & Test Interoperability: ASI Outcomes Processing Date: 11 February 2002