1.1 Architectural Overview

1.2 Goals

This specification has been developed with the goal of standardizing the interface between CAT engines and assessment delivery platforms, so that:

1. An assessment which contains an adaptive section relying on the service provided by a particular CAT engine can be delivered on, and easily moved to, any assessment platform which supports the CAT API.
2. An assessment which contains an adaptive section can be moved to a different CAT engine with no more than minimal and well-defined changes to the adaptive section configuration.
3. Consequently, an assessment which contains an adaptive section can be delivered by multiple organizations (for example, state testing authorities administering a multi-state test), with each organization able to select the pairing of assessment platform and CAT engine which best suits that organization.

Although portability and interoperability are the main goals of this specification, it also aims to foster competition and differentiation between CAT engine providers, not to require CAT engine providers to disclose trade secrets, and not to be opinionated regarding the many different CAT approaches, feature sets, and technical architectures, both existing and foreseeable.

For this reason, this specification generally treats CAT engines as "black boxes". While this black box approach provides considerable flexibility to CAT engine implementers, for the most part without sacrifice of portability and interoperability, accommodating the variety of CAT engine input data requirements means that in some cases assessment platforms will be required to have CAT engine-specific code related to the differing data requirements. While the CAT engine API is standardized by this specification, data requirements cannot be completely standardized, without loss of CAT engine flexibility. This comes at the cost of portability in some cases. We return to this topic in Section 7 of this document.

As a final introductory note, it should be mentioned that it was not a goal of this specification to add new selection and sequencing features to QTI itself. For example, the qti-selection, qti-ordering, qti-pre-condition and qti - branch-rule elements of QTI v3.0 might have been enhanced to allow more dynamic, "CAT-like", item selection and sequencing. Or a portable method for plugging custom sequencing modules into a QTI v3.0 delivery engine (similar to Portable Custom Interactions) might have been specified. While the possibility of such features being added to future versions of QTI is not ruled out, they were beyond the scope of the current work.

1.3 Related Documents

• CAT Service Information Model
• CAT Service REST/JSON Binding
• CAT Service Swagger/OpenApi Specification

1.4 Relationship to 1EdTech Standards

• QTI
• APIP
• QTI Results Reporting
• QTI Metadata
• QTI Usage Data
• QTI Content Packaging
• OneRoster
• Access for All (AfA) Personal Needs and Preferences (PNP)
• Security Framework
• Vocabulary Definition Exchange (VDEX)
• Curriculum Standards Metadata (CSM)
• Learning Object Metadata (LOM)

1.5 Terminology

Access for All (AfA)

An 1EdTech specification activity which provides information models focused on characterizing individuals' particular needs within a learning context, so as to increase usability, decrease exclusion, and meet legal accessibility requirements. AfA has defined information models for the Personal Needs and Preferences data which are used in APIP v1.0 (AfA PNP v2.0) and in QTI 3.0 (AfA PNP v3.0).

Accessible Portable Item Protocol (APIP)

Accessible Portable Item Protocol® (APIP®) is a profile of QTI v2.2 which focuses on making QTI tests and items accessible to candidates with disabilities and special needs. The current version of APIP is v1.0. There is no APIP profile for QTI v3.0 because the accessibility features added by the APIP v1.0 profile to QTI v2.2 have become part of the core specification in QTI v3.0. See Elevated Accessibility.

Adaptive Engine

A synonym for CAT Engine.

Adaptive Section

A section within an QTI assessment test which is configured according to this specification, so that item selection and sequencing for that section are delegated by the delivery engine to an external CAT engine.

AfA

Acronym for AccessForAll®.

API

Acronym for Application Programming Interface.

APIP

Acronym for Accessible Portable Item Protocol.

Application Programming Interface (API)

In this document, refers to a REST web service interface which allows item selection and sequencing to be delegated by an Assessment Platform or Assessment Delivery Engine to an external CAT engine.

Assessment

Synonym for test.

Assessment Delivery Engine

A system module responsible for presenting test items and related content to a candidate, and for capturing the candidate's responses.

Assessment Platform

A system responsible for maintaining a database concerning test and item content, candidates, rosters of candidates, candidates' PNP data, test delivery schedules, and other data; and for orchestrating a delivery engine, CAT engine, and other internal and external modules or "engines" involved in item generation or cloning, sequencing, delivery, scoring, proctoring, results reporting, and analytics when assessments are administered to groups of candidates.

Base64

A binary-to-text encoding scheme which represents each 24 bits (3 octets) of the binary input as 4 radix-64 "digits", 1 octet per digit, of text output. Ten ASCII digits, 52 upper- and lower-case letters, and 2 special characters are used as output digits, and one special character is used as padding. Several Base64 variants exist. This specification uses the Base64 variant specified by RFC 4648, which is the variant most commonly used on the Internet.

Black Box

In engineering and computer science generally, a system, device, module, or object which can be viewed entirely in terms of its inputs and outputs with no knowledge of its internal state or operation. In this specification, CAT engines are treated as black boxes: the input consists of section configuration data, candidate data (possibly), and candidate responses to items and other processing variables; the output consists of item stages and outcome variable values.

Candidate

A person whose traits (such as knowledge, ability, attitude, etc.) are being measured through a test, and to whom the items of the test are to be presented. The person "taking" the test.

CAT

Acronym for Computer Adaptive Testing.

CAT Engine

A system module responsible for dynamically selecting the next item or stage (set of items) to be presented to the candidate in an adaptive section of a test, using a Computer Adaptive Testing (CAT) algorithm. Because each candidate is potentially presented with a different selection of items from the Item Pool, a simple sum of item scores is not usually a good indicator of the performance of a candidate on an adaptive section. Consequently, CAT engines are typically also responsible for providing item- and test-level scores and other outcome values.

CAT Service

A synonym for CAT engine.

Classical Test Theory (CTT)

A psychometric paradigm in which the magnitude of a personal trait (knowledge, ability, attitude, etc) is estimated from the person's overall score on a test. The "true score" representing the magnitude of the trait of interest, is assumed to be the person's "observed score" on the test plus "error". Psychometricians working within CTT focus on characterizing the reliability of a test (that is, the amount of error) and on minimizing the error. CTT is implicitly the theory underpinning most traditional testing, with varying degrees of attention in practice to the possibility of error. CTT is often contrasted with Item Response Theory (IRT).

Client Credentials Flow

A server-to-server OAuth2 flow or grant type whereby a client obtains an access token from a server on its own behalf, outside of the context of a user, using a client id and client secret. In the context of this specification, the client is an assessment platform or delivery engine, and the server is a CAT engine.

Computer Adaptive Testing (CAT)

A form of test delivery in which the items presented to candidates are chosen item-by-item by a computer algorithm during delivery of the test, with the aim of adapting the test dynamically to the ability level of each candidate, subject to various additional constraints. CAT algorithms take into consideration such things as: the candidate's responses to previous items, the item response functions of items in the Item Pool, running estimates of the candidate's ability relative to the items, the candidate's Personal Needs and Preferences, relationships between the items, coverage of curriculum standards, the distribution of item interaction types, and item exposure goals or constraints. CAT is distinguished from the non-adaptive testing familiar from paper tests, where the same set of items is presented to all candidates. CAT is also distinguished from Staged Adaptive Testing. CAT often allows tests with fewer items than conventional tests, while arriving at more accurate scores, especially for candidates of high or low, rather than medium, ability. CAT algorithms are often, though not invariably, based on Item Response Theory (IRT).

Content Package

In this specification, a collection of files required to deliver an assessment. For QTI, a content package is a ZIP archive containing the files of the package, plus a manifest. A content package may contain the resources for several QTI tests. A content package is a convenience which facilitates import, export, storage and transmission of test resources. QTI delivery engines do not require that all the resources involved in a test be in a content package, since resources may also be included in the QTI assessment by means of URL references, with the resources located elsewhere. QTI Content Packages must conform to a QTI-defined profile of the 1EdTech Content Packaging specification.

Conventional Section

A section in a QTI assessment which is not an adaptive section, from which items are selected and ordered by the delivery engine as specified by the selection and ordering elements of the section, without involvement of an external CAT engine.

CSM

Acronym for Curriculum Standards Metadata.

CTT

Acronym for Classical Test Theory.

Curriculum Standards Metadata (CSM)

A data model developed by 1EdTech, generally encoded in XML, which represents how learning content, such as a QTI assessment item, is associated with curriculum standards. In the context of this specification, metadata in this format may be used to inform a CAT engine of the relationship between items in the item pool and particular curriculum standards -- which may be used by the CAT engine in conjunction with targets or constraints for coverage of particular standards.

Delivery Engine

Abbreviation of Assessment Delivery Engine.

Demographics

As defined in OneRoster v1.1, consists of data concerning the race or ethnicity, gender, date and place of birth, and public school residence status of a user. Optionally used in this specification in the Create Session request.

Elevated Accessibility

A conformance level defined for QTI v3.0 which requires delivery engines to be capable of processing AfA 3.0 PNPs. It also requires delivery engines to provide many (but by no means all) of the accessibility supports called for in AfA 3.0 PNPs.

Implicit Grant

An OAuth2 grant type wherein a public client (such as a browser or mobile application) directly obtains an access token for a secured resource. Used in this specification in conjunction with Third Party Login Initiation to allow browser-based delivery engines to make CAT API requests directly.

1EdTech

The 1EdTech Consortium®, a non-profit collaborative advancing educational technology interoperability, innovation, and learning impact.

IRT

Acronym for Item Response Theory.

Item

In the context of testing in general, a specific task which test takers are asked to perform. In the context of QTI, an item is a coherent unit of assessment content comprising one or more standardized interactions with the candidate, usually producing a response or responses and an outcome or set of outcomes, such as a score. In QTI, items reside in separate qti-assessment-item XML files (in v3.0) or in assessmentItem files (in v2.x), which are incorporated into assessment sections by reference. A section will typically comprise one or more items.

Item Metadata

A specification in the QTI suite of specifications which defines the qtiMetadata XML element, which may be used in a manifest to describe the following aspects of an assessment item resource: the interaction types used; whether or not feedback is adaptive; the scoring mode; whether it is composite; whether it has templates; whether a solution is available; whether it is time-dependent; the name, version, and vendor of the authoring tool used to create the item; and LOM or CSM data. A file containing item metadata using this qtiMetadata XML syntax can optionally be sent to the CAT engine in the Create Section request.

Item Ordering

A synonym for "item sequencing".

Item Pool

The set of items from which the CAT engine can select in an adaptive section.

Item Response Theory (IRT)

A psychometric paradigm in which an item response function is estimated for each test item, giving the probability that a person with a trait (knowledge, ability, attitude, etc) of specified magnitude will have a particular outcome (e. g. give the correct response) after interaction with the item. In IRT-based testing, the candidate trait of interest is estimated from the candidate's item responses and the item response functions. Item Response Theory is contrasted with, and is sometimes considered to have superseded, Classical Test Theory (CTT)a. IRT can be used to estimate the magnitude of a candidate trait even when the various candidates are presented with different items, including different numbers of items. While CAT algorithms can be based on many testing paradigms, it is common for CAT algorithms to be based on IRT.

Item Selection

The process of determining the subset of items from the Item Pool which should be presented to the candidate. It may be done "statically"; that is, before any items are presented to the candidate. Or it may be done "dynamically"; that is, the next item or stage is selected after the candidate has responded to the previous item or stage. Selection by CAT engines is dynamic. Item selection in a conventional QTI section is essentially static, but items may be dynamically skipped or branched over during delivery.

Item Sequencing

The process of determining the order in which items that have been selected from the Item Pool should be presented to the candidate. Item sequencing in QTI is done statically, but preconditions and branch rules may be used to skip or branch over items in the static sequence, making the sequence, as delivered, somewhat dynamic. Item sequencing by CAT engines is dynamic.

Item Stage

A list or batch of one or more items from the Item Pool. In staged testing, the candidate proceeds through a series of stages. The determination of the items in each stage may be done statically, before any stages are presented to the candidate; or dynamically, stage-by-stage, based on previous events in the candidate session, such as the candidate's responses. In some systems, as in this specification, a candidate may not be required to complete all the items in the stage before proceeding to the next stage. In staged testing, the item sequencing algorithm outputs a sequence of stages rather than a sequence of items. CAT engines in this specification emit stages, but the stage length may be as little as one, and a delivery engine is not required to present all the items in a stage to the candidate—both of which enable item-by-item sequencing, when needed.

Item Usage Data

A specification in the QTI suite of specifications which defines XML syntax for encoding "usage" statistics related to assessment items. The Usage Data specification does not define specific statistics, but provides an abstract syntax for transmitting statistics. The statistics themselves must be defined elsewhere. One option for defining statistics is a VDEX glossary, and the 1EdTech QTI activity has issued a VDEX-specified Item Statistics glossary, defining many IRT-related statistics which are potentially applicable in the context of CAT engines.

JavaScript Object Notation (JSON)

A lightweight data-interchange format based on a subset of the syntax of the JavaScript programming language used for defining objects, standardized by [RFC8259].

JSON

Acronym for JavaScript Object Notation.

JSON Web Token (JWT)

An Internet standard ([RFC7519]) for using JSON to encode a set of claims into a signed token. In the context of this specification, JWTs serve as access or identity tokens.

JWT

Acronym for JSON Web Token.

Learning Object Metadata (LOM)

A data model generally encoded in XML which describes digital resources ("learning objects") used to support learning. Jointly developed by 1EdTech and IEEE, LOM can be used in the context of this specification to describe such things as the interaction style and format of QTI items, and the relationships between items.

LOM

Acronym for Learning Object Metadata.

Manifest

The XML file in an 1EdTech Content Package which describes the structure of the package, listing the resources and files comprising the package, sometimes along with their dependencies and metadata describing the resources.

Metadata

See Item Metadata.

OAuth2

An industry-standard protocol for authorization which provides specific authorization flows for web applications, desktop applications, mobile phones, and devices. The OAuth2 Client Credentials and Implicit Grant flows are used for authorization in this specification.

OneRoster

OneRoster® is a specification within the 1EdTech suite of Learning Information Services specifications which defines an Information Model and API for the exchange of data concerning schools, users, courses, academic terms, class rosters, course and class resources, and gradebooks. Conformant OneRoster systems can bulk load information via CSV files or in real time using a REST API. The OneRoster v1.1 definition of user demographics is used for candidate demographics in this specification.

OpenApi Specification

A JSON or YAML format for describing a RESTful Web interface, originally developed as part of the Swagger framework (and sometimes referred to as the Swagger Specification or OpenApi/Swagger Specification). It is now maintained by the Open API Initiative. An OpenApi description of the CAT Service API is one of the documents provided with this specification.

OpenID Connect

A profile or layer on top of the OAuth2 protocol. Whereas OAuth2 by itself is designed for authorization (that is, to define what resources a principal may access), OpenID Connect allows a party to a network communication to securely authenticate ( or identify) itself. This specification uses a variant of the OpenID Connect Login flow to enable CAT API requests to be securely made from browsers and mobile applications on behalf of an end-user of an assessment platform.

Personal Needs and Preferences (PNP)

In this specification, data related to the needs and preferences of individual candidates with respect to the online delivery of tests, focusing particularly on the needs of candidates with disabilities or with heightened or non-default accessibility requirements. In the APIP v1.0 profile of QTI v2.x, PNP data is provided by an XML data file in the AccessForAll® (AfA®) 2.0 format, and in QTI v3.0, the AfA 3.0 format.

PNP

Acronym for Personal Needs and Preferences.

QTI

Acronym for Question and Test Interoperability®.

Question and Test Interoperability (QTI)

Question and Test Interoperability® (QTI®) is an 1EdTech specification enabling the exchange of item and test content and results data between authoring tools, item banks, test construction tools, learning platforms, assessment delivery systems, and scoring/analytics engines.

REST

Acronym for Representational State Transfer, a style for implementing web services. Web services that are in the REST style are often said to be "RESTful". A set of conventions and rules of thumb, rather than an official specification.

RESTful

An adjective applicable to a web service implemented in the REST style.

Section

A section of a test grouping a set of items. In QTI, besides items a section may also contain sub-sections, which in turn may contain sub-sections or items, so that sections and their items form a hierarchy.

Session

The administration of a test or a section to one candidate. In some systems and circumstances, a candidate may have multiple sessions with the test or section, as when a candidate "retakes" a test, or completes multiple parts of a test over several distinct sessions.

Stage

See Item Stage.

Staged Adaptive Testing

A form (sometimes considered a precursor) of Computer Adaptive Testing where candidates proceed from one pre-defined stage or batch of items to another, with different candidates possibly proceeding through different stages depending on their responses/scores on previous stages. QTI selection, ordering, and conditional skipping and branching lend themselves quite well to implementing Staged Adaptive Testing. Staged Adaptive Testing can be regarded as a constrained form of CAT.

Stateful

Antonym of stateless. In a stateful protocol or system, a communicating entity must retain information ("state") from the processing of previous messages in order to correctly process subsequent messages.

Stateless

A term applied to a communications system (typically, a server or receiver) which does not retain any information between messages concerning the state of its communication with other communicating systems (for example, the clients or senders). For a protocol or system to be stateless, each input message must be self-contained and provide all the information required for the processing of that message, without reference to previous input and output messages in the communication session. Stateless protocols are often preferred in high-volume applications because they reduce server load whenever constructing the state required to process a message is less expensive than retaining state from previous message processing. The opposite of stateless is stateful. The CAT Service API is defined so as to allow CAT engines to be either stateless or stateful. In this specification, statelessness is achieved by having the CAT engine return any session state data to the assessment platform in the sessionState property of the response, which is then sent back to the CAT engine on the next request by the assessment platform. In between the two requests, the CAT engine need not retain the state of the session.

Swagger

A software framework which facilitates the development of RESTful web services. One element of this framework was the Swagger Specification, which has since been contributed to the Open API Initiative, becoming the OpenApi Specification.

Swagger Specification

The former name of the OpenApi Specification.

Test Administrator

A person or organizational entity who arranges for a test to be delivered to a population of candidates.

Test Constructor

A person or organizational entity responsible for sourcing the items and other content of the test and assembling them into a coherent whole which can be delivered via one or more assessment platforms to one or more populations of candidates. In the context of this specification, the work product of the test constructor is a set of QTI XML files, web content, and possibly other files, usually assembled into a content package. The test constructor is commonly the author of the test-level QTI artifacts such as assessment test files, section files, and the content package manifest.

Third-Party Login Initiation

A variant of the OAuth2 Implicit flow wherein the "login" (that is, transmission of identity) made by a browser-based delivery engine on the CAT Engine is initiated by a third-party (in this case, the assessment platform, functioning as the identity provider).

VDEX

Acronym for Vocabulary Definition Exchange.

Vocabulary Definition Exchange (VDEX)

An 1EdTech specification which defines an XML grammar for specifying vocabularies or glossaries. A "VDEX file" is an XML file which defines a specific vocabulary or glossary. In the context of this specification, a VDEX file might be used to enumerate the specific statistics to be included in a QTI Usage Data XML file, with the glossary attribute of the usageData element giving the URL of a VDEX file for the glossary. The 1EdTech QTI activity has standardized a VDEX vocabulary for Item Statistics (itemstatisticsglossaryv1p0),which includes many IRT item parameters.

1.6 Document Set

2.1 XML Syntax

For QTI v3.0, the adaptive section configuration is provided by the new qti-adaptive-selection element in the XML source for the section:

Example: QTI v3.0 Adaptive Section

001
002
003
004
005
006
007
008
009
010
011
012
013

<qti-assessment-section … >
 …
 <qti-adaptive-selection>
  <qti-adaptive-engine-ref identifier="eng" href="http://example.org/eng"/>
  <qti-adaptive-settings-ref identifier="settings" href="settings.dat"/>
  <qti-usagedata-ref identifier="usagedata" href="usagedata.xml"/>
  <qti-metadata-ref identifier="metadata" href="metadata.json"/>
 </qti-adaptive-selection>
 …
 <qti-assessment-item-ref identifier="item434" href="item434.xml"/>
 <qti-assessment-item-ref identifier="item065" href="item065.xml"/>
 …
</qti-assessment-section>

The qti-adaptive-selection element in QTI v3.0 is an alternative to the qti-selection and qti-ordering elements already defined in QTI v3.0. A qti-assessment-section may have EITHER qti-adaptive-selection OR qti-selection and/or qti-ordering, but not both.

In QTI v2.x, for the sake of backwards compatibility, the adaptive section configuration is embedded in a selection element. Because the selection element is defined as a QTI "extension point", which may contain arbitrary markup that is ignored by assessment platforms which do not understand it, the new CAT-related markup is hidden from validators and conformant QTI v2.x implementations which have not been upgraded to support this specification.

Example: QTI v2.1 / v2.2 Adaptive Section

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015

<assessmentSection … >
 …
 <selection>
  <adaptiveItemSelection
  xmlns="https://purl.imsglobal.org/spec/cat/v1p0/schema/xsd/qtiv2-ais.xsd">
   <adaptiveEngineRef identifier="engine" href="http://example.org/engine"/>
   <adaptiveSettingsRef identifier="settings" href="settings.data"/>
   <qtiUsagedataRef identifier="usagedata" href="usagedata.xml"/>
   <qtiMetadataRef identifier="metadata" href="metadata.json"/>
  </adaptiveItemSelection> 
 </selection>
 …
 <assessmentItemRef identifier="item434" href="item434.xml"/>
 <assessmentItemRef identifier="item065" href="item065.xml"/>
 …
</assessmentSection>

In QTI v2.x, when used to wrap an adaptive section definition, the selection element should not include the standard select or withReplacement attributes, A section containing a selection with adaptiveItemSelection should also not have an ordering element. A delivery engine should ignore select, withReplacement, and ordering in adaptive sections.

The qti-assessment-section (in v3.0) or assessmentSection (in v2.x) should list the item references for the items in the section, as usual. These are the items in the "item pool" for the adaptive section, and the item list used by the adaptive engine (e.g. provided to the engine in the adaptive section configuration) should correspond exactly to the item references listed in the QTI XML source for the adaptive section.

It should be noted that the item list in the QTI XML source is not used to configure the adaptive section on the CAT engine. The list of items in the item pool is configured on the CAT engine by the data passed to the engine in the Create Section request. Alternatively, it may have been done through some other mechanism beyond the scope of this specification. It is the responsibility of the test administrator to ensure that the list of items in the QTI XML and the items configured in the CAT engine item pool correspond. Verification that the item list is synchronized between the CAT engine and delivery engine is facilitated by the Get Section API request.

2.2 Adaptive Section Configuration

When a QTI assessment test containing adaptive sections is registered on a QTI v2.x or QTI v3.0 assessment platform, the platform should use the Create Section endpoint of the CAT engine to configure those sections on the referenced CAT engines.

The child elements of the qti-adaptive-selection or adaptiveItemSelection element all relate to the Create Section request of the API and essentially provide the properties to the assessment platform for making that request. One of the child elements provides the URL prefix for the CAT Service API endpoints, including the Create Section endpoint, and the other three child elements give URL references to configuration data files which, after Base64-encoding, become the values of the fields in the request body of the Create Section request:

Whether pre-configured via a user interface or some other mechanism, or provided in the sectionConfiguration settings, the list of items contained in the Item Pool must be determined and the CAT engine's version of the Item Pool and the list of assessment item references in the QTI XML for the adaptive section must be in agreement. The assessment platform should also verify that the item references in the QTI XML are resolvable to deliverable assessment items.

qtiUsagedataRef may reference a file with usage statistics for items in the item pool. This parameter may also serve as a method for enumerating the items in the item pool. While the file must conform to the Usage Data model, the glossary or vocabulary used for the Usage Data is not constrained by this specification and is CAT engine-dependent. A recommended choice of vocabulary, suitable for IRT-based CAT engines, is the 1EdTech Usage Data Item Statistics VDEX, which defines typical IRT-related item parameters. However, CAT engines are not constrained to this vocabulary. If the usage statistics parameter is used, the vocabulary and content of the file are CAT engine-specific.

qtiMetadataRef may reference a file containing metadata about items in the item pool, such as the types of interactions used, whether or not an item is composite, whether a solution is provided, whether the item includes templates, and Learning Object Metadata (LOM) or Curriculum Standards Metadata (CSM) for the items. The referenced file is a JSON file with a format which is derived from the QTI Item Metadata information model. See the CAT Service Information Model [[@@@]] for details.

2.3 Restrictions

An adaptive section must not have nested subsections and cannot be nested in another QTI assessment section. That is, an adaptive section must be one of the top-level sections in the QTI test part containing it, and must specify all the items in the item pool directly, not through sections nested within it.

A test part containing multiple sections may have a mix of adaptive and conventional sections, and there may be more than one adaptive section in a test part. Adaptive sections within a test part may reference different CAT engines.

In QTI, the navigation mode of a test part may be either linear or nonlinear, which determines how the delivery engine presents items to the candidate and controls how the candidate may interact with the items.

In a linear test part the selected items are presented one at a time to the candidate, and the candidate may make one or more attempts on each item before submitting it. Once an item is submitted, it is not possible for the candidate to return to the item, unless the item is configured to allow "review". Review, if allowed, is read-only, permitting the candidate to see the item, as submitted, but not to make further attempts or to resubmit the item.

In a test part with nonlinear navigation mode and conventional sections, all the selected items in the test part are (in concept) presented to the candidate in parallel, and the candidate may decide the order in which he or she interacts with the items. A candidate may make an attempt on an item, move to a different item, come back to the first item and make another attempt, and so forth. Similar to a traditional paper test, in a nonlinear part, the candidates determine the order in which to work on and submit the items, in some cases (if permitted) making more than one attempt on items. The QTI specification explicitly permits delivery engines to present nonlinear test parts in the same, one-item-at-time, slideshow style as linear test parts, provided there are UI features (such as a "previous item" button), which permit the candidate to move back and forth and freely interact, nonlinearly, with all the items in the test part.

To maximize interoperability, it is recommended that test parts containing adaptive sections be linear test parts.

Use of nonlinear test parts with this specification is feasible, and is allowed by this specification. A test part with an adaptive section may be considered nonlinear, even though the set of items in the test part is not predetermined and not all items are initially presented to the candidate, provided the candidate can interact non-linearly with all items once they have been determined. Non-linear adaptive sections may continue to "grow at the end", item-by-item or stage-by-stage in response to candidate interactions, but for the section to be nonlinear, the candidate must be able to interact with all items presented so far. Beware that this will require CAT engines to handle nonlinear results sequences. (As in conventional sections, this sequence can be constrained somewhat by Item Session Control parameters like max-attempts).

In a nonlinear test part a candidate may interact with any item already presented in the adaptive section, as well as items in other sections of the nonlinear test part, in any order of the candidate's choosing. Accordingly, in a nonlinear test part, it is possible for a CAT engine to receive result reports related to any item in the adaptive section previously sent in the session by the CAT engine, in arbitrary order, possibly including items which have already been submitted one or more times in the session, or previously indicated as "skipped". An item response or score may change. The CAT engine may have included items in a stage because of the scores on previous items, only to see those scores change, making the item less appropriate than it was when sent. However, this specification makes no provision for a CAT engine to "take back" items.

Nonlinear CAT can be complex, is the subject of ongoing experimentation, and is not generally supported by CAT engines. This specification allows, but does not require, support for nonlinear CAT on the part of CAT engines or delivery engines. But most CAT engines are apt to be linear, and delivery engine integration with CAT engines, likewise. Consequently, tests with adaptive sections in nonlinear test parts are apt to have more limited interoperability than in linear test parts.

Though branch rules and preconditions are normally allowed in linear test parts, they are not allowed on items in adaptive sections. A delivery engine should ignore any branch rules and preconditions on items in adaptive sections. Branch rules which target items in an adaptive section are also not permitted, and such branch rules should also be ignored. Branch rules and preconditions are still allowed on the adaptive sections themselves, and adaptive sections as a whole may be targeted by branch rules.

All QTI item-level features except those noted may be used within adaptive section items. However, item authors and test constructors should consider whether the features used within an item are compatible with the testing paradigm of the chosen CAT engine. Features which may not be fully compatible with some CAT approaches include: time limits, weights, templates, and feedback (particularly adaptive feedback).

2.4 Item Processing Flow

In a test part with individual submission mode, results are "submitted" by the delivery engine after each attempt on an item. With individual submission mode, when the item is part of an adaptive section, the sequence of operations after an item attempt/submission should be:

1. The candidate completes the attempt, and response variable values are set.
2. The delivery engine runs Response and Outcome processing.
3. Possibly, the delivery engine invokes the Submit Results endpoint of the CAT engine to determine the next item or stage in the adaptive section to be presented. The results sent to the CAT engine include the values set in the Step 2 processing, plus item results determined in response to attempts on previous items, but not sent to the CAT engine at that time. If there remain unpresented items in the stage, this step may be deferred until there are no more unpresented items, or the delivery engine opts to proceed to the next stage.
4. The delivery engine "submits" the item. If it supports QTI Results Reporting, the delivery engine builds a results report, which merges the values of builtin and declared outcome variables with outcome variable values returned by the CAT engine in its Submit Results responses, and sends the report to a QTI Results Reporting endpoint. Otherwise, it does whatever "submission" means in the context of the particular delivery engine.
5. The delivery engine adds the new items sent in the Submit Results response to the end of the section.
6. The delivery engine presents the next item to the candidate.

QTI also allows test parts to be configured in simultaneous submission mode. In such a test part, a delivery engine may defer Response and Outcome processing on all items in the test part until the end of the test part, and submission is done for all items in one batch, at the end of the test part. The QTI specification states that in test parts with simultaneous submission mode, items may only be attempted once, and whether they may be reviewed or whether item-level feedback is displayed are out of the scope of the QTI specification.

If adaptive sections are included in simultaneous submission test parts the steps above for individual submission test parts should still be followed by a delivery engine, except that Step 4 should be omitted item-by-item and instead item submission should be deferred until the end of the test part. This makes the processing flow for simultaneous submission in conjunction with adaptive sections possibly different than the usual simultaneous submission flow, as a delivery engine interfaced to an external CAT engine may no longer defer Response and Outcome Processing to the end of the test part, but must do it before sending Submit Results requests to the CAT engine.

2.5 Sample Tests

This section provides sample tests for QTI v3.0 and QTI v2.x containing adaptive sections.

Example: QTI v3.0 Sample Test with Adaptive Section

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024

<qti-assessment-test xmlns="http://www.imsglobal.org/xsd/imsqti_v3p0"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation=
 "http://www.imsglobal.org/xsd/imsqti_v3p0
  https://purl.imsglobal.org/spec/qti/v3p0/schema/xsd/imsqti_asiv3p0_v1p0.xsd"
 identifier="ATEST" title="Sample Test with Adaptive Section">
 <qti-test-part navigation-mode="linear" submission-mode="individual" 
  identifier="ATEST-1">
<qti-assessment-section identifier="sec" title="Adaptive Section">
   <qti-adaptive-selection>
    <qti-adaptive-engine-ref identifier="engine" href="http://example.org/engine"/>
    <qti-adaptive-settings-ref identifier="settings" href="settings.data"/>
    <qti-usagedata-ref identifier="usagedata" href="usagedata.xml"/>
    <qti-metadata-ref identifier="metadata" href="metadata.json"/>
   </qti-adaptive-selection>
   <qti-assessment-item-ref identifier="item434" href="item434.xml"/>
   <qti-assessment-item-ref identifier="item065" href="item065.xml"/>
   <qti-assessment-item-ref identifier="item365" href="item365.xml"/>
   <qti-assessment-item-ref identifier="item347" href="item347.xml"/>
   <qti-assessment-item-ref identifier="item653" href="item653.xml"/>
   <qti-assessment-item-ref identifier="item656" href="item656.xml"/>
  </qti-assessment-section>
 </qti-test-part>
</qti-assessment-test>

Example: QTI v2.x Sample Test with Adaptive Section

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028

<assessmentTest
 xmlns="http://www.imsglobal.org/xsd/imsqti_v2p2"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation=
 "http://www.imsglobal.org/xsd/imsqti_v2p2
  http://www.imsglobal.org/xsd/qti/qtiv2p2/imsqti_v2p2.xsd"
 identifier="ATEST" title="Sample Test with Adaptive Section">
 <testPart navigationMode="linear" submissionMode="individual" 
  identifier="ATEST-1">
<assessmentSection identifier="sec" title="Adaptive Section">
   <selection>
    <adaptiveItemSelection
    xmlns="https://purl.imsglobal.org/spec/cat/v1p0/schema/xsd/qtiv2-ais.xsd">>
     <adaptiveEngineRef identifier="eng" href="
http://example.org/engine
" />
     <adaptiveSettingsRef identifier="settings" href="settings.data" />
     <qtiUsagedataRef identifier="usagedata" href="usagedata.xml" />
     <qtiMetadataRef identifier="metadata" href="metadata.json" />
    </adaptiveItemSelection>
   </selection>
   <assessmentItemRef identifier="item434" href="item434.xml"/>
   <assessmentItemRef identifier="item065" href="item065.xml"/>
   <assessmentItemRef identifier="item365" href="item365.xml"/>
   <assessmentItemRef identifier="item347" href="item347.xml"/>
   <assessmentItemRef identifier="item653" href="item653.xml"/>
   <assessmentItemRef identifier="item656" href="item656.xml"/>
  </assessmentSection>
 </testPart>
</assessmentTest>

2.6 Content Packaging

In adaptiveItemSelection / qti-adaptive-selection elements, if any of the URLs referring to the three adaptive section configuration files resolve to the content package, the files in question should be included in the content package and listed in the manifest. (This follows from the requirement for all files in a conformant QTI content package to be listed in the manifest.)

The example below shows an extract from the content package for the sample tests given above. The delivery engine would resolve the href attributes on the adaptiveSettingsRef, qtiUsagedatRef, and qtiMetadataRef files to the content package, assuming the assessmentTest XML file came from the content package.

Example: Adaptive Section Configuration Files Added to QTI v3.0 Manifest

001
002
003
004
005
006
007

<resource identifier="ATEST" type="imsqti_test_xmlv3p0" href="test.xml"
  <file href="test.xml"/>
 <file href="settings.data"/>
  <file href="usagedata.xml"/>
  <file href="metadata.json"/>
  …
</resource>

These additions to the manifest would not be necessary for a configuration file URL which resolved to an external location, rather than to a file included in the content package, as would be the case if they were absolute URLs rather than the relative URLs used in the examples.

If in the example the adaptive section had been included in a standalone XML file for the assessmentSection, rather than in the XML file for the assessmentTest, then the three configuration files would appear in the manifest under the resource element for the section (resource type imsqti_section_xmlv3p0) rather than the resource definition for the test.

3.1 Create Section

The Create Section request uploads an adaptive section configuration to a CAT engine. This request is not idempotent and should not be repeated, once it has succeeded. A CAT engine is not required to detect that the "same" section has been created more than once, but it may reject a request (with a "400 Bad Request" status), if it does.

In some CAT engine implementations, the adaptive section may already exist on the CAT engine, having previously been created via a user interface or some other mechanism beyond the scope of this specification. The sectionConfiguration within the Section Data might then consist of no more than an identifier of the pre-existing section.

3.2 Get Section

The Get Section request returns the parameters which were used by the CAT engine to create the adaptive section, and the list of the items in the adaptive section's item pool, as configured by the CAT engine. It is recommended that assessment platforms and/or delivery engines use this request to verify that the item pool, as configured by the CAT engine, corresponds exactly to the list of items in the QTI XML source for the adaptive section. It is also recommended that test administrators verify that each item in the item pool is accessible and deliverable before commencing delivery of candidate sessions. The Get Section request may be used for this, and may be made at any time after a successful Create Section request, as many times as required.

A CAT engine is not required to retain the exact parameters from the original Create Section request and return them in the Get Section response. The original parameters may in any case have included data which was invalid or disregarded by the CAT engine. Rather, the CAT engine should return parameters in the Get Section response which could be used to recreate the adaptive section as it currently exists on the CAT engine; in effect, Get Section returns a description of the section as actually deployed on the CAT engine.

3.3 Create Session

The Create Session request initializes a candidate session for the adaptive section. The request optionally provides the CAT engine with information related to the candidate. The CAT engine returns the initial stage (list of items) to be presented to the candidate. This request may be made multiple times after a successful Create Section request, once per candidate, when each candidate enters the adaptive section on the test.

3.4 Submit Results

The Submit Results request posts the results from the current item stage in a candidate session to the CAT engine. The CAT engine returns its computed outcome variable values for the current stage and the test as a whole. It also returns the next item stage to be presented to the candidate. Instead of the next item stage, it may also return an indication that the adaptive section has ended, by returning an empty or null list for nextItems.

The correct propagation of sessionState from previous response to next request is critical for maintaining the integrity of the candidate session. It is anticipated that many CAT engines will be stateless, and will rely for their correct operation on assessment platforms propagating sessionState from one request to the next.

CAT engines should validate the sequence of sessionStates, if possible, but are not required to do so, as this would necessitate the CAT engine maintaining state, which sessionState is designed to avoid. Because CAT engines may not be able to validate sessionStates, the responsibility of ensuring that requests with sessionStates are sent in the correct sequence falls on assessment platforms and delivery engines.

3.5 End Session

The End Session request ends the candidate session. The sessionIdentifier, and the sessionState returned from any previous request in the session, become invalid. The delivery engine should not make any further requests related to the session. This request uses the DELETE HTTP method because of REST conventions; but it is not required that CAT engines physically delete data or resources associated with the session. After a session is ended, Submit Results requests with the sessionIdentifier should return with an HTTP status of "404 Not Found."

Use of this request by delivery engines is optional. Generally, it is not necessary for a delivery engine to make this request and under normal circumstances it should not be used. The CAT engine normally ends sessions when it deems them completed, indicating that a session has ended by returning a null or empty list of items for the next stage in its response to the Submit Results request. The End Session request is provided by the API so that an assessment platform or delivery engine can indicate to the CAT engine that a candidate session should be aborted, exceptionally.

3.6 End Section

The End Section request ends all candidate sessions for the section which have not already ended, and ends the section. The sectionIdentifier, sessionIdentifiers, and sessionStates returned from previous requests related to the section become invalid. The delivery engine should not make any further requests related to the section or any session for the section. This request uses the DELETE HTTP method because of REST conventions, but the CAT engine is not required to physically delete any session or section data. This request should not be repeated once it has succeeded. After this request succeeds, requests related to the section in question should be returned with a "404 Not Found" status.

Use of this request by delivery engines is optional. Generally, it is not necessary for a delivery engine to make this request and under normal circumstances it should not be used. The endpoint exists so that an assessment platform can indicate to the CAT engine that delivery of a section and all related candidate sessions should be aborted, exceptionally.

4.1 Section Data

The Create Section request requires Section Data on input and the Get Section request returns Section Data in its response. This data is as follows:

4.2 Item Stage

The Create Session and the Submit Results requests return in their response bodies the next item stage to be presented in the candidate session. This data structure is also returned in the items property of the Get Section response, where it is used to represent all the items in the item pool of the adaptive section.

The items listed for the stage must be presented to the candidate in order. stageLength provides a hint from the CAT engine as to how many items from the state should be presented. It must be less than or equal to the itemIdentifiers list length. If N items from the list are presented to the candidate, they must be the first N items and, following the hint, N should be equal to stageLength, but may be more or less.

A CAT engine is allowed to call for an item in the pool to be presented more than once -- that is, an item identifier may appear in more than one item stage, or even more than once in the same item stage. A delivery engine should treat this as it would if with-replacement were true in a section with conventional QTI selection, or if an assessment item reference to the same item were to appear more than once in the test. That is, it should instance the multiply-selected items.

However, support of selection with-replacement, multiple selection of items, and item instancing generally, are advanced QTI features and are not required for conformance even in conventional QTI sections; so support of multiple selection of items and instancing in adaptive sections is not required for conformance to this specification. Accordingly, CAT engines will enhance their interoperability by not calling for an item from the pool to be presented more than once in a candidate session.

It is an error for a CAT engine to return a stage which contains only items which the candidate has previously skipped or to which the candidate has previously responded (unless the item or items in question are template items and may be instanced). However, such items may be included in an item stage provided they are accompanied by at least one new item. How a delivery engine should present such items is beyond the scope of this specification.

The CAT engine indicates that the candidate session has ended by omitting the nextItems property from the Submit Results response, or by giving it a null or empty value. A CAT engine may also indicate session-end by making itemIdentifiers null or empty, and setting stageLength to null or zero. At session-end, sessionState must also be null, empty, or omitted. After session-end, the sessionIdentifier becomes invalid and no further requests should be made to the CAT engine with that sessionIdentifier. A CAT engine may end a session before all the items in the item pool have been presented to the candidate.

At least one item stage, consisting of at least one item, must be administered to the candidate from the adaptive section. Therefore, a candidate session for an adaptive section must consist, minimally, of:

• one Create Session request returning at minimum a one-item stage; and
• one Submit Results request, with the results for the one stage in the request body, with the response returning the CAT engine results and a session-end indication in the response body.

4.3 Assessment Result

Assessment results are transmitted by the assessment platform to the CAT engine in the request body of the Submit Results request, and are returned by the CAT engine in the response body, in both cases using the assessmentResults property.

5.1 Access Token Scopes

Access token scopes are an optional feature of OAuth2. When used with the Client Credentials flow, clients may request access tokens with specific named scopes from the access token endpoint of the CAT engine, and the CAT engine may issue tokens with none, some, or all of the requested scopes, or different scopes than those requested. The CAT engine may also reject requests for scopes. The response to the access token request indicates the scopes granted. If used, token scopes allow a CAT engine to restrict access by clients to designated subsets of the CAT Service API.

The only requirement imposed by this specification regarding access token scopes is that a CAT engine must not reject an otherwise valid request (that is, valid client_id and client_secret) which omits the scope parameter, or which does not include scopes which it recognizes. In those cases the CAT engine access token endpoint must return an access token with a default scope providing access to, at least, the Create Session, Submit Results, and End Session endpoints of the CAT Service API, for adaptive sections associated with the client_id in question.

Beyond this requirement, a CAT engine implementation may adopt any scope structure and naming scheme which meets its requirements, including not using scopes at all. Beware that using scopes will potentially impact a CAT engine's portability, because conformant assessment platforms may not be prepared to handle the CAT engine's scope structure.

Accordingly, if scopes are used at all by a CAT engine, in the interests of interoperability with assessment platforms, this specification recommends, but does not require, the following scope names and structure:

In this scheme, deliver is the default scope: an access token with either the deliver scope, or the empty scope (equivalent to deliver) should be issued if a client request does not specify a scope, or specifies an unrecognized or disallowed scope.

6.1 CAT Engines

In order to be conformant with this specification, CAT engines:

1. Must implement all six API endpoints and provide all required and all conditionally required response fields, in cases where the conditions are met.
2. Must reject requests with the proper error code when request fields identified by this specification as required are missing or invalid, and likewise for fields identified as conditionally required in those cases where the conditions are met.
3. Must accept requests when fields identified by this specification as optional are omitted, incomplete, or invalid, and likewise for fields identified as conditionally required when the conditions are not met.
4. Must accept requests which contain unrecognized fields or subfields. (The delivery engine should ignore such fields.)
5. Must support the Server side of the OAuth2 Client Credentials flow; may also support the Tool side of the OpenID Connect with Third-Party Initiated Login flow.

A CAT engine may send empty or null values for, or entirely omit, response fields identified as optional.

6.2 Assessment Platforms and Delivery Engines

In order to be conformant with this specification, Assessment Platforms and Delivery Engines:

1. Must be QTI v2.1, QTI v2.2, APIP v1.0, or QTI v3.0 conformant, in each case at least at the entry- (or lowest) level of conformance.
2. Must be able to deploy QTI assessmentTest or assessmentSection XML files containing the adaptiveItemSelection element within a selection element, if a QTI v2.x/APIP v1.0 platform/engine.
3. Must be able to deploy QTI XML qti-assessment-test or qti-assessment-section files containing the qti-adaptive-selection element, if a QTI v3.0 platform/engine.
4. Must be able to correctly deliver an adaptive section as defined by this specification, by sending valid requests to the CAT engine Create Section, Create Session, and Submit Results endpoints, with all required fields, and all conditionally required fields where the conditions have been met.
5. Must capture outcome variables returned by the CAT engine as assessmentResults, and include these along with other context, template, response, and outcome variables generated by Response and Outcome Processing, in its own submission processing, including any QTI Results Reporting.
6. Must ignore unrecognized response fields and sub-fields, and must accept null or empty values for, or omission of, response fields identified as optional, or as conditionally required when the conditions are not met.
7. Must support the Client side of the OAuth2 Client Credentials flow; may also support the User Agent and Platform sides of the OpenID Connect with Third Party Initiated Login flow.

Assessment Platforms and Delivery Engines are not required to use the Get Section, End Session, or End Section endpoints of the CAT API, though the use of Get Section is recommended to verify adaptive sections after creation.

7.1 Assessment Platform Portability

If:

• an adaptive section does not use elective QTI features or QTI extension points other than selection; and
• the CAT engine does not require PNP, demographics, or priorData on the Create Session request

Then:

• an adaptive section when configured for that CAT engine should be deliverable on any assessment platform conformant with QTI and this specification.

This achieves one of the main portability goals of this specification.

If the CAT engine configured in an adaptive section DOES require PNP, demographics, or priorData, the adaptive section will be constrained to those assessment platforms which can furnish the data required by the CAT engine.

PNP data is currently required for conformance by assessment platforms with some 1EdTech accessibility-related standards. The APIP v1.0 profile of QTI v2.2 requires AfA v2 PNP support for conformance, and the Elevated Accessibility conformance level for QTI v3.0 requires AfA v3 support. As this specification is published, most QTI assessment platforms are not certified for APIP conformance and most likely do not have PNP data for candidates available. But some do, and it is not unreasonable to expect that more will do so in the future.

Demographics data is not currently required by any QTI assessment platform conformance level or profile, and is not needed even for elective features in the QTI standards. The demographics data format called for in this specification is the OneRoster v1.1 format, but it should be noted that while OneRoster v1.1 defines the format, no OneRoster conformance profile requires support for it. It is likely that in most cases OneRoster v1.1 demographics data is not available to QTI assessment platforms and would need to be developed specifically for the purposes of supporting this specification, where it is an optional feature.

In order to meet the goal of CAT engine flexibility, candidate priorData is not standardized in this specification, nor has standardization of this type of data been addressed by other standards bodies. Each CAT engine which requires candidate priorData is free to define its own format for this data. Meeting the input requirements of CAT engines for priorData will thus require delivery engines to recognize specific CAT engines and generate or obtain from a data store the per-candidate priorData required for each specific CAT engine which it elects to support. Examples of prior data include: results of previous (adaptive) tests or sections, candidate ability as estimated by a teacher, or a list of previously selected items (to prevent consecutive adaptive tests to select items already presented to the candidate).

Thus, a CAT engine requirement for priorData on the Create Session request narrows the set of delivery engines capable of delivering the assessment to those which can furnish the prior candidate data required by that specific CAT engine.

In summary, when interoperability and portability are important for an adaptive section:

• Portability is maximized if the CAT engine in question does NOT call for PNP, demographics, or priorData on the Create Session request.
• If required by the CAT engine, PNP data is the least likely to be an obstacle to assessment platform portability, but circumspection on the part of test constructors and administrators is advisable because availability of PNP data on assessment platforms is not universal.
• If required by the CAT engine, demographics data is very likely to be an impediment to portability because QTI assessment platforms generally do not have it available, and currently have no reason other than support for this specification to make it available.
• If required by the CAT engine, priorData will necessitate an integration for that specific CAT engine into assessment platforms so that the platforms furnish the required candidate priorData in the format expected by that specific CAT engine. The CAT engine will not be usable in conjunction with platforms which have not done such an integration.

It may be argued that the optional data fields of the Create Session request tend to defeat the portability goals of this specification. However, it is important to bear in mind that the CAT engine-specific integration required for an assessment platform to meet the data requirements of that CAT engine is likely to be significantly less effort than integrating an entirely non-conformant CAT engine would be, assuming the non-conformant CAT engine has the same data requirements as the conformant one, but also features an entirely proprietary, non-conformant API.

Finally, it should be mentioned that QTI as a whole has many elective features which are not required for QTI conformance certification and which are not uniformly supported by conformant assessment platforms, including some which are not supported by any currently certified platforms. In addition, QTI contains a variety of "extension points", the use of which is inherently not portable across assessment platforms. Use of elective QTI features and extension points will limit the portability of adaptive sections in the same way as it limits the portability of conventional sections.

7.2 CAT Engine Portability

Adaptive sections and their associated data files contain configuration for a specific CAT engine; so an adaptive section is not portable to a different CAT engine.

For a new CAT engine, at least the engine URL would be inapplicable, and in all likelihood the settings would also be different, as these are CAT engine-specific. If usage statistics are included, those would possibly also be different because the content of usage statistics files is not standardized, though their XML format is. In the case of usage statistics, the test constructor or administrator might be fortunate enough to be moving from a CAT engine that uses the 1EdTech-standardized Item Statistics glossary (VDEX) to another CAT engine which uses the same glossary; but this is not guaranteed.

It is, of course, possible to rewrite an adaptive section and the associated configuration data files for a different CAT engine. Usually, this would involve providing a new set of CAT-engine specific settings, possibly providing a new set of CAT-engine specific item usage statistics, and altering the adaptive section element ( adaptiveItemSelection or qti-item-selection) to point to the changed data files and the new CAT engine endpoints.

Some of the work of configuring the adaptive section for the original CAT engine may be reusable when reconfiguring it for a new CAT engine. In general, the ease or difficulty of configuring or reconfiguring an adaptive section for a CAT engine will depend on the engine. But it is expected that in most cases rewriting an adaptive section to use a different CAT engine will not be a significant effort.

Having reconfigured an adaptive section for a new CAT engine, how likely is it that the new CAT engine will interoperate correctly with the assessment platform where the adaptive section is currently deployed?

This is very similar to the question of assessment platform portability. In general, an adaptive section can be reconfigured to use a different CAT engine and still run on the same assessment platform, without issue, provided the new CAT engine is conformant with this specification and has no requirements for PNP, demographics, or candidate priorData.

As stated in the previous section, if the new CAT engine does require any of the optional data elements in the Create Session request, then it will be possible to switch to it, while staying on the same assessment platform, ONLY IF the current assessment platform can furnish the data required by the new CAT engine. This is not assured with PNP data, even less assured with demographics, and true with priorData only if the assessment platform has specifically been integrated with the CAT engine in question.

With that caution stated, it should be pointed out that if an assessment platform must be enhanced to meet the input data requirements of a particular CAT engine, such an integration will be facilitated by the fact that the CAT engine supports the standard API defined in this specification. Because of this, the integration needs only to address the CAT engine's data requirements, rather than having to implement support for a different API.