IMS Final Release

IMS Logo

IMS Caliper Analytics™ Conformance and Certification Guide

 

Version 1.0 Final

Date Issued:            15 October 2015

Latest version:         http://www.imsglobal.org/caliper

IPR and Distribution Notices
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS’s procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Copyright © 2015 IMS Global Learning Consortium. All Rights Reserved.

Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/license.html.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

© 2015 IMS Global Learning Consortium, Inc.
All Rights Reserved.
Trademark information: http://www.imsglobal.org/copyright.html
Document Name:  IMS Global Caliper Analytics Conformance and Certification Guide – Final v1.0
Revision: 15 October 2015

 

Table of Contents

1  Introduction

   1.1  IMS Caliper Analytics

   1.2  IMS Caliper Analytics Community

2  Conformance Overview

   2.1  Definition of Conformance

   2.2  Conformance Certifications

   2.3  Conformance Process

   2.4  The Conformance Certification Process

3  Conformance Test Submission

   3.1  Conformance Test Requirements

      3.1.1  Session Metric Profile Certification

      3.1.2  Reading Metric Profile Certification

      3.1.3  Annotation Metric Profile Certification

      3.1.4  Assessment Metric Profile Certification

      3.1.5  Assessment Item Metric Profile Certification

      3.1.6  Assignable Metric Profile Certification

      3.1.7  Media Metric Profile Certification

      3.1.8  Outcomes Metric Profile Certification

   3.2  Conformance Test Submission

About This Document

List of Contributors

Revision History

 

1  Introduction

IMS is establishing a well-defined conformance certification program for Caliper Analytics™ to ensure high levels of interoperability by testing and evaluating adherence to the Caliper standard framework.  The conformance certification suite tools test for the ability to support different levels of Caliper supported capabilities as it relates to the Sensor API™ and Caliper Event model.

The purpose of this document is to introduce the specific areas of conformance available to Caliper and to describe the procedures for achieving certification. Caliper is a standard created and governed by IMS Global Learning Consortium (IMS Global), a non-profit learning standards organization. This document is intended to be used, for example, by IMS  Members who wish to certify their systems as officially conforming to Caliper as supported by IMS. Ultimately, certification of systems will help ensure the common interchange of analytics data.

1.1  IMS Caliper Analytics

The purpose of the IMS Caliper project is to define a standard for learning analytics, basic metrics for measuring learner activity and an API for capturing and reporting this data. This work will enable learning environments to capture data from learning interactions and share it with other learning environments and consumers of learning analytics.  This work will build on existing IMS standards and culminate in a new IMS standard.

1.2  IMS Caliper Analytics Community

You can find more information about the Caliper Developer and Caliper Conformance at http://www.imsglobal.org/caliper/ Public feedback can be posted in the Caliper Forum.

 

2  Conformance Overview

2.1  Definition of Conformance

The definition of Caliper conformance is an evolving definition.  To encourage the early adoption of the standard we have started with a flexible  definition of what is required to be conformant as it applies to the Sensor API enabled Caliper Event model metric profile definitions.  As new versions of Caliper are released it is likely that additional events with their specific actions and properties will be required to attain conformance certification.

Also, in the future, the scope of the conformance test suite and certification may apply to related Caliper integration endpoints such as Learning Tools Interoperability ® (LTI®), the  event store, and persistence and other services.

2.2  Conformance Certifications

IMS is establishing well-defined conformance certification for Caliper to ensure high levels of analytics data interoperability.  The conformance certification process involves testing for the ability to support one or more of the  Caliper Event metric profiles, which are basically  the Information Models for Caliper. These metric profiles  act as the Sensor API parameterized payloads that constitute a fully qualified  event and event stream originating from the application leveraging Caliper to collect learning activity data.

2.3  Conformance Process

IMS Caliper Analytics certification demands features and capabilities beyond those which are strictly required by the specification. The specification is intentionally flexible in nature in order to allow it to be used for many purposes. This certification is particularly aimed at maximizing system interoperability  and meeting the minimal requirements of the Sensor API and Caliper Event Model metric profile protocols.

Conformance is measured by a test suite framework that is outlined below.

The application using the Caliper SensorAPI declares which Caliper Event metric profiles are integrated and should  be validated for certification.  At a minimum, the required actions and properties must be implemented in accordance with the Caliper JSON-LD event structure. Likewise, any additional optional actions or properties implemented will also be validated, evaluated and reported on for certification as well.  Optional elements however are not required and therefore not mandatory to attain conformance certification at this time.

2.4  The Conformance Certification Process

The Conformance certification process involves a user bringing an application that is intended to be Caliper compliant to the Conformance Suite. The user is then guided through a multi-step process:

  1. The user is first  presented with instructions and a form to collect details for the test, as listed in section 3.0.   The information is included in the submission. For each selected Metric Profile all "required" elements are needed to pass, with optional elements being validated for feedback only. Not selecting any Metric Profiles will mean that the Conformance Suite will only test for transport-level conformance.
  2. Once the test information is provided, the user can click on a start button that begins the test and assumes the target endpoint for the app sensors is set to the specified / noted conformance test service endpoint which appear in the instructions with error handling for any improper endpoint spec/access.
  3. Once the test finishes running,  the results are presented to the user, and the results of the conformance test run are allowed to be submitted for IMS review and certification.
  4. After IMS reviews your report and notifies you that the report is approved and adds you to the Official IMS  product directory, you can claim certification to IMS Caliper Analytics and display the certification badge on your web site and in your software.

The submission process allows you to report explanations of any anomalies you encountered during the testing process.

3  Conformance Test Submission

Information to be included along with the test results in a submission for Conformance:

Submitter’s Name & Email

The submitter’s name and email

Product Name & Version

The product name and version

Organization Name

Name of the organization that is an IMS Contributing/ Affiliate Member

 Metric Profiles Used

Session, Reading, Annotation, Media, Assignable, Assessment, Outcomes,  Engagement Scenario (eventually)

 

3.1  Conformance Test Requirements

Caliper Event Model metric profiles that are supported by the application are declared for conformance certification and define the scope of the test.  Currently, requirements defined for conformance are described below.  For each Caliper Event Model metric profile, the required events and object properties are listed in tables corresponding to the actions in the specific metric profile.  If a property is not mentioned specifically it is optional or does not apply.

LEGEND

R

Required

O

Optional

N/A

Does not apply

 

3.1.1 Session Metric Profile Certification

Logged In (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Software Application

O

R Session

R

 

O

O

O

O

Comments: Session.startedAtTime required

 

Logged Out (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Session

N/A

N/A

R

 

O

O

O

O

Comments: Session.endedAtTime required

 

Timed Out (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Session

N/A

N/A

R

 

O

O

O

O

Comments: Session.endedAtTime required

 

3.1.2 Reading Metric Profile Certification

Navigated To (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Digital Resource

R - Digital Resource

N/A

R

 

O

O

O

O

 

Viewed (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Digital Resource

N/A

N/A

R

 

O

O

O

O

 

Searched (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Digital Resource

N/A

R - Search Result

R

 

O

O

O

O

Comments:  Need to create SearchResult Entry

 

3.1.3 Annotation Metric Profile Certification

Bookmarked (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Digital Resource

N/A

R - Bookmark Annotation

R

 

O

O

O

O

 

Highlighted (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Digital Resource

N/A

R - Highlight Annotation

R

 

O

O

O

O

 

Shared (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Digital Resource

N/A

R - Shared Annotation

R

 

O

O

O

O

 

Tagged (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Digital Resource

N/A

R - Tag Annotation

R

 

O

O

O

O

 

3.1.4 Assessment Metric Profile Certification

Started (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

R - Attempt

R

 

O

O

 

O

O

 

Paused (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable Digital Resource

N/A

R

 

O

O

 

O

O

 

Restarted (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable Digital Resource

N/A

R

 

O

O

 

O

O

 

Submitted  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable Digital Resource

N/A

R

 

O

O

 

O

O

 

3.1.5 Assessment Item Metric Profile Certification

Started (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

R - Attempt

R

 

O

O

 

O

O

 

Completed  (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable Digital Resource

R - Response

R

 

O

O

 

O

O

 

Skipped  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

N/A

R

 

O

O

 

O

O

 

Reviewed  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R- Assignable Digital Resource

N/A

R

 

O

O

 

O

O

 

Viewed  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

N/A

R

 

O

O

 

O

O

 

3.1.6 Assignable Metric Profile Certification

Started (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

R - Attempt

R

 

O

O

 

O

O

 

Completed (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable Digital Resource

N/A

R

 

O

O

 

O

O

 

Activated (Optional)

Actor

Object

Target

Generated

eventTme

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

N/A

R

 

O

O

 

O

O

 

Deactivated (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

N/A

R

 

O

O

 

O

O

 

Showed (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

N/A

R

 

O

O

 

O

O

 

Hid (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Assignable Digital Resource

N/A

N/A

R

 

O

O

 

O

O

 

Reviewed (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable Digital Resource

N/A

R

 

O

O

 

O

O

 

Abandoned (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable Digital Resource

N/A

R

 

O

O

 

O

O

 

3.1.7 Media Metric Profile Certification

Opened Popout (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Closed Popout (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Entered Fullscreen (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Exited Full Screen (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Changed Size (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Changed Resolution (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Started (Required)

Actor

Object

Target

Generated

eventTIme

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

R - Media Location

N/A

R

 

O

O

 

O

O

 

Rewound  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

R - Media Location

N/A

R

 

O

O

 

O

O

 

Resumed (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

R - Media Location

N/A

R

 

O

O

 

O

O

 

Forwarded To  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

R - Media Location

N/A

R

 

O

O

 

O

O

 

Paused (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

R - Media Location

N/A

R

 

O

O

 

O

O

 

Jumped To (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

R - Media Location

N/A

R

 

O

O

 

O

O

 

Ended  (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Changed Speed (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Muted (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Unmuted (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Changed Volume (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Enabled Closed Captioning  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

 

O

O

 

O

O

 

Disabled Closed Captioning  (Optional)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Media Object

N/A

N/A

R

=

O

O

 

O

O

 

EVENTS REQUIRED:

PROPERTIES REQUIRED:

3.1.8 Outcomes Metric Profile Certification

Graded  (Required)

Actor

Object

Target

Generated

eventTime

 

edApp

Group

Membership

Federated

Session

R

R - Attempt

R - Assignable

N/A

R

 

O

O

 

O

O

3.2  Conformance Test Submission

To pass certification, you must take the following steps:

  • You must be an IMS Affiliate Member or an IMS Contributing Member
  •  You must pass the tests using the certification suite hosted on the IMS web site.
  • The tests must be completed by a designated representative of the member organization and you must agree that there is no misrepresentation or manipulation of the results in the submitted report.
  •  You must submit your report via this site to conformance@imsglobal.org.

The certification is for a specific declared and supported set of Caliper Event  metric profiles and therefore subsequent certification must be repeated  for each new release of the conforming application software or to conform with newer releases of  Caliper.

 

About this Document

Title:                              IMS Caliper Analytics Conformance and Certification Guide

Co-chairs:                              Chris Vento, Intellify Learning, Chris Millet, Penn State University

Version:                                  1.0
Version Date:                         15 October 2015
Release:                                  Final v1
Summary:                              This document introduces the different areas of conformance available to Caliper and describes the procedures for achieving certification.
Document Location:             http://www.imsglobal.org/caliper/

 

List of Contributors

The following individuals contributed to the development of this document:

Viktor Haag

D2L

Mary Millar

Elsevier

Prashant Nayak

Intellify Learning

Vineet Sinha

Cengage Learning

Chris Vento

Intellify Learning

Anthony Whyte

University of Michigan

 

Revision History

Version No.

Release Date

Comments

Final v1.0

15 October 2015

Final formal release of this specifcation.

 

 

IMS Global Learning Consortium, Inc. (“IMS Global”) is publishing the information contained in this document (“Specification”) for purposes of scientific, experimental, and scholarly collaboration only.

IMS Global makes no warranty or representation regarding the accuracy or completeness of the Specification.

This material is provided on an “As Is” and “As Available” basis.

The Specification is at all times subject to change and revision without notice. 

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

IMS Global would appreciate receiving your comments and suggestions.

Please contact IMS Global through our website at http://www.imsglobal.org.

Please refer to Document Name: IMS Caliper Analytics Conformance and Certification v1.0

Date: 15 October 2015