1EdTech Caliper Analytics™ Conformance and Certification Guide
Version 1.0 Final
Date Issued: 15 October 2015
Latest version: http://www.imsglobal.org/caliper
IPR and Distribution Notices
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.
1EdTech takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on 1EdTech’s procedures with respect to rights in 1EdTech specifications can be found at the 1EdTech Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.
Copyright © 2015 1EdTech Consortium. All Rights Reserved.
Use of this specification to develop products or services is governed by the license with 1EdTech found on the 1EdTech website: http://www.imsglobal.org/license.html.
Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.
The limited permissions granted above are perpetual and will not be revoked by 1EdTech or its successors or assigns.
THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.
© 2015 1EdTech Consortium, Inc.
All Rights Reserved.
Trademark information: http://www.imsglobal.org/copyright.html
Document Name: 1EdTech Caliper Analytics Conformance and Certification Guide – Final v1.0
Revision: 15 October 2015
Table of Contents
1.2 1EdTech Caliper Analytics Community
2.2 Conformance Certifications
2.4 The Conformance Certification Process
3.1 Conformance Test Requirements
3.1.1 Session Metric Profile Certification
3.1.2 Reading Metric Profile Certification
3.1.3 Annotation Metric Profile Certification
3.1.4 Assessment Metric Profile Certification
3.1.5 Assessment Item Metric Profile Certification
3.1.6 Assignable Metric Profile Certification
3.1.7 Media Metric Profile Certification
3.1.8 Outcomes Metric Profile Certification
3.2 Conformance Test Submission
1 Introduction
1EdTech is establishing a well-defined conformance certification program for Caliper Analytics™ to ensure high levels of interoperability by testing and evaluating adherence to the Caliper standard framework. The conformance certification suite tools test for the ability to support different levels of Caliper supported capabilities as it relates to the Sensor API™ and Caliper Event model.
The purpose of this document is to introduce the specific areas of conformance available to Caliper and to describe the procedures for achieving certification. Caliper is a standard created and governed by 1EdTech Consortium (1EdTech), a non-profit learning standards organization. This document is intended to be used, for example, by 1EdTech Members who wish to certify their systems as officially conforming to Caliper as supported by 1EdTech. Ultimately, certification of systems will help ensure the common interchange of analytics data.
1.1 1EdTech Caliper Analytics
The purpose of the 1EdTech Caliper project is to define a standard for learning analytics, basic metrics for measuring learner activity and an API for capturing and reporting this data. This work will enable learning environments to capture data from learning interactions and share it with other learning environments and consumers of learning analytics. This work will build on existing 1EdTech standards and culminate in a new 1EdTech standard.
1.2 1EdTech Caliper Analytics Community
You can find more information about the Caliper Developer and Caliper Conformance at http://www.imsglobal.org/caliper/ Public feedback can be posted in the Caliper Forum.
2 Conformance Overview
2.1 Definition of Conformance
The definition of Caliper conformance is an evolving definition. To encourage the early adoption of the standard we have started with a flexible definition of what is required to be conformant as it applies to the Sensor API enabled Caliper Event model metric profile definitions. As new versions of Caliper are released it is likely that additional events with their specific actions and properties will be required to attain conformance certification.
Also, in the future, the scope of the conformance test suite and certification may apply to related Caliper integration endpoints such as Learning Tools Interoperability ® (LTI®), the event store, and persistence and other services.
2.2 Conformance Certifications
1EdTech is establishing well-defined conformance certification for Caliper to ensure high levels of analytics data interoperability. The conformance certification process involves testing for the ability to support one or more of the Caliper Event metric profiles, which are basically the Information Models for Caliper. These metric profiles act as the Sensor API parameterized payloads that constitute a fully qualified event and event stream originating from the application leveraging Caliper to collect learning activity data.
2.3 Conformance Process
1EdTech Caliper Analytics certification demands features and capabilities beyond those which are strictly required by the specification. The specification is intentionally flexible in nature in order to allow it to be used for many purposes. This certification is particularly aimed at maximizing system interoperability and meeting the minimal requirements of the Sensor API and Caliper Event Model metric profile protocols.
Conformance is measured by a test suite framework that is outlined below.
The application using the Caliper SensorAPI declares which Caliper Event metric profiles are integrated and should be validated for certification. At a minimum, the required actions and properties must be implemented in accordance with the Caliper JSON-LD event structure. Likewise, any additional optional actions or properties implemented will also be validated, evaluated and reported on for certification as well. Optional elements however are not required and therefore not mandatory to attain conformance certification at this time.
2.4 The Conformance Certification Process
The Conformance certification process involves a user bringing an application that is intended to be Caliper compliant to the Conformance Suite. The user is then guided through a multi-step process:
- The user is first presented with instructions and a form to collect details for the test, as listed in section 3.0. The information is included in the submission. For each selected Metric Profile all "required" elements are needed to pass, with optional elements being validated for feedback only. Not selecting any Metric Profiles will mean that the Conformance Suite will only test for transport-level conformance.
- Once the test information is provided, the user can click on a start button that begins the test and assumes the target endpoint for the app sensors is set to the specified / noted conformance test service endpoint which appear in the instructions with error handling for any improper endpoint spec/access.
- Once the test finishes running, the results are presented to the user, and the results of the conformance test run are allowed to be submitted for 1EdTech review and certification.
- After 1EdTech reviews your report and notifies you that the report is approved and adds you to the Official 1EdTech product directory, you can claim certification to 1EdTech Caliper Analytics and display the certification badge on your web site and in your software.
The submission process allows you to report explanations of any anomalies you encountered during the testing process.
3 Conformance Test Submission
Information to be included along with the test results in a submission for Conformance:
|
Submitter’s Name & Email |
The submitter’s name and email |
|
Product Name & Version |
The product name and version |
|
Organization Name |
Name of the organization that is an 1EdTech Contributing/ Affiliate Member |
|
Metric Profiles Used |
Session, Reading, Annotation, Media, Assignable, Assessment, Outcomes, Engagement Scenario (eventually) |
3.1 Conformance Test Requirements
Caliper Event Model metric profiles that are supported by the application are declared for conformance certification and define the scope of the test. Currently, requirements defined for conformance are described below. For each Caliper Event Model metric profile, the required events and object properties are listed in tables corresponding to the actions in the specific metric profile. If a property is not mentioned specifically it is optional or does not apply.
|
LEGEND |
|
|
R |
Required |
|
O |
Optional |
|
N/A |
Does not apply |
3.1.1 Session Metric Profile Certification
|
Logged In (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Software Application |
O |
R Session |
R |
|
O |
O |
O |
O |
Comments: Session.startedAtTime required
|
Logged Out (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Session |
N/A |
N/A |
R |
|
O |
O |
O |
O |
Comments: Session.endedAtTime required
|
Timed Out (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Session |
N/A |
N/A |
R |
|
O |
O |
O |
O |
Comments: Session.endedAtTime required
3.1.2 Reading Metric Profile Certification
|
Navigated To (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Digital Resource |
R - Digital Resource |
N/A |
R |
|
O |
O |
O |
O |
|
Viewed (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Digital Resource |
N/A |
N/A |
R |
|
O |
O |
O |
O |
|
Searched (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Digital Resource |
N/A |
R - Search Result |
R |
|
O |
O |
O |
O |
Comments: Need to create SearchResult Entry
3.1.3 Annotation Metric Profile Certification
|
Bookmarked (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Digital Resource |
N/A |
R - Bookmark Annotation |
R |
|
O |
O |
O |
O |
|
Highlighted (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Digital Resource |
N/A |
R - Highlight Annotation |
R |
|
O |
O |
O |
O |
|
Shared (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Digital Resource |
N/A |
R - Shared Annotation |
R |
|
O |
O |
O |
O |
|
Tagged (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Digital Resource |
N/A |
R - Tag Annotation |
R |
|
O |
O |
O |
O |
3.1.4 Assessment Metric Profile Certification
|
Started (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
R - Attempt |
R |
|
O |
O
|
O |
O |
|
Paused (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable Digital Resource |
N/A |
R |
|
O |
O
|
O |
O |
|
Restarted (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable Digital Resource |
N/A |
R |
|
O |
O
|
O |
O |
|
Submitted (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable Digital Resource |
N/A |
R |
|
O |
O
|
O |
O |
3.1.5 Assessment Item Metric Profile Certification
|
Started (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
R - Attempt |
R |
|
O |
O
|
O |
O |
|
Completed (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable Digital Resource |
R - Response |
R |
|
O |
O
|
O |
O |
|
Skipped (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Reviewed (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R- Assignable Digital Resource |
N/A |
R |
|
O |
O
|
O |
O |
|
Viewed (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
3.1.6 Assignable Metric Profile Certification
|
Started (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
R - Attempt |
R |
|
O |
O
|
O |
O |
|
Completed (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable Digital Resource |
N/A |
R |
|
O |
O
|
O |
O |
|
Activated (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTme |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Deactivated (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Showed (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Hid (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Assignable Digital Resource |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Reviewed (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable Digital Resource |
N/A |
R |
|
O |
O
|
O |
O |
|
Abandoned (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable Digital Resource |
N/A |
R |
|
O |
O
|
O |
O |
3.1.7 Media Metric Profile Certification
|
Opened Popout (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Closed Popout (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Entered Fullscreen (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Exited Full Screen (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Changed Size (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Changed Resolution (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Started (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTIme |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
R - Media Location |
N/A |
R |
|
O |
O
|
O |
O |
|
Rewound (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
R - Media Location |
N/A |
R |
|
O |
O
|
O |
O |
|
Resumed (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
R - Media Location |
N/A |
R |
|
O |
O
|
O |
O |
|
Forwarded To (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
R - Media Location |
N/A |
R |
|
O |
O
|
O |
O |
|
Paused (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
R - Media Location |
N/A |
R |
|
O |
O
|
O |
O |
|
Jumped To (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
R - Media Location |
N/A |
R |
|
O |
O
|
O |
O |
|
Ended (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Changed Speed (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Muted (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Unmuted (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Changed Volume (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Enabled Closed Captioning (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
|
O |
O
|
O |
O |
|
Disabled Closed Captioning (Optional) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Media Object |
N/A |
N/A |
R |
= |
O |
O
|
O |
O |
EVENTS REQUIRED:
PROPERTIES REQUIRED:
3.1.8 Outcomes Metric Profile Certification
|
Graded (Required) |
|||||||||
|
Actor |
Object |
Target |
Generated |
eventTime |
|
edApp |
Group |
Membership |
Federated Session |
|
R |
R - Attempt |
R - Assignable |
N/A |
R |
|
O |
O
|
O |
O |
3.2 Conformance Test Submission
To pass certification, you must take the following steps:
- You must be an 1EdTech Affiliate Member or an 1EdTech Contributing Member
- You must pass the tests using the certification suite hosted on the 1EdTech web site.
- The tests must be completed by a designated representative of the member organization and you must agree that there is no misrepresentation or manipulation of the results in the submitted report.
- You must submit your report via this site to conformance@imsglobal.org.
The certification is for a specific declared and supported set of Caliper Event metric profiles and therefore subsequent certification must be repeated for each new release of the conforming application software or to conform with newer releases of Caliper.
About this Document
Title: 1EdTech Caliper Analytics Conformance and Certification Guide
Co-chairs: Chris Vento, Intellify Learning, Chris Millet, Penn State University
Version: 1.0
Version Date: 15 October 2015
Release: Final v1
Summary: This document introduces the different areas of conformance available to Caliper and describes the procedures for achieving certification.
Document Location: http://www.imsglobal.org/caliper/
List of Contributors
The following individuals contributed to the development of this document:
|
Viktor Haag |
D2L |
|
Mary Millar |
Elsevier |
|
Prashant Nayak |
Intellify Learning |
|
Vineet Sinha |
Cengage Learning |
|
Chris Vento |
Intellify Learning |
|
Anthony Whyte |
University of Michigan |
Revision History
|
Version No. |
Release Date |
Comments |
|
Final v1.0 |
15 October 2015 |
Final formal release of this specifcation. |
1EdTech Consortium, Inc. (“1EdTech”) is publishing the information contained in this document (“Specification”) for purposes of scientific, experimental, and scholarly collaboration only.
1EdTech makes no warranty or representation regarding the accuracy or completeness of the Specification.
This material is provided on an “As Is” and “As Available” basis.
The Specification is at all times subject to change and revision without notice.
It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.
1EdTech would appreciate receiving your comments and suggestions.
Please contact 1EdTech through our website at http://www.imsglobal.org.
Please refer to Document Name: 1EdTech Caliper Analytics Conformance and Certification v1.0
Date: 15 October 2015
