Sharebar?

IMS Caliper Analytics Best Practice Guide

IMS Logo

IMS Caliper Analytics™ Best Practice Guide

 

Version 1.0

Date Issued:            15 October  2015

Latest version:         http://www.imsglobal.org/caliper

IPR and Distribution Notices
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS’s procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Copyright © 2015 IMS Global Learning Consortium. All Rights Reserved.

Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/license.html.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.


© 2015 IMS Global Learning Consortium, Inc.
All Rights Reserved.
Trademark information: http://www.imsglobal.org/copyright.html
Document Name:  IMS Global Caliper Analytics Best Practice Guide – Final v1.0
Revision: 15 October 2015

 

Table of Contents

1     Introduction

   1.1      Terminology

   1.2      Metric Profiles

   1.3      Engagement Scenario

2     Quick Start Guide

   2.1      Download the reference Caliper Sensor implementation

   2.2      Build sensor project, run unit tests and deploy

   2.3      Install the sensor in your application

   2.4      View your events using an HTTP Request Inspector

   2.5      Store your events using a Caliper Reference Event Store Implementation

   2.6      Process for Implementing a Caliper Event Store

   2.7      Implement and send an event in your application

   2.8      LTI® Connectivity (If you are an LTI compliant Provider /tool)

3     Best Practice Guidelines

   3.1      How best to manage event size to send across the wire?

      3.1.1      Use the Sensor API DESCRIBE method with an object to qualify events with more educational context

      3.1.2      Use the JSON-LD @id, @type and @context to request the data

      3.1.3      Send all the events complete with detailed information

   3.2      LTI Best Practices

   3.3      Authentication suggestions

   3.4      SSL/HTTPS

   3.5      How do I transmit additional information that is not included in the Metric Profiles?

       3.5.1      You can add additional key/value pairs  to an Entity’s extensions property

   3.6      Envelope for the Metric Profile

About This Document

List of Contributors

Revision History

 

1  Introduction

As part of developing the Caliper Analytics™ standard and sensor reference architecture, the development team and implementers of the proof of concepts have come up with a best practice guide, including a quick start section.

The purpose of this document is to provide you with a detailed exploration of how Caliper works and the issues that arise so as to give you guidance on good practice, if not, best practice.

Please provide feedback based on your own experiences to help improve the quality of the recommendations included here.

 

1.1  Terminology

We will start with a bit of terminology to help make the descriptions that follow more precise.  The Caliper specification uses the following terms:

  • Caliper:  The name of the IMS Caliper Analytics Learning Measurement Framework which provides a standard for representing, capturing, and marshalling metrics generated by learning activity and targeted for consumption by any conforming analytics store or service.
  • Learning Activity: Any activity considered a component of a learning sequence presented in a digital learning environment (e.g. LTI® tool) typically denoted as a lesson.  Activities can be assignable and/or gradable as required.
  • Caliper SensorAPI™:  Defines and specifies interaction between Learning Application and Caliper.
  • Caliper Sensor:  Code/library that assists in the measurement of Caliper Events.  Deployed into the Learning/Digital Application.  Sensors are implemented in Java, Javascript, PHP, Python, Ruby and .NET..
  • Metric Profile:  The Information Model for Caliper – organized by Learning Activity Type.
  • Event Store: System or software service that stores event data emitted by the Caliper Sensors.

 

1.2 Metric Profiles

The Metric Profiles represent the Information Model and are organized by Learning Activity type.  When you begin implementing Caliper you should compare your application’s features with the metric profiles and implement the ones necessary to capture the user’s activities based on your features.  For example a Quizzing tool would want to implement base, session, assessment, assessment item and outcome metric profiles. An eReader would at a minimum implement base, session and reading metric profiles.

A complete list of the metric profiles and detailed information about each including properties and their data types, description, source and whether they are required are included in the Implementation Guide.

 

Profile

What can I use it for?

Base

Represents the base Caliper Event, Entity, and Actions used in other profiles

Session

Track users as they log in, log out, and also session time outs across multiple eduApps

Reading

Track engagement of users with reading material – navigation to/from, pages, etc.  Leverages ePUB, IMS Learning Information Services (LIS) and schema.org vocabularies

Annotation

Track learning activity related to bookmarks, highlights, sharing, notes, attachments, tagging and recommendations.   Leverages schema.org, LIS and OpenAnnotations vocabularies

Media

Track learning activity related to multi-media objects – Video, Audio and Image media types.  Leverages schema.org and LIS vocabularies

Assignable

Track learning activity on Assignable objects.  This profile allows you to make any object (e.g. reading, media, assessments, etc.) “assignable”.  Leverages schema.org and LIS vocabularies

Assessment

Track learning activity on Assessments.  Leverages LIS, Question and Test Interoperability® (QTI®), schema.org vocabularies

Outcomes

Track outcomes (Results) on assignable objects.  Leverages LIS Outcomes vocabulary

Engagement Scenario

Not a metric profile per se, but one common scenario / applied use case that applies a blended collection of metrics and context derived from other metric profile elements to explicitly collect engagement activity/usage associated data across a wide spectrum of learning activities

 

1.3  Engagement Scenario

This scenario contains a list of Events and corresponding actions from the current set of Caliper Metric Profiles that indicate minimum student engagement with Learning Activities.

  • In order to quantify engagement, three elements can be measured:  (1) the overall duration of a “session” comprising multiple events;(2) the duration of engagement with each individual activity; and, (3) the total number of (each) type of event in a session.
  • Each event has an “opening” action that indicates engagement.  E.g. “started”.  Other actions in the sequence (e.g. paused, resumed, completed) can also be included, including a “closing” action to indicate the end of a sequence.  However, to simply get an indicator of (one or many) students engagement with a specific Learning Activity, the opening action could be sufficient.
  • All attributes/objects (e.g. actor, action, object, startedAtTime, endedAtTime, duration, etc) of Events are implicitly part of the Engagement Profile.

 

Event

Action

Metric Profile

Comments

SessionEvent

loggedIn

Session

 

Navigation Event

navigatedTo

Mulitple

Common across all Learning Activity types

ViewedEvent

viewed

Reading

 

AnnotationEvent

All

Annotation

 

MediaEvent

started, [paused,resumed,etc}

Media

 

AssignableEvent

started [completed,submitted}

Assignable

 

AssessementEvent

started, [submitted]

Assessment

 

 

 

2  Quick Start Guide

In order to make it easier for programmers to get started implementing Caliper, we have created a getting started video. 

Getting Started Video (http://youtu.be/0AHKZpaPD2k)

There is also an exercise in Plunker written in Javascript that will walk a developer through implementing a caliper sensor.  The application is modeled as a simple Course delivery app, that delivers a syllabus, reading and quiz. Here is the link to the Sample Application.

2.1  Download the reference Caliper Sensor implementation.

The first step in implementing the Caliper sensor is to download the reference Caliper Sensor implementation that is applicable to your technology stack.  The links below point to the Github repository for each sensor.

● Java - Caliper Java Code

● Javascript - Caliper Javascript Code

● PHP - Caliper PHP Code

● .NET - Caliper .NET Code

● Python - Caliper Python Code

● Ruby - Caliper Ruby Code

2.2  Build sensor project, run unit tests and deploy

Build the project and run the test suites using the directions in the project’s README. Based on the type of sensor, you can use it on the server (all sensor types) or the client (e.g. Javascript sensor, note also for client-side sensors please note the authentication-related suggestions to maximize security.)   The mechanism for using the sensor within your code will also vary depending upon the sensor.  For example, you would include the Java sensor as a Maven dependency, the Ruby sensor as a Ruby Gem, the Python sensor as a Python Module, etc.  Each individual README provides information on this.

● Java – Web App (Server)

● Javascript – Node.js (Server), Mobile (Browser),Web (Browser)

● PHP – PHP Web App (Server)

● .NET – Windows (Server)

● Python – Django, Bottle (Server)

● Ruby –RoR, Sinatra (Server)

2.3  Install the sensor in your application

Install the sensor into your application following the instructions in the README. 

E.g., for Javascript: 

Install the caliper sensor into your application by placing the URL of the Caliper sensor Javascript library. 

  <script src="http://www.imsglobal.org/caliper/caliperSensor-1.0.0.js"></script>

2.4  View your events using an HTTP Request Inspector

You will need a place to capture events emitted by the Caliper Sensor.  The sensor sends events as a HTTP POST to a RESTful service.

For a very quick start, you can use a tool that allows you to inspect HTTP requests such as http://requestb.in/.  This will allow you access to the events being sent so you can inspect them and decouple the Event Store implementation from testing the sensor and event generation.

In addition, you can also send events to the Caliper Conformance Service located here:  Conformance Test Suite

2.5  Store your events using a Caliper Reference Event Store Implementation

The Caliper team has provided a reference EventStore implementation.  The EventStore is intended as a development/test/demo environment and is not intended for use as part of a production implementation of Caliper.  You can find it at:  Event Store.    

Follow the directions in the readme, to get the reference implementation of the Event Store up and running.

Each sensor needs to be updated to include the correct endpoint for the datastore that you have up and running.

2.6  Process for Implementing a Caliper Event Store

Additional best practices, implementation guidelines and possible alternative supporting event stores are currently underway and forthcoming as both Caliper, event store technologies and related support matures.  For now, utilizing the current Caliper Reference Event Store as a guide and understanding the JSON-LD based event stream so as to adapt any potential target persistence or event consuming service is a good starting point.   Further information and assistance will be provided by the IMS Caliper work group upon request. 

2.7  Implement and send an event in your application

Select sensor code bindings may include a sample application that provides implementation details for your chosen technology.  Currently, sample applications have been written in Java, Javascript and .NET.   Sample applications for the other sensors will be developed as time and interest permits.

Available Sample Applications

 Java Sample Application

Let’s take a look at the pieces that make up a Caliper event, using a navigation event as the stepwise example:

  1. Set the actor to be the current user (i.e. that was passed over in the LTI launch, or equivalent as part of the application context).
  2. Set the action for the event, in this case Caliper.Actions.ReadingActions.NAVIGATED_TO.
  3. Set the Object being interacted with by the Actor.
  4. Set the Target Object within the Event Object.
  5. Set the Educational Application (edApp) that is part of this Learning Context.
  6. Set the organization, which is the LIS course section that is part of this learning action.
  7. Set the navigatedFrom Object, which is the location from where the user navigated from.  This gives you the ability to create a graph of user actions.
  8. Set the eventTime for the navigation. 
  9. Add any other optional information that you want to include in the event message.
  10. Send the navigation event using the sensor.

2.8  LTI® Connectivity (If you are an LTI compliant Provider /tool)

For full detail please refer to the IMS Global Learning Tools Interoperability® Caliper - LTI - Implementation Guide.

 

Item

Description

Value type

Event Store URL

Endpoint for Caliper service messages

URL

Caliper API key

Key to identify system owner of message data

Variable length string. Minimally a 128 bit UUID (e.g. rfc 4122 like UUID), but other (and possibly more secure) keys are preferable

Caliper Profile URL

The endpoint for the Caliper Profile Service

URL

Federated Session Identifier

A URI that represents a federated LTI+Caliper set of launches/interactions (changes per session)

URI

Caliper Sensor Id

A key that represents a Caliper Sensor installed within a system (Tool Consumer or Provider)

Opaque String (Ideally a URI)

 

3  Best Practice Guidelines

3.1  How best to manage event size to send across the wire?

The Caliper payload is sent inside an "envelope", that allows for multiple events to be grouped together

{
  "sensor": "https://learning-app.some-university.edu/sensor",
  "sendTime": "2015-09-15T11:05:01.000Z",
  "data": [ {event1}, {event2}, …, {eventN}]
}

The size of the event "data" list is up to the Tool Consumer/Provider to negotiate.

3.1.1  Use the Sensor API DESCRIBE method with an object to qualify events with more educational context

The sensor includes a describe method  that can take either  a Caliper Entity or array of Caliper Entities.  So you can set elements of the learning graph  that are consistent throughout a learner session and stream of events without having to specify such contextual elements each time an event is sent  (i.e., the actor, agent, course section, etc.).  This provides a mechanism for reducing the size of serialized event payloads .                                    

3.1.2  Use the JSON-LD @id, @type and @context to request the data.

Since all Caliper Entities include a JSON-LD @id, @type and @context, you can specify larger and possibly more secondary event data entities independent from the event itself and therefore only the @id transmitted which can optionally be dereferenced as required  to retrieve all of the additional information provided by a given event related service.

3.1.3  Send all the events complete with detailed information.

 Another option is to send detailed information for all the event-related entities.  That said, be sure to specify null values for anything unspecified that is to sent across the wire.  There is optimizing compression etc during the serialization process to mitigate somewhat the event payload size.  In future revisions of Caliper there are plans to continue to streamline the event payloads via optimizations in the event model itself.

3.2  LTI Best Practices

In order to preserve the confidentiality of the shared secret, this data item is not passed in an LTI launch request.  Instead a Caliper Profile service should be declared in the tool consumer profile which may then be used by the tool provider to obtain the secret via a server-to-server web service request.

For full detail please refer to the IMS Global Learning Tools Interoperability® Caliper - LTI - Implementation Guide.

3.3  Authentication suggestions

For Caliper launches from an LTI Consumer/Provider the LTI security context is passed on. 

Without LTI integration, the Caliper sensor API endpoint is secured via similar LTI-OAuth signing and key/secret exchange .  As a best practice we recommend the use of OAuth. 

The calls to an Event Store should be secured by an API Key.  For the Javascript sensors the call to the Consumer Event Store should be performed server-to-server to ensure the integrity of the key.  The API key is stored  in the authorization header.

3.4  SSL/HTTPS

Caliper strongly recommends SSL/HTTPS for everything going over the wire.  Caliper Event Store endpoints should be secured by SSL to ensure secure communications.  The Sensors should accept and transmit through SSL/HTTPS.

3.5 How do I transmit additional information that is not included in the Metric Profiles?

You have two options if you require additional information to be sent over the wire that are not included by default in the metric profile.                         

3.5.1  You can add additional key/value pairs  to an Entity’s extensions property.

The extensions property is a dictionary that can contain a list of properties.  So for example you can add: 

"extensions": {
  "com.vendor.example.myNeededProperty": "its value"
}

Extensions should use a technique to ensure uniqueness across organizations (for example, a reverse-domain formatted string).  Extend the Caliper JSON-LD Event schema 

Another option would be to extend the appropriate Caliper Metric Profile JSON-LD Event schema with additional properties.

3.6  Envelope for the Metric Profile

The method envelope standard for all Caliper Event sequences includes:

{
  "sensor": "http://learning-app.some-university.edu/sensor",
  "sendTime": "2015-09-15T11:05:01.000Z",
  "data": [ {event1}, {event2}, …, {eventN}]
}

 

About This Document

 

Title:                                                     IMS Global

Editor:                                                  Caliper Workgroup

Co-chairs:                                            Chris Vento, Chris Millet

Version:                                                v1.0

Version Date:                                       15 October 2015

Release:                                                Final v1.0

Status:                                                  Final Release

Purpose:                                               This document provides best practice for using Caliper Analytics.

Document Location:                            http://www.imsglobal.org/caliper

 

List of Contributors

The following individuals contributed to the development of this document:

Viktor Haag

D2L

Mary Millar

Elsevier

Prashant Nayak

Intellify Learning

Chris Vento

Intellify Learning

Anthony Whyte

University of Michigan

 

 

Revision History

 

Version No.

Release Date

Comments

Final v1.0

15 October 2015

Final formal release of this specifcation.

 

 

 

 

 

 

 

 

IMS Global Learning Consortium, Inc. (“IMS Global”) is publishing the information contained in this document (“Specification”) for purposes of scientific, experimental, and scholarly collaboration only.

IMS Global makes no warranty or representation regarding the accuracy or completeness of the Specification.

This material is provided on an “As Is” and “As Available” basis.

The Specification is at all times subject to change and revision without notice. 

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

IMS Global would appreciate receiving your comments and suggestions.

Please contact IMS Global through our website at http://www.imsglobal.org.

Please refer to Document Name: IMS Global Caliper Analytics Best Practice Guide v1.0 Final

Date: 15 October 2015