Caliper Analytics 1.2 Implementation Guide

Caliper Implementation Guide

IMS Final Release
Version 1.2
IMS Final Release
Date Issued: 27 March 2020
Status: This document is made available for adoption by the public community at large.
This version: https://www.imsglobal.org/spec/caliper/v1p2/impl/

IPR and Distribution Notice

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS's procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Use of this specification to develop products or services is governed by the license with IMS found on the IMS website: http://www.imsglobal.org/speclicense.html.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

Public contributions, comments and questions can be posted here: http://www.imsglobal.org/forums/ims-glc-public-forums-and-resources.

© 2020 IMS Global Learning Consortium, Inc. All Rights Reserved.

Trademark information: http://www.imsglobal.org/copyright.html

Abstract

IMS Caliper Analytics® is a technical specification that describes a structured set of vocabulary that assists institutions in collecting learning and usage data from digital resources and learning tools. This data can be used to present information to students, instructors, advisers, and administrators in order to drive effective decision making and promote learner success.

Caliper can help to answer various questions about learner activity, a few examples questions that Caliper might help answer.

  • How often is a learner logging in to view their digital resources?
  • Are learners interacting with a forum post?
  • How often is a learner consuming course readings or videos?
  • What activities is a learner interacting with while taking an assessment?
  • Are learners annotating certain digital resources?

With the answers to these questions in tow, decision makers can drive analysis that answers critical questions such as:

  • Is there a performance difference between learners who interact with a certain digital resource versus learners who do not?

The sky is the limit with the ways to use Caliper data to measurably improve the learning ecosystem.

1. Introduction

1.1 What is this document?

This document is an informative part of the Caliper Analytics © 1.2 document set. As such, it may be revised without the specification version being incremented. Please refer to the revision history below for more information about revisions to this document.

As an informative resource it does not include any normative requirements. Occurrences in this document of terms such as MAY, MUST, MUST NOT, SHOULD or RECOMMENDED have no impact on the conformance criteria for implementors of this specification.

The primary goal of this document is to lead you to successful implementation of the Caliper Analytics® v1.2 specification. More than that, this guide helps you along the way to achieving conformance certification.

1.2 How to use this document

This document is intended as a starting point for those looking to implement the Caliper Analytics® standard in their educational software ecosystem.

This guide can be used to get a fundamental understanding of the Caliper messaging structure, review specific code examples of Caliper events, and as a central hub containing links to conformance requirements and other important resources. The document can also be used as a reference for collated best practices on how to use Caliper in their digital ecosystem and guidance on using Caliper in collaboration with other IMS specifications.

This document is also meant to assist the reader in learning how to use the main Caliper specification to look up specific items during implementation.

2. Terminology

Learning this vocabulary in the context of the Caliper specification will be very helpful when using this document. Caliper Analytics describes events using triples, a semantic structure that consists of the combination of an action being undertaken by an actor to an object. A good way to think of it as someone (actor) did something (action) to someone/something (object)

Here are a few useful definitions for terms used throughout this document. Full Caliper terminology list is available in the Terminology section of the Caliper Spec.

Action
Part of an Event that describes the what.
Actor
Part of an Event that describes the who.
Entity
Part of an Event that describes the to whom.
Agent
Part of an Event that represents a generic form of an Entity and describes the to whom when a more specific Entity is not available.
Event
A collection of an Actor, an Action, and an Object.
Envelope
A JSON payload that can contain one or many Caliper events.
Object
Part of an Event that describes the to whom or to what.
Metric Profile
Groupings of Caliper vocabulary that model a learning activity or a supporting activity. Each Metric Profile provides a domain-specific set of terms and concepts that application designers and developers can draw upon to describe common user interactions in a consistent manner. Metric Profiles also serve as the unit of certification for the Caliper Analytics® specification.
Sensor
Software deployed within a learning application that capture and transmit Caliper data to a target endpoint.

Here is a short explanation of how Caliper works using this vocabulary. If you can understand this you're well on your way to understanding Caliper!

When a Caliper-observable interaction, as defined by a Metric Profile, is detected by a Sensor, an Event is created which consists of an Actor, an Object, and an Action. (The Actor did the Action to the Object) To emit an Event to another system via an API call, it is described as a JSON object which is wrapped in an Envelope with a little metadata about the Sensor. The JSON is then sent via a secured HTTP call with an authentication token in an HTTP header.

3. Conformance Certification

3.1 Conformance Statements

This document is an informative resource in the Document Set of the Caliper 1.2 specification [CALIPER-12]. As such, it does not include any normative requirements. Occurrences in this document of terms such as MAY, MUST, MUST NOT, SHOULD or RECOMMENDED have no impact on the conformance criteria for implementors of this specification.

3.2 What is Conformance Certification?

IMS offers a process for testing the conformance of products using the IMS certification test suite. Certification designates passing a set of tests that verify the standard has been implemented correctly and guarantees a product’s interoperability across hundreds of other certified products. The Caliper Analytics Conformance Certification Guide provides details about the testing process, requirements, and how to get started.

3.3 What are the benefits of Conformance Certification?

Conformance certification offers more than claims of "compliance,". The only way IMS can guarantee interoperability is by obtaining certification for the latest version of the standard. Only products listed in the official IMS Certified Product Directory can claim conformance certification. IMS certification provides the assurance that a solution will integrate securely and seamlessly into an institution's digital learning ecosystem.

3.4 How does my product get certified?

Certification for Caliper Analytics involves certifying against one or more Metric Profiles. This means passing the appropriate tests for the profile or profiles that you are implementing using the IMS Global suite of Caliper Analytics certification tools. For more information on the conformance and certification process, please refer to the The Caliper Analytics Conformance Certification Guide.

4. What's new in Caliper 1.2?

In addition to improvements and additions to vocabulary throughout the Caliper 1.2 information model, Caliper introduces 6 new metric profiles built based on market need and member feedback. Please follow the associated links below to learn more about them. Those profiles are:

Additionally, the Tool Use Profile has been improved to account for the collection of measures that track student progress via the collection of aggregate measures.

5. How to create a Caliper Event

5.1 Events

Link to full Event documentation

An Event is the combination of an Actor, an Action, and an Object. In a sentence, an Event describes what an Actor did (Action) to an Object. Each of those will be described in the following sections.

It's important to have a clear understanding of Caliper Events since they describe the semantics of what's happening in the system.

5.1.1 Event Types

Caliper Events each specify which type of event they are. Events of the same type have several things in common, like required and optional properties. Some event types include:

AnnotationEvent, AssignableEvent, AssessmentEvent, AssessmentItemEvent, ForumEvent, MediaEvent, MessageEvent, NavigationEvent, GradeEvent, SessionEvent, ToolUseEvent, ThreadEvent, ViewEvent

A full list of event types is available in the Caliper Spec (See "Subtype" subsection of Event).

5.1.2 Choosing the appropriate Event

To choose the best Event for your situation you can scan the list of event types and see which one seems to match best, or you can also start by looking at a Metric Profile and see what events are described there that might meet your needs already.

If there is no existing Event that meets your needs you can use a "generic" event from the General Profile. These events allow any actor, action, and object you need and aren't constrained like the specific events from other profiles. This should provide you with the flexibility you need when your use case isn't already met by existing profiles.

5.1.3 Event Required Properties

Each Event Type can have different required properties. To review what's required, find the specific event you're creating in the Caliper spec document and review the table there. For example, you can look at the Annotation Event's properties table.

All property tables will look like the following table. This table shows the base Event property requirements that all events share.

Property Type Description Disposition
id UUID A version 4 UUID. The UUID MUST be expressed as a URN using the form urn:uuid:. Required
type Term A string value for the Event Sub Type, or for a generic Event set the type to the string value Event. Required
actor Agent or IRI The Agent who initiated the Event. This must be either:
  • An Entity of the type Agent or one of its subtypes (e.g., Person, SoftwareApplication, etc.).
  • A unique IRI that represents an Entity as described above. Ideally this should refer to an Entity previously defined in the payload containing the Event or in an eventstore that contains the Event.
Required
action Term The action or predicate that binds the actor or subject to the object. The action range is limited to the set of actions described in this specification and may be further constrained by the chosen Event type. Only one action Term may be specified per Event. Required
object Entity or IRI The Entity that comprises the object of the interaction. The object value MUST be expressed either as an object or as a string corresponding to the object's IRI. Required
eventTime DateTime An ISO 8601 date and time value expressed with millisecond precision that indicates when the Event occurred. The value MUST be expressed using the format YYYY-MM-DDTHH:mm:ss.SSSZ set to UTC with no offset specified. Required
profile Profile Term A string value corresponding to the Profile Term value defined for the Profile that governs the rules of interpretation for this Event. For a generic Event set the profile property value to the string term GeneralProfile. Optional

5.1.4 Additional properties on Events

While the base information required by most Events is represented in the above example, each Event type could require more properties, and allow for more properties and context as desired by the implementors. To understand what information is required and possible you should consult each Event Type's documentation for the activities you're trying to implement.

5.1.5 Entity or IRI

Properties with the type "Entity or IRI" may be represented as a Caliper entity or an IRI string. An IRI may be used to refer to an entity that has been defined earlier, either in the same event, in another event in the same payload, or even in another event in the same eventstore. The IRI may be a URL, but it's not required. It could be a URN instead, using the "urn" scheme rather than the "http" or "https" schemes commonly used with URLs. Often, using a blank node scheme (represented by a single underscore character, "_") offers flexibility to specify an IRI without the burden of requiring it to resolve to a resource like a URL or to follow specific syntax, like a URN.

5.1.6 Event JSON stub

This is an example of what an Event's JSON looks like without the actor/action/object. A full example will be given below.


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "id": "urn:uuid:3a648e68-f00d-4c08-aa59-8738e1884f2c",
  "type": "ViewEvent",
  "profile": "ReadingProfile",
  "eventTime": "2020-01-15T10:15:00.000Z",
  "actor": {},
  "action": "",
  "object": {}
}

5.2 Actors

In a Caliper Event, the Actor performs the action related to a learning activity. The value of an Event's actor attribute must be an Agent or one of its subtypes. While often the Actor is a Person, it could also be another Entity type, such as an Organization or SoftwareApplication.

Each Event type further limits what Agents can be used as the value of the actor property; for example, the actor for a QuestionnaireEvent must be a Person, while the actor for a GradeEvent could be either a Person or a SoftwareApplication (such as an autograder).

The generic Event allows any Agent declared in the specification.

5.2.1 Actor JSON


{
  "id": "https://example.edu/users/554433",
  "type": "Person"
}

5.3 Action

Link to full Action documentation

An Action connects an Actor with an Object, helping to describe what learning activity has taken place. Examples include Bookmarked, Launched, OptedIn, and Skipped.

A specific Event Type specifies what actions are valid for that type. For example, you can look at the Annotation Event's properties table to see that the Action must be one of: Bookmarked, Highlighted, Shared, or Tagged.

The generic Event allows any Action declared in the specification.

5.3.1 Action JSON


"Viewed"

5.4 Object

The Object of an Event must be an Entity or one of its subtypes.

A specific Event Type specifies what types of Entities are valid for that type. For example, you can look at the Annotation Event's properties table to see that the Object must be a DigitalResource.

The generic Event allows any Entity declared in the specification.

5.4.1 Object JSON


{
  "id": "https://example.edu/terms/202001/courses/7/sections/1/readings/1",
  "type": "DigitalResource",
  "name": "Chapter 1 reading"
}

5.5 Caliper Event JSON Examples

There are many examples of events to help you learn how to construct them. Below is an example combining the above documentation.

Other sources for example Event JSON:

  • The main Caliper spec document has many examples for each Event Type. For example see the AnnotationEvent and scroll past the requirements table to the examples below.
  • The public Github repository has many examples for testing purposes: Caliper 1.2 JSON Examples
  • If you're an IMS member and have access to the Caliper certification tool, you can access those same examples in the Caliper Playground's "Manual Testing" area.

5.5.1 Complete JSON

To bring the above examples together, an Event would look something like this:


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "id": "urn:uuid:a2f41f9c-d57d-4400-b3fe-716b9026334e",
  "type": "ViewEvent",
  "profile": "ReadingProfile",
  "eventTime": "2020-01-15T10:16:00.000Z",
  "actor": {
    "id": "https://example.edu/users/554433",
    "type": "Person"
  },
  "action": "Viewed",
  "object": {
    "id": "https://example.edu/terms/202001/courses/7/sections/1/readings/1",
    "type": "DigitalResource",
    "name": "Chapter 1 reading"
  }
}

This Event can be read as "This Person Viewed this DigitalResource at this eventTime".

5.5.2 Smaller, ID-Only JSON

As explained in the Entity or IRI section above, Caliper allows the use just the ID string instead of the whole JSON object if the size of your events needs to be smaller.

This ID-only example is equivalent to the full example above:


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "id": "urn:uuid:a2f41f9c-d57d-4400-b3fe-716b9026334e",
  "type": "ForumEvent",
  "profile": "ForumProfile",
  "eventTime": "2020-01-15T10:16:00.000Z",
  "actor": "https://example.edu/users/554433",
  "action": "Subscribed",
  "object": "https://example.edu/terms/202001/courses/7/sections/1/forums/1"
}

6. Metric Profiles

Each Caliper Metric Profile is a domain-specific set of terms and concepts to describe common education learning activities. Their purpose is to help ensure users of Caliper have consistent syntax and semantics when describing these events. Some examples of what profiles attempt to describe are Annotating a reading, playing a video, taking a test, or grading an assignment submission.

A great way to think of a Metric Profile is what questions it can help answer. For example the Reading Profile could help Instructors and researchers answer questions such as:

  • Who is consuming the content?

  • What materials are being accessed?

  • When are the resources accessed?

  • How often is the content viewed?

  • What paths are taken to reach the content?

    These examples are given in each profile section in the Caliper specification document.

6.1 Profile Requirements

Each profile is a collection of one or more Caliper events that together help describe a set of related activities. Each Event type included in a profile places constraints on the entities and actions that can be used. Vocabulary restrictions are outlined in each profile description under the following headings:

  • supported events
  • supported actors
  • supported actions
  • supported objects
  • supported target entities
  • supported generated entities
  • other requirements

For example, the Forum Profile models a set of activities associated with online discussions involving instructors and learners. The profile currently includes a ForumEvent, MessageEvent, NavigationEvent, ThreadEvent and ViewEvent. You can see those events clearly defined in the table in the Forum Profile documentation.

The sequence of events from a Forum Profile implementation might involve a learner navigating to a forum, subscribing to it, viewing a thread, posting a message in reply to an earlier post and then marking the message as read.

6.2 List of Profiles

Profile Description Events Quick-Reference
General Profile provides a generic Event for describing learning or supporting activities that have yet to be modeled by Caliper. Event
Annotation Profile models activities related to the annotation of a DigitalResource AnnotationEvent
Assessment Profile models assessment-related activities including interactions with individual assessment items. AssessmentEvent, AssessmentItemEvent, NavigationEvent, ViewEvent
Assignable Profile models activities associated with the assignment of digital content to a learner for completion according to specific criteria. AssignableEvent, NavigationEvent, ViewEvent
Feedback Profile models providing feedback and comments on Entities. FeedbackEvent
Forum Profile models learners and others participating in online forum communities. ForumEvent, MessageEvent, ThreadEvent, NavigationEvent, ViewEvent
Grading Profile models grading activities performed by an Agent or a SoftwareApplication GradeEvent, ViewEvent
Media Profile models interactions between learners and rich content such as audio, images and video. MediaEvent, NavigationEvent, ViewEvent
Reading Profile models activities associated with navigating to and viewing digital textual content. NavigationEvent, ViewEvent
Resource Management Profile models a person managing a digital resource ResourceManagementEvent
Search Profile models an actor, typically a learner, querying a resource for information SearchEvent
Session Profile models the creation and subsequent termination of a user session SessionEvent
Survey Profile provides a vocabulary for describing events associated with a respondent's participation in online surveys SurveyInvitationEvent, SurveyEvent, QuestionnaireEvent, QuestionnaireItemEvent, NavigationEvent, ViewEvent
Tool launch Profile captures the utilization of learning tools from a centralized location, such as an LTI Tool Platform (often an LMS). As such, it is distinct from the Caliper Tool Use Profile, also defined in this specification, which captures tool usage as reported by the individual tools themselves (i.e. decentralized capture) ToolLaunchEvent
Tool Use Profile models an intended interaction between a user and a tool. ToolUseEvent

6.3 Profiles are used for certification

Profiles also serve as the unit of certification for Caliper. The Caliper Conformance and Certification Guide describes the certification process and requirements. Each profile has a section that, in addition to clearly displaying what the certification requirements are, gives an excellent overview of each Event Type and the supported Actions.

6.4 Creating new profiles

The official Caliper specification will never be able to describe every Event or activity needed for all institutions, districts, and vendors. The Caliper workgroup and Product Steering committee are charged with working with all parties to help create new profiles as needed to help the community continue to move forward. Caliper has a profile extension mechanism for adding new profiles without having to release a whole new version of Caliper.

7. Sending Events to an endpoint

7.1 Envelopes are required

Caliper Event and Entity data MUST be transmitted inside a Caliper Envelope, a purpose-built JSON data structure that includes metadata about the emitting Sensor and the data payload.

A Caliper envelope MUST contain the sensor, sendTime, dataVersion and data properties. Each property MUST be referenced only once. No custom properties are permitted.

The data element can contain multiple Events and Entities. It is good practice to batch related Events if your Sensor is structured to do so.

7.1.1 Base Envelope JSON


{
  "sensor": "https://example.edu/sensors/1",
  "sendTime": "2020-01-15T11:05:01.000Z",
  "dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "data": [ {event1}, {event2}, {eventN} ]
}

7.1.2 Example Envelope with an Event JSON


{
  "sensor": "https://example.edu/sensors/1",
  "sendTime": "2020-01-15T11:05:01.000Z",
  "dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "data": [
      {
        "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
        "id": "urn:uuid:7e10e4f3-a0d8-4430-95bd-783ffae4d916",
        "type": "ToolUseEvent",
        "profile": "ToolUseProfile",
        "eventTime": "2020-01-15T10:15:00.000Z",
        "actor": {
          "id": "https://example.edu/users/554433",
          "type": "Person"
        },
        "action": "Used",
        "object": {
          "id": "https://example.edu",
          "type": "SoftwareApplication"
        }
      }
  ]
}

7.1.3 Mixed payload Envelope JSON

In this example, the Person and SoftwareApplication are part of the data array and referenced from the events for reuse.


{
  "sensor": "https://example.edu/sensors/1",
  "sendTime": "2020-01-15T11:05:01.000Z",
  "dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "data": [
   {
    "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
    "id": "https://example.edu/users/554433",
    "type": "Person",
    "dateCreated": "2018-08-01T06:00:00.000Z",
    "dateModified": "2018-09-02T11:30:00.000Z"
  },
  {
    "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
    "id": "https://example.edu",
    "type": "SoftwareApplication",
    "version": "v2"
  },
  {
    "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
    "id": "urn:uuid:7e10e4f3-a0d8-4430-95bd-783ffae4d916",
    "type": "ToolUseEvent",
    "profile": "ToolUseProfile",
    "eventTime": "2020-01-15T10:15:00.000Z",
    "actor": "https://example.edu/users/554433",
    "action": "Used",
    "object": "https://example.edu"
  },
  {
    "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
    "id": "urn:uuid:9r34jdfj-a0d8-4430-95bd-783ffae4d916",
    "type": "ToolUseEvent",
    "profile": "ToolUseProfile",
    "eventTime": "2020-01-15T11:15:00.000Z",
    "actor": "https://example.edu/users/554433",
    "action": "Used",
    "object": "https://example.edu"
  }
  ]
}

7.2 Sending to a secured HTTP endpoint

The current standard way of sending Caliper events is via a secured HTTP endpoint using an Authorization header with a Bearer token. This is described in the Caliper spec HTTP Request section.

This is currently the only method supported for Caliper Conformance.

An application that wants to send these events must either work with a Caliper Consumer directly to get an endpoint URL and authorization token, or use the connection passed via an LTI service connection.

7.3 Other ways to send Events

Although HTTP messages with an authorization header is the only method currently supported for certification, there are many other ways a Caliper Sensor could send events to a Caliper Consumer. Each integration should use the mechanisms that work best for them and allow the desired scaling possibilities.

For example, a common method is putting Caliper Events into a queue provided by a IaaS provider's queueing service and providing the Caliper Consumer the ability to connect to the queue for consumption.

8. Consuming Caliper Events

There are many considerations for consuming Caliper Events. This document will discuss some high-level issues, but the many privacy and security implications of receiving and holding learner data must be managed by a team capable of implementing the proper infrastructure.

8.1 Authorization

As discussed above in Sending to a secured HTTP endpoint, the current standard for authenticating Caliper Events from a sender is via an Authorization header with a Bearer token. This is described in the Caliper spec HTTP Request section. Each learning tool a Caliper Consumer wishes to receive events from will require an exchange of an HTTPS endpoint to send the events to and an authorization token. Another method is using the LTI Caliper Connector service.

If a consumer of Caliper data is utilizing the LTI service connection in conjunction with an LMS implementing that LTI service, they will have to work together to establish the needed connection and authorization infrastructure.

There are many other methods of sending and receiving events such as utilizing a queueing system. These are excellent options but they're not covered in this document.

8.2 What implementation considerations exist for Caliper Consumers?

Several implementation considerations exist for Caliper Consumers. Typically, these consumers play a role during the initial ingestion portion of an overall data "pipeline". In such a pipeline, raw source data is emitted or generated, then ingested into a "Data Lake". A Data Lake is a storage repository than can store large amounts of structured, semi-structured or unstructured data. Ultimately, the benefit of a data lake approach is to enable flexibility, providing a place for data to be gathered from multiple sources in a variety of formats.

The following best practices are typically recommended for data pipeline components that process Caliper data:

  • Use Cheap Storage to Preserve Raw Event Messages. When storing event data, it's important not to lose the resolution of information available in the original, raw event data. Because of the volume, and size, of event data en masse, many Caliper consumers might use inexpensive, cloud-hosted persistent storage solutions to preserve raw event messages. Some available solutions can support annotation with metadata for cataloguing and lifecycle purposes, and integrate with other cloud-hosted services that can act as different functional components of a data lake.
  • Take Advantage of Stream Processing Components. Many existing cloud infrastructure components are useful for bundling groups of Caliper messages arriving at the same time into JSON files, which are then kept in folders organized by year/month/day/hour. In addition, these existing components from IaaS providers can be used to keep running totals or to detect anomalies in event streams.
  • Set Up Ingest Processes With Known Mapping and Transform Rules. Examples of tasks which may need to be performed with inbound Caliper data are as follows:
    • Identify and locate common identifiers across the incoming data records.
    • Identify mappings between similar but differently named data fields and define logic for any transformations (parsing out specific identifiers from string fields, for example).
    • Determine how to handle display fields that contain strings that might be too long or have unsupported characters
    • Some data records may not contain an identifier that can be used to commonly join them across sources. In those cases, you will need to maintain a set of mapping tables with locally-sourced identifiers and their mappings to global identifiers.
    • This may mean that you need to manufacture a global set of identifiers to unify the data across systems.
    • Coordinate and communicate with any departments/groups who own or can help locate the "source of truth" for identifiers.
  • Don’t Forget About Reference Data. Set up processes to bring in reference data (users, departments, calendar events, work project names).

9. Code Libraries

Caliper makes available reference implementations for Sensors in the following programming languages. Links are provided to the GitHub repositories for each

These libraries are written and maintained by IMS Global Learning Consortium members. They should be considered as reference implementations only and as such are offered without any warranty. Use of these code libraries in a production environment should be thoroughly vetted by your development team. We welcome the posting of issues by non IMS Global Learning Consortium members (e.g., feature requests, bug reports, questions, etc.) but we do not accept contributions in the form of pull requests from non-members. For more information, please refer to the readme in the associated repo.

10. Use Cases

This section describes a handful common scenarios for the Caliper Specification. This section is in no way comprehensive but is intended to give the reader an idea of the type of education ecosystem questions that Caliper Analytics® can help to answer for.

10.1 Student Facing Analytics

As students interact with content within an LMS course site, Caliper events are triggered and transported to an event store. The Caliper events are represented in the controlled vocabulary of the Caliper metric profiles. The events are then parsed to extract commonly sought values into columns in a relational database. For example, information such as the event type, the event action, event time, actor, and course is extracted. The event record can also be enriched using related information based on global identifiers such as the course and actor which facilitates joining data from multiple sources.

Downstream analytics applications can then query the event store for events that represent select interactions the students within a course have had with learning content such as a student previewing a file, starting a video, or reviewing a lecture recording. Those events are then visualized so that a student can see what percentage of the class has viewed those resources, when I as the student last viewed that file, and how many times I’ve viewed it. Students can use this information to gauge their performance against their peers and take corrective actions where there may be an issue.

10.2 Understanding value of licensed learning tools

Using the same Caliper data being collected at the course level in the prior scenario, but at the cohort or system level, can be leveraged to determine in aggregate how much usage a licensed tool is getting across a cohort or within a segment of students. This data could be used to determine when it is safe to perform upgrades when it is time to begin migration planning or other operational concerns with various learning tools

10.3 Profile Use Cases

Caliper Metric profiles represent foundational usage cases. Each Metric Profile has it's own subset of use cases and user stories listed in it's primary specification document as linked to in this document.

For example, the Assessment Profile presents the following use case:

The Caliper Assessment Profile models assessment-related activities including interactions with individual assessment items. Caliper provides Assessment and AssessmentItem entities for describing the object of these activities, as well as a learner's Attempt for recording a count of the number of times an assigned resource has been attempted.

11. Best Practices

Below are some of the collected best practices from members who have successfully implemented the Caliper Analytics® standard.

11.1 Extending Caliper

Major Caliper releases of the base specification will not be released more frequently than every 18 months. This is to help the market have confidence in upgrading to the latest version each time. However, there are extension mechanisms in place to provide sufficient flexibility for all use cases. New Metric Profiles can also be created and published without a new release of Caliper.

11.1.1 Extending Entities and Events

All Entities and Events have an optional property called extensions which is a map of undefined key:value pairs. Other than syntax, this property is ignored during certification. Since these properties are not defined as part of Caliper the emitter and consumer of these events will need to work together to understand the meanings of extension properties and it should not be assumed all consumers will understand the data included in the extensions area.

There are no restrictions on key names, but it is best practice to use a name less likely to collide with others who are also using custom extensions. A common practice for this is using a name like "com.toolVendor.identifier_type".

The values can also be any type or complex objects, though simpler values are recommended.

Using extension properties is encouraged, but they should still be used with caution and purpose since over-use can limit interoperability, and increase payload sizes. There are other such considerations discussed in this section: Deciding how much data to send in an Event

11.1.1.1 Example: Extended AssessmentItem Entity

It is recommended that extra information needed for analytics purposes is retrievable by using the ID of an entity, but that isn't always a viable or scalable option so in this example the implementors decided to include the questionType with each assessment item in the extensions area.


{
  "id": "https://example.edu/terms/201601/courses/7/sections/1/assess/1/items/6",
  "type": "AssessmentItem",
  "dateCreated": "2016-08-01T06:00:00.000Z",
  "datePublished": "2016-08-15T09:30:00.000Z",
  "maxScore": 5.0,
  "extensions": {
    "questionType": "Short Answer",
    "questionText": "Define a Caliper Event and provide examples."
  }
}

11.2 Deciding how much data to send in an Event

Deciding what data to send depends heavily on the context and use-case. The consumer and producers of the Caliper Events should work together to figure what is needed. Here are some considerations for this conversation:

  • Data generated by the emitter which would otherwise be lost (i.e. not persisted) is a good candidate to convey in a Caliper event.
  • If an object is readily accessible via dereferenceble IRI's then use those instead of including all the object details.
  • If there are cases where downstream processes would need quick access to the actual generated data, it is ok to include those data values as well.
  • It can be good to only emit what you generate: in other words, if your application has to do any extra work to retrieve related information, it's not recommended to include it in the event payload.

11.3 Tracking sessions across multiple LTI tools

When receiving an LTI Launch via the LTI service connection use the caliper_federated_session_id as the id of the LtiSession you include in events:


 {
   "id": "urn:uuid:1c519ff7-3dfa-4764-be48-d2fb35a2925a",
   "type": "LtiSession"
 }

You add it to an event using the federatedSession key:


 {
   "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
   "id": "urn:uuid:9r34jdfj-a0d8-4430-95bd-783ffae4d916",
   "type": "ToolUseEvent",
   "profile": "ToolUseProfile",
   "eventTime": "2020-01-15T11:15:00.000Z",
   "actor": "https://example.edu/users/554433",
   "action": "Used",
   "object": "https://example.edu",
   "federatedSession": {
     "id": "urn:uuid:1c519ff7-3dfa-4764-be48-d2fb35a2925a",
     "type": "LtiSession"
   }
 }

For more details about the LtiSession see the Creating a ToolLaunchEvent section.

11.3.1 Other LTI Entity IDs

There is a lot of other information that can be passed along with an LTI Launch. Here are some guidelines to consider when deciding what information to include with your Caliper events:

  • pick out the system identifiers from the LTI message that are relevant to you as a Tool, and put them in as SystemIdentifiers on the entities in your events that are relevant. This helps:
    • bind those identifiers to the things they belong to, and not to the "session"
    • data consumers easily find those properties rather than having to dig through a big bag of Event.federatedSession.messageProperties keys.
      • Any identifiers generated by your app should be represented as first class properties in the Caliper event; however any identifiers which were passed into your application (say as LTI launch parameters) should be included as federatedSession properties.

11.4 Including Identifiers from other systems

All Entities support an ordered collection of SystemIdentifier entities that represent other identifiers for the Entity that are known to the Sensor. The list of supported types and their descriptions can be found in the main Caliper specification here: identifierTypes.

Here is an example of adding an extra LisSourcedId identifier to a course offering:


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "id": "https://example.edu/terms/201601/courses/7",
  "type": "CourseOffering",
  "courseNumber": "CPS 435",
  "academicSession": "Fall 2016",
  "name": "CPS 435 Learning Analytics",
  "otherIdentifiers": [
    {
      "type": "SystemIdentifier",
      "identifier": "example.edu:SI182-F16",
      "identifierType": "LisSourcedId"
    }
  ],
  "dateCreated": "2016-08-01T06:00:00.000Z",
  "dateModified": "2016-09-02T11:30:00.000Z"
}

The main Caliper documentation has other examples if you scroll to the example JSON after the properties table: CourseOffering, CourseSection, Person, Envelope with Mixed Payload Example

11.5 Robustness expectations

As an analytics specification, Caliper is not generally well suited to other uses that require more robust guarantees around timeliness and completeness. A few items of consideration are listed below.

  • Caliper is not specified as a messaging or transaction service
  • There are no guarantees around delivery timing in the specification
  • Events in Caliper are not ordered and may come much later, or in batches, etc.
  • Any integration parties may plan to use events as transaction but they will need to design their requirements together and not rely on Caliper certification implying others meet their extra expectations.
  • Using Caliper for official grade or enrollment data exchange is not recommended. Use the other IMS specifications built to solve those integration problems.

11.6 JSON-LD

JSON-LD is a specification providing a JSON-based data serialization and messaging format, processing algorithms, and API for working with Linked Data. Implementations of Caliper do not need to utilize JSON-LD itself to emit or consume Caliper events as they can be treated as regular JSON objects. If your implementation utilizes some of the benefits of JSON-LD that is excellent, but it is important to realize most implementations will not.

12. Caliper in the IMS ecosystem

As an analytics standard, Caliper is relevant to almost every other IMS specification in some way. These sections will cover some of ways Caliper interacts with other specifications.

12.1 CASE - Competencies and Academic Standards Exchange

is used to align learning content and activities with academic standards. It might be useful to pass the CASE standard a Caliper Entity is aligned with via a LearningObjective Entity included with the Object in a Caliper Event.

Researchers and analytics who collect data aligned to learning objectives can use it to correlate between learning activities and academic standards.

Wherever CASE-aligned learning objectives are associated to DigitalResources you may want to include them in the learningObjectives array on your resource.

If the Caliper endpoint is able to retrieve this information from another source then it may be better to not include the LearningObjective to help keep the Caliper Event's JSON smaller.

Here is an example of a CASE LearningObjective with a full object (name, description, and otherIdentifiers are optional)


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p1",
  "id": "https://example.edu/terms/202001/courses/7/sections/1/assign/2",
  "type": "AssignableDigitalResource",
  "name": "Reading Assignment: Research techniques",
  "learningObjectives": [
    {
      "id": "urn:uuid:2f5e8130-46fa-11e7-b197-cd3432e719f9",
      "type": "LearningObjective",
      "name": "Research techniques",
      "description": "ELAGSE1RL1 Ask and answer questions about key details in a text.",
      "otherIdentifiers": [
          {
              "type": "SystemIdentifier",
              "identifier": "https://case.georgiastandards.org/ims/case/v1p0/CFItems/2f5e8130-46fa-11e7-b197-cd3432e719f9",
              "identifierType": "CaseItemUri"

          }
      ]
    }
  ]
}

For brevity, it is also valid to just include the id in the learningObjectives array like this:


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p1",
  "id": "https://example.edu/terms/202001/courses/7/sections/1/assign/2",
  "type": "AssignableDigitalResource",
  "name": "Reading Assignment: Research techniques",
  "learningObjectives": [
    "urn:uuid:2f5e8130-46fa-11e7-b197-cd3432e719f9",
    "urn:uuid:2f5eda0e-46fa-11e7-b90a-62096a00843f"
  ]
}

12.2 Learning Tools Interoperability

12.2.1 How does an LTI Tool know where to send Caliper events?

An LTI Platform can use the LTI Caliper Connector service details to indicate to an LTI Tool where it should send Caliper events and entities. The LTI Tool will use the standard LTI workflow for authentication in order to use this service.

This service also specifies that the Caliper.sessionId value should be sent on an LTI launch. This is what Caliper calls the "Federated Session ID" as described in the How are sessions tracked across multiple LTI tools? section.

12.2.2 Tracking LTI tool usage and privacy issues with Tool Launch Profile & LTI Insights

For the institutions that heavily use LTI tools they often want to know how much a tool is used, and what information about the student and class is being sent to 3rd party vendors. There are 2 Caliper Profiles that help make this information available to faculty and administrators.

The Tool Use and Tool Launch profiles are meant to help understand how LTI tools are used at an institution. Their use is highly recommended for all LTI tools.

The LTI Insights project is a great example of monitoring LTI usage at an institution.

12.2.3 Creating a ToolLaunchEvent

The ToolLaunchEvent is an event created when a user "launches" an LTI tool from a Platform, usually an LMS. This event can be used help schools know what tools are being used and how much. The federatedSession is required and allows you to track a user's session across many launches and multiple tools.

A simple ToolLaunchEvent would look like this:


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "id": "urn:uuid:a2e8b214-4d4a-4456-bb4c-099945749117",
  "type": "ToolLaunchEvent",
  "profile": "ToolLaunchProfile",
  "actor": {
    "id": "https://example.edu/users/554433",
    "type": "Person"
  },
  "action": "Launched",
  "object": {
    "id": "https://example.com/lti/tool",
    "type": "SoftwareApplication"
  },
  "eventTime": "2018-11-15T10:15:00.000Z",
  "federatedSession": {
    "id": "urn:uuid:1c519ff7-3dfa-4764-be48-d2fb35a2925a",
    "type": "LtiSession"
   }
}

In an LtiSession object you can capture the messageParameters from an LTI launch message. This launch information is primarily needed for the LTI tool to operate and not usually used for conveying learning activity so a sender of this event should generally only include the information actually needed for their purposes. For example, if you're tracking what privacy-related information is being sent to all your tool vendors, it's useful to send all the parameters to evaluate, but if you're just interested in usage then most of the launch parameters may be superfluous.

The MessageParameters look a bit different for different versions of LTI. Below are examples for LTI 1.3, 1.1 and family, and also an example of using this same object for integrations that aren't necessarily using LTI but can include data sent to a 3rd party tool.

12.2.3.1 MessageParameters Example LTI 1.3

LTI 1.3 uses JSON as its core message parameter mechanic, so it straight forward to include them in the MessageParameters of an LTISession like this:


{
   "id": "urn:uuid:1c519ff7-3dfa-4764-be48-d2fb35a2925a",
   "type": "LtiSession",
   "user": "https://example.edu/users/554433",
   "messageParameters": {
       "iss": "https://example.edu",
       "sub": "https://example.edu/users/554433",
       "aud": ["https://example.com/lti/tool"],
       "exp": 1510185728,
       "iat": 1510185228,
       "azp": "962fa4d8-bcbf-49a0-94b2-2de05ad274af",
       "nonce": "fc5fdc6d-5dd6-47f4-b2c9-5d1216e9b771",
       "name": "Ms Jane Marie Doe",
       "given_name": "Jane",
       "family_name": "Doe",
       "middle_name": "Marie",
       "picture": "https://example.edu/jane.jpg",
       "email": "jane@example.edu",
       "locale": "en-US",
       "https://purl.imsglobal.org/spec/lti/claim/deployment_id": "07940580-b309-415e-a37c-914d387c1150",
       "https://purl.imsglobal.org/spec/lti/claim/message_type": "LtiResourceLinkRequest",
       "https://purl.imsglobal.org/spec/lti/claim/version": "1.3.0",
       "https://purl.imsglobal.org/spec/lti/claim/roles": [
          "http://purl.imsglobal.org/vocab/lis/v2/institution/person#Student",
          "http://purl.imsglobal.org/vocab/lis/v2/membership#Learner"
       ],
       "https://purl.imsglobal.org/spec/lti/claim/context": {
          "id": "https://example.edu/terms/201801/courses/7/sections/1",
          "label": "CPS 435-01",
          "title": "CPS 435 Learning Analytics, Section 01",
          "type": ["http://purl.imsglobal.org/vocab/lis/v2/course#CourseSection"]
       },
       "https://purl.imsglobal.org/spec/lti/claim/resource_link": {
          "id": "200d101f-2c14-434a-a0f3-57c2a42369fd",
          "description": "Assignment to introduce who you are",
          "title": "Introduction Assignment"
       },
       "https://purl.imsglobal.org/spec/lti/claim/tool_platform": {
          "guid": "https://example.edu",
          "name": "Example Tool Platform",
          "url": "https://example.edu",
          "version": "1.0"
       },
       "https://purl.imsglobal.org/spec/lti/claim/custom": {
          "xstart": "2017-04-21T01:00:00Z",
          "request_url": "https://tool.com/link/123"
       },
       "https://purl.imsglobal.org/spec/lti/claim/lis": {
          "person_sourcedid": "example.edu:71ee7e42-f6d2-414a-80db-b69ac2defd4",
          "course_offering_sourcedid": "example.edu:SI182-F16"
       }
    },
   "dateCreated": "2020-01-15T10:15:00.000Z",
   "startedAtTime": "2020-01-15T10:15:00.000Z"
 }
12.2.3.2 MessageParameters Example for LTI 1.0, 1.1, 1.2

Previous versions of LTI used properties in a form POST, so you'll need to convert each key/value pair for the MessageParameters like this:


{
   "id": "urn:uuid:1c519ff7-3dfa-4764-be48-d2fb35a2925a",
   "type": "LtiSession",
   "user": "https://example.edu/users/554433",
   "messageParameters": {
        "lti_version": "LTI-1p0",
        "lti_message_type": "basic-lti-launch-request",
        "resource_link_id": "120988f929-274612",
        "user_id": "292832126",
        "roles": "Instructor",
        "lis_person_name_full": "Jane Q. Public",
        "lis_person_contact_email_primary": "user@school.edu",
        "lis_person_sourcedid": "school.edu:user",
        "context_id": "456434513",
        "context_title": "Design of Personal Environments",
        "context_label": "SI182",
        "tool_consumer_instance_guid": "lmsng.school.edu",
        "tool_consumer_instance_description": "University of School (LMSng)",
        "custom_xstart": "2017-04-21T01:00:00Z",
        "custom_request_url": "https://tool.com/link/123"
   },
   "dateCreated": "2020-01-15T10:15:00.000Z",
   "startedAtTime": "2020-01-15T10:15:00.000Z"
 }
12.2.3.3 MessageParameters Example for an integrated non-LTI tool

Despite the name "LtiSession", these session objects can be used to model non-LTI integrations just as well as LTI ones. The messageParameters object has no restrictions and can thus be used to hold data from any kind of integration.

For example, LTI 1.3 builds on top of the popular OpenID Connect (OIDC) standard for verifying a user's identity between integrated systems. If a platform was using a non-LTI OIDC integration the messageParameters could look something like this:


{
   "id": "urn:uuid:1c519ff7-3dfa-4764-be48-d2fb35a2925a",
   "type": "LtiSession",
   "user": "https://example.edu/users/554433",
   "messageParameters": {
       "iss": "https://example.edu",
       "sub": "https://example.edu/users/554433",
       "aud": ["https://example.com/lti/tool"],
       "exp": 1510185728,
       "iat": 1510185228,
       "azp": "962fa4d8-bcbf-49a0-94b2-2de05ad274af",
       "nonce": "fc5fdc6d-5dd6-47f4-b2c9-5d1216e9b771",
       "name": "Ms Jane Marie Doe",
       "given_name": "Jane",
       "family_name": "Doe",
       "middle_name": "Marie",
       "picture": "https://example.edu/jane.jpg",
       "email": "jane@example.edu",
       "locale": "en-US"
    },
   "dateCreated": "2020-01-15T10:15:00.000Z",
   "startedAtTime": "2020-01-15T10:15:00.000Z"
 }

Perhaps a school would like to also track logins via SAML or Shibboleth via a Caliper event. The assertions and messages could be converted from their XML values into a simple format that fits into a messageParameters object. That could look something like this:


{
   "id": "urn:uuid:1c519ff7-3dfa-4764-be48-d2fb35a2925a",
   "type": "LtiSession",
   "user": "https://example.edu/users/554433",
   "messageParameters": {
       "login_type": "saml",
       "saml_version": "2.0",
       "response_id": "07940580-b309-415e-a37c-914d387c1150",
       "IssueInstant": "2020-01-15T10:15:00Z",
       "Destination": "http://sp.example.com/demo1/index.php?acs",
       "InResponseTo": "_4fee3b04709234efa4234755273d56685",
       "Issuer": "http://idp.example.com/metadata.php",
       "signed": "true",
       "Status": "urn:oasis:names:tc:SAML:2.0:status:Success",
       "Subject_NameID": "_ce3d2948b4cf20146dee0a0b3dd6f69b6cf86f62d7",
       "Audience": "http://sp.example.com/demo1/metadata.php",
       "AuthnContextClassRef": "urn:oasis:names:tc:SAML:2.0:ac:classes:Password",
       "mail": "test@example.com",
       "eduPersonAffiliation": ["users", "learners"]
   },
   "dateCreated": "2020-01-15T10:15:00.000Z",
   "startedAtTime": "2020-01-15T10:15:00.000Z"
 }

Since the format and contents of the messageParameters for non-LTI tools isn't strictly specified, the senders and receivers of these events should work closely together to make sure both sides understand the meaning of the fields and what information is needed for the desired use case.

LTI Resource Search is a standard that defines how to search digital repositories for a set of resources. It addresses searching learning object repositories (LORs), and other catalogs of learning resources, from learning tools using various attributes of resources and returning full metadata about the resources to the learning tools. Results can be launched either as URLs or LTI links.

If you wish to collect Caliper analytics events for users interacting with a Resource Search application, you can use the SearchEvent that is within the Search Profile.

In Caliper terms, a search "generates" a SearchResponse object to be included in the event. You can read the details in the SearchResponse documentation.

The searchProvider is the SoftwareApplication providing the search interface and returning the results. The searchTarget is what is being searched such as a text book or LOR application. Both these fields are optional but may be useful depending on your use case. Generally the object of the Search Event should be the same as the searchTarget.

Since the SearchResponse will usually be a desired object to include, this event will usually be sent after the search results have been generated, not before.

The id of the Query should be the URL used for the search. The id of the SearchResponse can be any IRI, but depending on your usage, it may useful to also make that ID contain the search URL.

This is an example of what a Resource Search event might look like:


{
  "@context": "http://purl.imsglobal.org/ctx/caliper/v1p2",
  "id": "urn:uuid:cb3878ed-8240-4c6d-9fee-77221810f5e4",
  "type": "SearchEvent",
  "profile": "SearchProfile",
  "eventTime": "2020-01-15T10:05:00.000Z",
  "actor": {
    "id": "https://example.edu/users/554433",
    "type": "Person"
  },
  "action": "Searched",
  "object": {
    "id": "https://example.edu/catalog",
    "type": "SoftwareApplication"
  },
  "generated": {
    "id": "https://example.edu/ims/rs/v1p0/resources?fields=url,name,description&filter=search~'cats'&limit=25",
    "type": "SearchResponse",
    "searchProvider": "https://example.edu",
    "searchTarget": "https://example.edu/catalog",
    "searchResultsItemCount": 3
    "query": {
      "id": "https://example.edu/ims/rs/v1p0/resources?fields=url,name,description&filter=search~'cats'&limit=25",
      "type": "Query",
      "creator": "https://example.edu/users/554433",
      "searchTarget": "https://example.edu/catalog",
      "searchTerms": "cats",
      "dateCreated": "2020-01-15T10:05:00.000Z"
    }
  }
}

12.4 OneRoster & LIS

Caliper Entities can contain a lot of contextual information beyond the interaction described in an Event. However, it isn't always good practice to include all that information because that can make your Events very large and repetitive. In many cases it's preferred to include as little contextual information as possible in an included Entity or Agent to keep the amount of data being sent to a minimum. In this case, since that contextual information is still likely needed for analysis, services like OneRoster and LIS can be used to fetch that information from SIS systems allowing you to get everything needed about a user, course, or section.

The identifiers included in your Caliper Events should be aligned to the identifiers in these SIS systems to allow this functionality. When planning an analytics project, you should make sure all the appropriate IDs are included for your needs.

For instructions on how to do that see the section Including Identifiers from other systems.

12.5 OpenVideo

The Media Profile can be used by platforms implementing OpenVideo.

12.6 QTI - Question and Test Interoperability

For a learner's interactions within a QTI assessment the Assessment Profile should be used. This profile is useful for all assessments whether driven by QTI or not allowing researchers and tool creators to send events about assessment generically.

A. Revision History

This section is non-normative.

Release Date Comments
21 October, 2020 Base Document Release
27 March, 2020 Candidate Final Release

B. References

B.1 Normative references

[CALIPER-12]
Caliper® Analytics Specification 1.2. Whyte, Anthony; Haag, Viktor; Feng, Linda; Gylling, Markus; Ashbourne, Matt; LaMarche, Wes; Pelaprat, Etienne. IMS Global Learning Consortium. URL: https://www.imsglobal.org/spec/caliper/v1p2
[CASE-10]
Reference not found.
[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. IETF. March 1997. Best Current Practice. URL: https://tools.ietf.org/html/rfc2119

C. List of Contributors

The following individuals contributed to the development of this document:

Name Organization
Anthony WhyteUniversity of Michigan
Viktor HaagD2L
Andrew GardenerUniversity of British Columbia
Yeona JangExplorance
Dan CarrollClever
Daniel GreenVitalSource
Sam SciollaUniversity of Michigan
Lance E SloanUniversity of Michigan
Kara ArmstrongUnizin
Markus GyllingIMS Global
Linda FengUnicon
André N'guettiaPennsylvania State University
Eric PrestonBlackboard, Inc.
Oxana JurosevicInstructure
Bracken MosbackerIMS Global
Joshua McGheeIMS Global

IMS Global Learning Consortium, Inc. ("IMS Global") is publishing the information contained in this document ("Specification") for purposes of scientific, experimental, and scholarly collaboration only.

IMS Global makes no warranty or representation regarding the accuracy or completeness of the Specification.

This material is provided on an "As Is" and "As Available" basis.

The Specification is at all times subject to change and revision without notice.

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

IMS Global would appreciate receiving your comments and suggestions.

Please contact IMS Global through our website at http://www.imsglobal.org.

Please refer to Document Name: Caliper Implementation Guide 1.2

Date: 27 March 2020