Suggested Caliper Analytics Requirements for Institutional RFP & Procurement Agreements

This document is intended as a resource for IT organizations and procurement offices. It provides basic background information on the 1EdTech Caliper Analytics^® standard, an overview of versions, profiles, conformance standards, and sample RFP language when specifying Caliper Analytics as part of the procurement process. The suggested RFP wording is intended for learning tools and/or learning management systems (LMS).

What is Learning Data?

The 1EdTech Consortium Learning Data & Analytics Innovation Leadership Network has defined learning data as “... [referring] to data generated by students, faculty, and/or staff that relates to and documents the teaching and learning experience and academic achievement. It can be used alone or combined with the student record and other data points to support student success research.”¹ Within the scope of this resource, this definition is applied.

What is Caliper Analytics, and Why is Caliper Important?

The Caliper Analytics standard developed by 1EdTech members enables the collection, storage, and transport of data about learning. It supports an interoperable standard for data collection from multiple learning tools, enterprise systems, and the LMS. Caliper provides a common structure for improving the exchange of information and enables institutions to conduct more complete insight into student learning activities and more holistically inform learning dashboards, academic early warning systems, and advising tools.

Versioning & Certification

The current version of Caliper Analytics supports multiple metric profiles that reflect learning activity type. When your institution requires edtech suppliers to support Caliper Analytics you should compare the features of the learning tool(s) with Caliper’s metric profile to ensure the tool(s) support the learning activities that are to be measured. As the Caliper framework matures, additional metric profiles will be developed. It is strongly recommended that you outline in your RFP that suppliers must implement the most current version and metric profiles within a specified number of days of their release. This will ensure your institution can collect the broadest types of data and keeps your supplier in sync with the rest of the edtech industry.

1EdTech provides and maintains a directory of products that have met conformance criteria for all of its standards, including Caliper Analytics. Conformance and certification should not be confused with compliance. 1EdTech certification requires extensive testing that exceeds claims of “compliance.” Products listed in the directory are guaranteed to meet the 1EdTech standards for which they have passed testing. If you experience an integration issue with an 1EdTech certified product, 1EdTech will work with the supplier to resolve the problem.

All certified products are issued a certification number and are listed in the 1EdTech Certified Product Directory. To ensure that any tool consumer or tool provider is interoperable it is recommended that you require 1EdTech conformance certification as part of your procurement process and RFP documents.

Current and Future Metric Profiles

Suggested RFP Language and Resources for Procurement

There are a number of ways to include Caliper Analytics requirements as part of the RFP and acquisition process, ranging from a basic conformance inquiry to stringent adherence to a specific Caliper release by version and supported metric profile. Below is sample RFP wording that spans this range, as well as additional guidelines to request from the vendor partner as part of the procurement process:

Suggested RFP wording and clauses for requiring Caliper Analytics (open inquiry to stringent specification):
1. Meets the latest Caliper Analytics Specifications (https://www.imsglobal.org/caliper/caliperv1p0/ims-caliper-analytics-conformance-and-certification-guide-v10 ). Describe.
2. [The Institution name] is committed to a standards-based interoperable teaching and learning technology ecosystem. We are requesting [vendor/product] obtain certification by 1EdTech as a tool provider for Caliper Analytics version [specify] and the following profiles [specify] no later than [insert specific date].
3. [The Institution name] is requiring support for Caliper Analytics version [specify] and the following metric profiles [specify] and the tool consumer (LMS) must be certified as compliant with 1EdTech Caliper Analytics conformance certification. The vendor is required to provide a valid, current conformance certification registration number. See imscert.org. The vendor is also required to remain current with the subsequent Caliper Analytic versions, metric profiles and receive certification within a “reasonable” [define] time period.
Additional Caliper Analytics requirements to consider:
1. Data generated by student/faculty interaction with the [specific tool provider], whose stewardship is the responsibility of the institution of which those individuals are members.
2. [Specify tool provider] will provide to the transport protocol and institution with a real-time data stream in a Caliper format with specific user events to be defined in consultation with the institution.
3. The tool provider will also provide buyer access to a file containing monthly aggregations <a full ‘replay’ log?> of user activity data in Caliper format.
Suggested Guidelines
1. Verify that the vendor is 1EdTech certified for Caliper and have the vendor provide their certification number as listed in the 1EdTech product directory.
2. Any features of the specified Caliper Analytics version that are not supported by the tool, or if there are custom parameter substitution variables that are supported, these should be noted explicitly in the RFP response. See the conformance overview at https://www.imsglobal.org/caliper/caliperv1p0/ims-caliper-analytics-conformance-and-certification-guide-v10#2
3. Cloud-hosted application providers shall provide content that is certified compliant with Caliper Analytics and whatever metric profiles are relevant to the institution’s application. Evidence of a valid certification, including a current 1EdTech certified registration number, must be provided to the institution. See http://www.imscert.org/.
4. The vendor partner is required to disclose what parameters, variables, and services the tool requires so it can be matched and supported by the tool consumer or tool provider. The partner is also required to disclose optional services used.

Additional Items to Consider

Clear articulation as to ownership of the data.
Clear articulation and agreement specific to learning data privacy and security.
Sign FERPA agreement.
Is student identified data really necessary for the performance of the tool?
Tool consumer must provide an ISO security review.
Frequency and formatting of the data collected by the tool (ideally, daily and in Caliper Analytics format).
Is data synced between devices, cloud services, the client and the tool consumer (LMS)?
Where data is stored and how long?
When data is removed, how is the data removed, deleted or erased?

¹Learning Data & Analytics Key Principles

Suggested Caliper Requirements for RFPs

Suggested Caliper Analytics Requirements for Institutional RFP & Procurement Agreements

Current and Future Metric Profiles