Resource Efficacy with Caliper Analytics
By: Adam Blum, ACT
Published July 2017
Introduction
In this document, we describe one approach to computing instructional resource efficacy to help deliver the most effective resources at the right time for learners. We then describe how the 1EdTech Caliper Analytics® specification for storing educational events such as assessment results and media views can be used to compute those efficacy measures. Finally, we suggest enhancements to add query REST web services to the 1EdTech Caliper standard that should make it much easier for people wishing to gain insights into the instructional resource and assessment usage for efficacy and other purposes.
Background
ACT offers a large educational resource library for assessment and instruction of K-12 students. ACT focuses on determining the learning objectives of those resources and then the efficacy of the instructional resources towards achieving those determined learning goals. ACT uses third-party assessment platforms to execute its assessment content, which return assessment events via Caliper. This allows us to compute efficacy for instructional resources from those assessment results, even though we do not control the assessment platform or have access to the underlying database.
Resource Efficacy
One straightforward way to measure resource efficacy is to average the assessment scores of all users who take a targeted assessment immediately after interacting with the resource. That resource is rated by evaluating how students perform on an assessment aligned with a specific learning goal, subsequent to that student viewing the resource. If the same assessment is later taken again it is not used to evaluate that particular resource.
As a specific scenario, imagine student A watches video V1. She then takes the assessment A1 and gets a 90%. Student B watches the same video V1 and then takes assessment A1 and gets a 70%. So far the video will have an efficacy rating of 80% for that academic standard. Resources get an efficacy rating for each standard they are aligned, derived from all student viewings and subsequent assessments on that standard. Details of this process are shown in Figure 1 below.
Figure 1. Process for rating efficacy of learning resources.
Computing resource efficacy involves looking at each “resource consumption” event (such as viewing a video) and then analyzing other actions taken by the student consuming the resource. For all assessments taken by the student subsequent to the resource consumption event, if any of the learning goals of the resource (such as a Common Core standard) intersect with the learning goals of the assessment, add the assessment score to the resource’s ratings for that specific learning goal. Once this is complete, the resource is labeled with an average effective rating for each learning objective associated with the resource.
Detailed Technical Description
The Caliper standard has several built-in events that make it easier to compute instructional resource efficacy with any resource library or software platform that supports Caliper events. The efficacy computation methods described above can be performed with just GradeEvent and MediaEvent. We will describe how these definitions of resource efficacy can be computed with these stored Caliper events.
GradeEvent
Each time there is a GradeEvent (corresponding to an assessment’s final outcome) we need to determine who took the assessment, the assessment involved, the grade on the assessment, and the specific learning objectives being assessed in order to rate the efficacy of prior related instructional resources consumption. We bold those attributes in the sample GradeEvent below, aligned with the 1EdTech Competencies and Academic Standards Exchange™ (CASE™) specification. We also bolded the learningObjectives attribute because we at ACT use an extension of this attribute to reflect the CASE item URL as a link to the standard.
{
"@context": "http://purl.imsglobal.org/ctx/caliper/v1p1",
"id": "urn:uuid:a50ca17f-5971-47bb-8fca-4e6e6879001d",
"type": "GradeEvent",
"actor": {
"id": "https://example.edu/autograder",
"type": "SoftwareApplication",
"version": "v2"
},
"action": "Graded",
"object": {
"id": "https://example.edu/terms/201601/courses/7/sections/1/assess/1/users/554433/attempts/1",
"type": "Attempt",
"assignee": {
"id": "https://example.edu/users/554433",
"type": "Person"
},
"assignable": {
"id": "https://example.edu/terms/201601/courses/7/sections/1/assess/1",
"type": "Assessment"
"learningObjectives": [
{
"id": "http://opensalt.opened.com/cftree/item/19033",
"type": "LearningObjective",
"dateCreated": "2017-08-01T06:00:00.000Z",
"extensions": {
"case_item": "http://opensalt.opened.com/cftree/item/19033"
}
}
]
},
"count": 1,
"dateCreated": "2017-11-15T10:05:00.000Z",
"startedAtTime": "2017-11-15T10:05:00.000Z",
"endedAtTime": "2017-11-15T10:55:12.000Z",
"duration": "PT50M12S"
},
"Score": {
"id": "https://example.edu/terms/201601/courses/7/sections/1/assess/1/users/554433/results/1",
"type": "Score",
"attempt": "https://example.edu/terms/201601/courses/7/sections/1/assess/1/users/554433/attempts/1",
"normalScore": 3,
"totalScore": 3,
"comment": "Well done.",
"scoredBy": "https://example.edu/autograder",
"dateCreated": "2017-11-15T10:55:05.000Z"
},
"eventTime": "2017-11-15T10:57:06.000Z"
}
MediaEvent
From each MediaEvent the attributes for the user ID (specifically the “actor:{@id}” attribute), action (specifically “"action": "http://purl.imsglobal.org/vocab/caliper/v1/action#Ended"), an object ID (the “object:{@id}”) should be retrieved from the Caliper store. The following MediaEvent example has these attributes in bold. We also bold the learningObjective attribute.
{
"@context": "http://purl.imsglobal.org/ctx/caliper/v1p1",
"type": "MediaEvent",
"actor": {
"id": "https://example.edu/user/554433",
"type": "Person",
"dateCreated": "2017-08-01T06:00:00.000Z"
},
"action": "Ended",
"object": {
"id": "https://example.com/super-media-tool/video/1225",
"type": "VideoObject",
"name": "American Revolution - Key Figures Video",
"learningObjectives": [
{
"id": "http://opensalt.opened.com/cftree/item/19033",
"type": "LearningObjective",
"dateCreated": "2017-08-01T06:00:00.000Z",
"extensions": {
"caseItemUri": "http://opensalt.opened.com/cftree/item/19033"
}
}
]
}
}
Querying for Caliper Events
So all that needs to be done is to query for each GradeEvent and use that event to query for relevant MediaEvents from the same user on the same standard. The 1EdTech Caliper standard doesn’t quite layout simple query services for searching for either assessment outcomes or media events, by the user or by the standard. Instead, each Caliper event’s JSON-LD description has to be individually retrieved and compared against the criteria. Clearly, this is not something a developer would do in a production system with a lot of events. Instead, they would store the Caliper events (or a subset of the attributes of those events) into a relational database to enable querying by an appropriate set of attributes.
ACT created a free and open source service to enable this called Callisto. Source code for Callisto is on GitHub. Callisto defines several query methods that are relevant for computing efficacy of resources. These include the MediaEvents query which takes the following possible arguments:
- actor_id - the ID of the media viewer (typically student), e.g. "https://example.edu/user/554433"
- action_id - what happened with the video, e.g. "Ended"
- object_id - the URL of the video itself, e.g. "https://example.com/super-media-tool/video/1225"
- learning_objective - the standard associated with the video. For example, this is a CASE Item URI (pointing to CASE services exposed for the standard CCSS.Math.Content.6.SP.A.1): “http://opensalt.opened.com/cftree/item/19033”
For example:
curl https://opencallisto.org/media_events.json?actor_id=https://example.edu/user/554433
Here is the pseudo-code and the 1EdTech Caliper queries necessary to compute efficacy with the Callisto tool with the straightforward method described earlier. Note the pass required to compile the list of “votes on efficacy” for each resource based on each subsequently related assessments. And then the subsequent pass to take averages of the “votes on efficacy” generated by each assessment. That average will then be stored as the efficacy of each resource for each learning goal.
FORALL resources having a learning objective alignment: R
(performed with future 1EdTech resource search standard)
FORALL students S consuming that resource
(performed with Caliper MediaEvent query with actor_id=S, object_id=R, action_id=Ended)
FORALL assessments A taken subsequent to dated resource R consumed
(GradeEvent EventTime > MediaEvent EventTime attribute)
FORALL learning objectives on the resource: R.LO
If SET(A.alignedLearningObjectives)
INTERSECTS SET(R.alignedLearningObjectives)
(add the assessments “vote” on efficacy to the list of scores for
that particular resource and learning objective
R.votesOnEfficacy[R.LO].LIST_ADD(A.normalScore)
End
End
End
FORALL assessments A taken subsequent to resource consumed
FORALL learning objectives on resource: R.LO
R.efficacyRating[R.LO]=ListAverage(R.votesOnEfficacy(R.LO))
End
End
End
End
Extending the Specification
Caliper is a powerful tool to make efficacy analysis possible across boundaries of individual tools. To realize the full power of an event store, simple REST-based query return JSON payloads would be ideal. Below are proposed extensions to the Caliper standard that would address this “query gap” that could be included in Caliper 1.2 to enable the query of APIs for each Caliper event, such as GradeEvents. Parameters include:
- actor_id - the ID of the assessment taker, e.g. "https://example.edu/user/554433"
- object_id - the ID of the assessment attempt, e.g. "https://example.edu/politicalScience/2015/american-revolution-101/assessment/001/attempt/5678"
- object_assignable - the ID of the assessment, e.g. "https://example.edu/politicalScience/2015/american-revolution-101/assessment/0011"
- generated_id - the ID of the assessment result, e.g. "https://example.edu/politicalScience/2015/american-revolution-101/assessment/001/attempt/5678/result
- learning_objective - all GradeEvents for a particular learning objective, expressed as a CASE URL
Example REST call (all outcome events for specified user):
curl https://<hostname>/outcome_events.json?actor_id=https://example.edu/user/554433
Summary and Call to Action
Caliper is an exciting specification that should enable instructional resource efficacy to be determined regardless of the educational platform used. This is likely to be the case whether the described method for computing efficacy is used or some other approach is taken. However, any such method will need easy query of large amounts of educational events, in particular assessment results and media views, by various criteria including the learner (the Caliper “actor”), the resource or assessment (the Caliper “object”) or the action (such as “Ended” or “Graded”). Standardizing REST-based query web services, such as those provided by the Callisto open source project should make this much easier.
For More Information
All 1EdTech standards are free to download and free to use. To claim conformance to an 1EdTech standard, an implementing organization must complete the conformance certification process, which requires 1EdTech membership. Learn more about the 1EdTech Conformance Certification Program.
All products that have achieved 1EdTech conformance certification of any kind are listed and updated at imscert.org.
To find out how to use 1EdTech specification documents, go to https://www.imsglobal.org/usingimsdocuments.html.