Sharebar?

IMS App Note: Resource Efficacy with Caliper Analytics

IMS App Note header image

Resource Efficacy with Caliper Analytics

By: Adam Blum, CEO, OpenEd
Published July 2017

Introduction

In this document, we describe one approach to computing instructional resource efficacy to help deliver the most effective resources at the right time for learners. We then describe how the IMS Global Caliper Analytics® specification for storing educational events such as assessment results and media views can be used to compute those efficacy measures. Finally, we suggest enhancements to add query REST web services to the IMS Global Caliper standard that should make it much easier for people wishing to gain insights into the instructional resource and assessment usage for efficacy and other purposes.

Background

OpenEd offers a large educational resource library for assessment and instruction of K-12 students. OpenEd focuses on determining the learning objectives of those resources and then the efficacy of the instructional resources towards achieving those determined learning goals. OpenEd uses third-party assessment platforms to execute its assessment content, which return assessment events via Caliper. This allows us to compute efficacy for instructional resources from those assessment results, even though we do not control the assessment platform or have access to the underlying database.

Resource Efficacy

One straightforward way to measure resource efficacy is to average the assessment scores of all users who take a targeted assessment immediately after interacting with the resource. That resource is rated by evaluating how students perform on an assessment aligned with a specific learning goal, subsequent to that student viewing the resource. If the same assessment is later taken again it is not used to evaluate that particular resource.

As a specific scenario, imagine student A watches video V1. She then takes the assessment A1 and gets a 90%. Student B watches the same video V1 and then takes assessment A1 and gets a 70%. So far the video will have an efficacy rating of 80% for that academic standard. Resources get an efficacy rating for each standard they are aligned, derived from all student viewings and subsequent assessments on that standard.  Details of this process are shown in Figure 1 below.

 



Figure 1. Process for rating efficacy of learning resources.

Computing resource efficacy involves looking at each “resource consumption” event (such as viewing a video) and then analyzing other actions taken by the student consuming the resource. For all assessments taken by the student subsequent to the resource consumption event, if any of the learning goals of the resource (such as a Common Core standard) intersect with the learning goals of the assessment, add the assessment score to the resource’s ratings for that specific learning goal. Once this is complete, the resource is labeled with an average effective rating for each learning objective associated with the resource.   

Detailed Technical Description

The Caliper standard has several built-in events that make it easier to compute instructional resource efficacy with any resource library or software platform that supports Caliper events. The efficacy computation methods described above can be performed with just GradeEvent and MediaEvent. We will describe how these definitions of resource efficacy can be computed with these stored Caliper events.

GradeEvent

Each time there is a GradeEvent (corresponding to an assessment’s final outcome) we need to determine who took the assessment, the assessment involved, the grade on the assessment, and the specific learning objectives being assessed in order to rate the efficacy of prior related instructional resources consumption. We bold those attributes in the sample GradeEvent below, aligned with the IMS Global Competencies and Academic Standards Exchange™ (CASE™) specification. We also bolded the learningObjectives attribute because we at OpenEd use an extension of this attribute to reflect the CASE item URL as a link to the standard.

{
   "@context": "http://purl.imsglobal.org/ctx/caliper/v1p1",
   "id": "urn:uuid:a50ca17f-5971-47bb-8fca-4e6e6879001d",
   "type": "GradeEvent",
   "actor": {
       "id": "https://example.edu/autograder",
       "type": "SoftwareApplication",
       "version": "v2"
   },
   "action": "Graded",
   "object": {
       "id": "https://example.edu/terms/201601/courses/7/sections/1/assess/1/users/554433/attempts/1",
       "type": "Attempt",
       "assignee": {
           "id": "https://example.edu/users/554433",
           "type": "Person"
       },
       "assignable": {
           "id": "https://example.edu/terms/201601/courses/7/sections/1/assess/1",
           "type": "Assessment"

      "learningObjectives": [

           {

               "id": "http://opensalt.opened.com/cftree/item/19033",

               "type": "LearningObjective",

               "dateCreated": "2017-08-01T06:00:00.000Z",

               "extensions": {

                       "case_item": "http://opensalt.opened.com/cftree/item/19033"

                }

           }

       ]


       },
       "count": 1,
       "dateCreated": "2017-11-15T10:05:00.000Z",
       "startedAtTime": "2017-11-15T10:05:00.000Z",
       "endedAtTime": "2017-11-15T10:55:12.000Z",
       "duration": "PT50M12S"
   },
   "Score": {
       "id": "https://example.edu/terms/201601/courses/7/sections/1/assess/1/users/554433/results/1",
       "type": "Score",
       "attempt": "https://example.edu/terms/201601/courses/7/sections/1/assess/1/users/554433/attempts/1",
       "normalScore": 3,
       "totalScore": 3,
       "comment": "Well done.",
       "scoredBy": "https://example.edu/autograder",
       "dateCreated": "2017-11-15T10:55:05.000Z"
   },
   "eventTime": "2017-11-15T10:57:06.000Z"
}

 

MediaEvent

From each MediaEvent the attributes for the user ID (specifically the “actor:{@id}” attribute), action (specifically “"action": "http://purl.imsglobal.org/vocab/caliper/v1/action#Ended"), an object ID (the “object:{@id}”) should be retrieved from the Caliper store. The following MediaEvent example has these attributes in bold. We also bold the learningObjective attribute.

 

{

   "@context": "http://purl.imsglobal.org/ctx/caliper/v1p1",

   "type": "MediaEvent",

   "actor": {

       "id": "https://example.edu/user/554433",

       "type": "Person",

       "dateCreated": "2017-08-01T06:00:00.000Z"

   },

   "action": "Ended",

   "object": {

       "id": "https://example.com/super-media-tool/video/1225",

       "type": "VideoObject",

       "name": "American Revolution - Key Figures Video",

       "learningObjectives": [

           {

               "id": "http://opensalt.opened.com/cftree/item/19033",

               "type": "LearningObjective",

               "dateCreated": "2017-08-01T06:00:00.000Z",

               "extensions": {

                "caseItemUri": "http://opensalt.opened.com/cftree/item/19033"

               }

           }

       ]

   }

}

 

Querying for Caliper Events

So all that needs to be done is to query for each GradeEvent and use that event to query for relevant MediaEvents from the same user on the same standard. The IMS Caliper standard doesn’t quite layout simple query services for searching for either assessment outcomes or media events, by the user or by the standard. Instead, each Caliper event’s JSON-LD description has to be individually retrieved and compared against the criteria. Clearly, this is not something a developer would do in a production system with a lot of events. Instead, they would store the Caliper events (or a subset of the attributes of those events) into a relational database to enable querying by an appropriate set of attributes.  

OpenEd created a free and open source service to enable this called Callisto. Source code for Callisto is on GitHub. Callisto defines several query methods that are relevant for computing efficacy of resources.  These include the MediaEvents query which takes the following possible arguments:

For example:

curl https://opencallisto.org/media_events.json?actor_id=https://example.edu/user/554433

Here is the pseudo-code and the IMS Caliper queries necessary to compute efficacy with the Callisto tool with the straightforward method described earlier. Note the pass required to compile the list of “votes on efficacy” for each resource based on each subsequently related assessments. And then the subsequent pass to take averages of the “votes on efficacy” generated by each assessment. That average will then be stored as the efficacy of each resource for each learning goal.


  FORALL resources having a learning objective alignment: R

(performed with future IMS Global resource search standard)

FORALL students S consuming that resource

(performed with Caliper MediaEvent query with actor_id=S, object_id=R, action_id=Ended)

              FORALL assessments A taken subsequent to dated resource R consumed

(GradeEvent EventTime > MediaEvent EventTime attribute)

                    FORALL learning objectives on the resource: R.LO

                   If SET(A.alignedLearningObjectives)

INTERSECTS SET(R.alignedLearningObjectives)

     (add the assessments “vote” on efficacy to the list of scores for

that particular resource and learning objective            

                           R.votesOnEfficacy[R.LO].LIST_ADD(A.normalScore)

 End

                    End

              End

          FORALL assessments A taken subsequent to resource consumed

FORALL learning objectives on resource: R.LO

          R.efficacyRating[R.LO]=ListAverage(R.votesOnEfficacy(R.LO))

End

End

   End

  End

 

Extending the Specification

Caliper is a powerful tool to make efficacy analysis possible across boundaries of individual tools. To realize the full power of an event store, simple REST-based query return JSON payloads would be ideal. Below are proposed extensions to the Caliper standard that would address this “query gap” that could be included in Caliper 1.2 to enable the query of APIs for each Caliper event, such as GradeEvents. Parameters include:

 Example REST call (all outcome events for specified user):

curl https://<hostname>/outcome_events.json?actor_id=https://example.edu/user/554433

Summary and Call to Action

Caliper is an exciting specification that should enable instructional resource efficacy to be determined regardless of the educational platform used. This is likely to be the case whether the described method for computing efficacy is used or some other approach is taken. However, any such method will need easy query of large amounts of educational events, in particular assessment results and media views, by various criteria including the learner (the Caliper “actor”), the resource or assessment (the Caliper “object”) or the action (such as “Ended” or “Graded”).  Standardizing REST-based query web services, such as those provided by the Callisto open source project should make this much easier.    

For More Information

All IMS Global standards are free to download and free to use. To claim conformance to an IMS standard, an implementing organization must complete the conformance certification process, which requires IMS Global membership. Learn more about the IMS Conformance Certification Program

All products that have achieved IMS conformance certification of any kind are listed and updated at imscert.org.

To find out how to use IMS specification documents, go to https://www.imsglobal.org/usingimsdocuments.html.