Sharebar?

CASE vs Rubrics standard vs Embedded Rubric blocks QTI3.

CASE vs Rubrics standard vs Embedded Rubric blocks QTI3.

Hi

We want to support performance- based 'exams' in our e-assessment platform. This by using rubrics and/or a checklist that an 'assessor/scorer' scores based on an artifact from a student. To give an idea, see this wireframe attachment

So we want to support the entire process from creation of the form/rubric sheet of the form to scoring. It is confusing for us to understand which standard is most suitable for this purpose (we have implemented a significant part of QTI 3.0 for construction and deliver engine). on the IMS website several topics relate to Rubrics

* QTI 3.0 also supports some from of rubrics (rubricblocks), how does this relate to CASE or bullet 2.
* There is standard specific for rubrics was published in 2005, should we use this instead ? https://www.imsglobal.org/ep/epv1p0/imsrubric_specv1p0.html
* We thought it was an interchange standard between system about learning activities and not designed to create performance based packages (like an QTI package)

Regards,

Adnan

ThomasHoffmann's picture

QTI 3 Rubrics

In QTI 2.x and 3.0, a "rubricBlock" is anything I want to show the candidate, the scorer, or the author – you can specify the audience(s) for the content. Then it's just QTI content (subset of HTML5 and what QTI 3 calls "catalog" content). You can also refer to a URL lookup. It is a much wider concept than a scoring rubric. It doesn't connect to the CASE standard really, though you can connect the item itself to curriculum standards via metadata in the package.

The Rubric specification is new to me -- not sure why our QTI group never seemed to discuss it (or even know it existed), and we actually were wondering how we might address a standardized approach to scoring rubrics! It would be interesting to see how much of your use case this Rubric Specification might fit.

I can raise it with the Assessment Product Steering Committee. Given that the standard was approved in 2005, it did not include CASE as a consideration. Might be a great time to see if this is worth updating. 

MarkMolenaar's picture

Use of Outcome Variables

In addition to Tom's point, the QTI rubric block with view="scorer" can contain holistic instructions on how to (manually) score an item.
You could then have a single Outcome Variable (e.g. the built-in one named SCORE) and have a scorer set it manually. An Outcome Variable can be assigned a (min and) max score, so you could then use these values to present a basic UI to a scorer. Important note: the rubric block and outcome variable are not technically linked, other than that they are part of the same item definition. There is a proposal from Cito (see https://docs.google.com/document/d/1T_mN7WxoMvi08gJZYm83HOhazspD1vHb/edit) to actually formally connect an Outcome Variable to a Rubric Block (by adding a data-outcome-idref attribute to the Rubric Block definition), but this proposal has not been implemented yet.

Alternatively, if you want to be more concise and prescriptive and fine grained, as you seem to be from the sample provided, you could also define multiple QTI Outcome Variables, e.g. SCORE_CLARITY, SCORE_ORGANIZATION, SCORE_MECHANICS) and set min and max scores to each of them. In addition you could also set the "interpretation" attribute for a short interpretation for each of these variables (e.g. "provide 0 if not clear, 1 if a bit clear, 2 if very clear") OR use the "long interpretation" to associate the Outcome Variable with an external file, e.g. a PDF with a marked up table containing fine grained instructions per Outcome Variable (scoring rubric). Based on this definition, you could then provide a UI to the scorer wich would allow them to set sub-scores individually (each with their own score range) and either display the short interpretation or link to the external files for instructions on how to score.

Considering CASE linking, you can link an entire item to a CASE definition through the manifest (see an example here: https://github.com/IMSGlobal/qti-examples/blob/27b93ba208fe2fdbe619c20b3...) and in theory could also use this approach to link individual Outcome Variable definitions if you would XInclude them and then put the appropriate metadata on the individual resources in the manifest. Not a very elegant solution though, so this could definitely be improved in the future.

Good luck!

Mark.

Use of Outcome Variables to assess Practical use-cases

Hi Mark , Thomas,

Thank you for the reply.

We have incorporated the structure of performance tasks/rubrics in the authoring module (UI/backend). We will move to a more QTI3.0 compliant 'mapping' based on your feedback. If we bump into new challenges, hope you can provide some feedback as well.

Regards,

Adnan Maakoul