Sharebar?

Critical Milestone Met: Conformance Certification by Leading e-Assessment Product Suppliers

Printer-friendly version

Today’s IMS announcement of the first group of winning products and organizations to undergo conformance certifications for QTI and APIP is a very, very big step for e-assessment interoperability worldwide. 

IMS certification seal

For years, no decades, the majority of e-assessment suppliers worldwide have been riding for free on the back of the IMS Question & Test Interoperability TM (QTI TM) specification. QTI has been a labor of love and importance within IMS and led by organizations such as SURF, JISC, University of Cambridge, ETS, BPS Bildungsportal Sachsen (BPS) and University of Pierre & Marie Curie.

These organizations, other than ETS, are not household names in most of U.S. education sector. But they tirelessly carried the load of developing really the world’s only viable interoperability standard for digital interoperability of tests, test items and associated results.

About two years ago the U.S. decided to invest in the Common Core learning standards for K-12 and also launched the Race-to-the-Top Assessment program to encourage states to cooperate on designing and delivering new electronic assessments in conjunction with the Common Core. Shortly ahead of that IMS collaborated with a group of U.S. states to define an evolution of QTI called APIP TM (Accessible Portable Item Protocol TM)which consists of QTI plus some new features to address requirements for special needs students. And, the Netherlands also began partnering with IMS to develop a countrywide initiative to evolve to e-assessment starting with one particular test (similar to the U.S. SAT or college entrance).

These investments in e-assessment have led to a dramatic rise in participation in QTI and APIP in IMS. If you look at the IMS membership list today it is arguably the who’s who of leading assessment organizations, certainly in the U.S. but also perhaps worldwide. The IMS APIP/QTI work over the last 2 years has been co-chaired by Measured Progress, ETS and Pearson with heavy involvement from McGraw-Hill CTB, ACT, Pacific Metrics, NWEA, Data Recognition Corporation and a variety of other assessment industry heavy weights. And in the Netherlands CITO has been leading the charge.

Life has been good. But market development and adoption of standards is always a kind of “chicken and egg” sort of thing.  As mentioned at the very beginning of this post worldwide assessment suppliers of many stripes had been talking up QTI for a very long time. Problem was that every supplier had their own version of QTI and therefore very little interoperability ensued. As we have discussed in other posts, this type of standardization does not deliver on the actual cost and time reduction that standards need to deliver on in the digital world. If conformance to a standard still requires lots of custom programming to get interoperability, well, then it isn’t a very good standard.

Thus, realizing this issues of “loose standards” running rampant in the ed tech sector the IMS members decided to get serious – and also save themselves lots of time and money in redoing integrations – by implementing IMS conformance certification. As we have discussed elsewhere, IMS conformance certification is not a marketing program (although those that go through it obviously do have the right to market that fact) but more of a “UL certified” designation of getting through a testing program. The conformance certification is much more than a “final test to the specification.” The conformance certification program is actually critical to evolving to the best possible specification for the needs of the marketplace. Typically only by going through the testing can the specification be refined and improved.  IMS has seen this process work over and over again with all our specifications the last several years.

The problem is that many vendors often kind of “hope for a miracle” many times with specifications. They hope that even without going through implementation and testing that magically a specification will work.  I think anyone that has ever developed software and does a little projecting of that experience on to a specification – that essentially must bring together the development process/experience of numerous software products – will realize that a good specification requires development participation and feedback from multiple vendors. The IMS conformance certification process – and ongoing developer community and related specification evolution (we call it an APMG: Accredited Profile Management Group) – is that “hub” where the development experience of the multiple suppliers comes together into a great specification.

It’s really a very simple concept but it is greatly complicated by the realities of new markets and new product development where suppliers are challenged to respond to the needs of their project deliverables and the needs to cooperate on the standards testing and evolution.

All that background so that you know that what IMS announced today, that five leading organizations have now completed conformance certification for APIP/QTI across a range of product types, is a huge step forward for the e-assessment community. By “community” we mean the suppliers and the states and end-users of e-assessments.  In addition to the leadership shown by the suppliers listed in this post, this milestone has required exemplary leadership from the end-user organizations that have been key partners in this, namely Maryland, Minnesota, WIDA, Smarter Balanced and the College Voor Examens Netherlands.

/apip/alliance.html

We are still relatively early in the adoption of high quality e-assessment worldwide. But what today’s announcement proves is that leading supplier and end-user organizations can come together to enable all the many benefits of interoperable assessments (for a more detailed discussion of these benefits see What You Need to Know About e-Assessment).

It is now time for those organizations that have either gathered around the IMS QTI/APIP table or been long claiming that they are “conforming” to these standards, to contribute to the community by participating in the conformance certification process.

Today’s announced winners were:

APIP:

Platinum:Educational Testing Service/Computerized Assessments and Learning TOMS v3.0.0.0 PNP system (APIP v1.0 PNP Core Compliant) and Sample Students’ Instances v1.0 (APIP v1.0 PNP Content Core Compliant)

Gold:Pacific Metrics Unity v1.9 (APIP v1.0 PNP Core Compliant, APIP v1.0 Item Test Bank Import Compliant)

Silver:Computerized Assessments and Learning Test Delivery system v2.3 (APIP v1.0 Delivery Entry Compliant)

QTI:

Platinum:BPS Bildungsportal Sachsen GmbH ONYX Testsuite v5.3.1 (QTIv2.1 Authoring Compliant, QTIv2.1 Delivery Compliant, QTIv2.1 Item Test Bank Compliant)

Gold:Northwest Evaluation Association Formative Assessment Item Bank v14.1 (QTI v2.1 Item Test Bank Compliant) and NWEA SCIP v14.1 (QTI v2.1 Content Compliant)

The winners will be honored and presented with their awards during the Learning Impact Awards ceremony at the 2014 Learning Impact Leadership Institute 5-8 May 2014 in New Orleans, Louisiana, USA www.imsglobal.org/learningimpact2014/.

Here are some links to addition press releases regarding this important milestone:

ETS Assessment Management System Provides Standardized Platform to Manage Statewide Assessments

Pacific Metrics’ Unity Platform Earns IMS Global Learning Consortium Assessment Conformance Certification

/apip/alliance.html