1EdTech TECH TALK
Contributed by Dr. Colin Smythe, 1EdTech Chief Architect
Compliance, Conformance, and Certification: Why Getting 1EdTech Certified is Important
One of the benefits of 1EdTech membership is having your certified products listed in the 1EdTech Product Directory—the official list of all learning apps and tools that have passed 1EdTech interoperability certification. A product must demonstrate support of one or more 1EdTech specifications through conformance testing to appear in the directory. 1EdTech awards each certification for 12 months, so every product must undergo successful recertification to maintain its listing. This 12-month cycle allows vendors to use agile development processes without requiring recertification for every product release. New major versions of a product must be certified. It is not unusual for several versions of a product to show up in the product directory. It is important to note that the product receives certification and not a deployment of the product.
The 1EdTech Product Directory also includes products vetted for student data privacy using the 1EdTech TrustEd Apps process. Many, but not all vetted products, have also achieved 1EdTech standards’ certification. This blog focuses on the products that go through conformance testing for 1EdTech certification.
Defining the conformance requirements and providing the associated conformance test capabilities are essential to the 1EdTech specification development process. Each 1EdTech specification must have a Conformance & Certification document. These documents describe the certification process and define the conformance criteria that a product must achieve for each available certification. Most specifications have more than one certification. An example of this is a service-based specification with certifications as a Service Provider and a Service Consumer (a product must be either or both). The conformance and certification aspects are addressed once the project group responsible for developing the specification publishes the Member Candidate Final documents. These documents are available to 1EdTech members only. A minimum number of products must be certified before the Final Release of an 1EdTech specification can be published. This means the document set is publicly available to everyone.
An 1EdTech specification cannot have a Final Release until 1EdTech members define conformance and certification and create and use the conformance test.
One objective of 1EdTech certification is to demonstrate the level of adoption by vendors committed to open solutions to the market. When a new version 1.0 specification is first published, the conformance requirements are defined to encourage broad adoption. Over time the level of adoption will change, and the specification itself will evolve. Therefore, the 1EdTech specification maintenance process allows for the certification requirements to be changed to fit the changing needs of a market—even when there are no changes to the functionality supported by the specification. Also, it is not unusual for the conformance testing to be continually improved. This flexible approach to certification is another reason why products must undergo annual recertification.
It is becoming more common for organizations to require 1EdTech specifications as part of the procurement process. It is natural for vendors to claim compliance. From an 1EdTech perspective, compliance is claimed by vendors who are not 1EdTech Certified. 1EdTech members can provide their 1EdTech product registration numbers, and users can easily confirm their certification by a quick inspection of the 1EdTech Product Directory. If support of an 1EdTech specification is required, the tendering process should include checking the product’s 1EdTech Registration Number. In most cases, the claim for compliance is wishful thinking and based on unjustified confidence in the in-house interoperability testing. Sometimes, it is a cynical misrepresentation. Buyer beware.
When products claim compliance but are not certified, there are two implications. First, the product has not been through 1EdTech conformance testing. Usage of the 1EdTech conformance test systems is essential in producing a correct implementation of the specification. In most cases, a solution goes through several iterations of conformance testing before being certified. There is no restriction on the usage of our conformance test system for 1EdTech members, meaning they are not just for certification. Secondly, if there is a failure of interoperability when using the 1EdTech specifications, the 1EdTech certification requires the vendors to work together, and if appropriate with 1EdTech, to resolve the problem. In some cases, 1EdTech may have to: improve the implementation guidance, correct the specification, and improve the conformance test systems to avoid such incompatibilities in the future. Experience has shown that products that are not certified do not implement the corresponding 1EdTech specification correctly. Superficially, they appear to work, but there will likely be many significant errors in the implementation.
Certification requires 1EdTech membership. Is access to certification sufficient justification for 1EdTech membership? Undoubtedly, YES!
1EdTech has invested millions of dollars in developing and supporting our extensive test and conformance systems and related artifacts. Five to ten full-time software developers are working on the various 1EdTech test and conformance systems at any one time. Even the largest organizations see significant benefits in using the 1EdTech test and conformance systems. A further benefit is that the 1EdTech technical team provides a wide range of support to help 1EdTech members adopt and adapt 1EdTech specifications. As part of our specification development process, 1EdTech creates the following testing and conformance artifacts:
Service Provider and Consumer conformance test systems (used for OneRoster, LTI, CASE, etc.)
Reference implementations of the full specification
Online validation of content instances (used for Common Cartridge, QTI, etc.)
Reference test sets for testing data import capabilities (used for Common Cartridge, OneRoster, QTI, etc.)
It is important to stress that all of these artifacts are available, for unlimited use, to 1EdTech members. All of these artifacts are being continually improved.
While Certification is very important, it is product-focused and not deployment-specific. There is a limit to the degree of interoperability guaranteed through certification only. It is possible for two products certified for the same specification not to interoperate. For example, in OneRoster, there are both REST-based and CSV-based bindings, and interoperability between these is not possible. Therefore being certified alone is insufficient; the right type of certification is required for interoperability. The 1EdTech Product Directory provides sufficient details to ensure the right type of certification is available. In the case of deployed systems, there are many different ways in which a certified system can be configured (this may also depend on the business model used by the vendor).
As the next step beyond certification, 1EdTech has created the Compatibility Check (CCx), which provides the Characterization of a deployment. Furthermore, CCx enables characterizations to be compared. This means that we can compare the characterizations of certified Service Providers and Consumers to show all of the interoperable and non-interoperable features (including the usage of extensions). At present, CCx supports OneRoster and Common Cartridge, but it will be extended to cover many of the 1EdTech specifications over time. I will go into more details about characterization and CCx in a later blog but understand that the characterization of products is the way forward. It is a far better measure of interoperability than certification alone.