The following general screening criteria can be applied to screen available standards for implementation within a solution.

Review Fit for Purpose Criteria

Review Stewardship Criteria

Review Quality Criteria

Continue to Standards Specific Assessment Criteria.

Fit for Purpose Criteria

Fit for purpose criteria is used to evaluate the appropriateness of a standard within the intended business, clinical and technical context.

Aligns to the Digital Health Blueprint

Selection of existing interoperability specifications and terminology subsets which have been deployed to support similar use cases improves compatibility between solutions, supports rapid project delivery and decreases speed to value.

Selection of standards which are health care delivery setting independent can be reused across multiple health care delivery settings.

  • Has the proposed standard already been deployed in Canada to support substantially similar use cases?
  • Is the standard currently in use in Canada across a range of use cases and care settings?

Reuses an Existing, Shared Standard

Reuse of widely adopted standards across different interoperability specifications or terminology subsets allows reuse of code, applications, conformance testing environments, etc.  This also allows reuse of existing tools/skills/knowledge sets to establish the standard.

  • Was the proposed Interoperability Specification or Subset adopted/adapted from a widely used standard?
  • Was the proposed Interoperability Specification or Subset developed for use in Canada?
  • Is the base standard widely used in Canada?

Business Requirement Support

Selected standards must address all in-scope business requirements or be adapted to meet project needs.

  • Does the proposed interoperability specification have all the messages to support the required functions for data exchange? 
  • Do the messages have the appropriate fields to express the information required for the business functions?
  • Does the document standard have all the sections to express the required business information?
  • Does the terminology subset contain all of the concepts that need to be captured?
  • If bi-lingual information needs to be exchanged, does the standard allow for English and French terms to be mapped to common codes? 
  • Are there user interface data collection/display requirements that should be considered when choosing the interoperability standard?

Supports Technical Requirements

Standards selected must be feasible to adopt and implement according to the technical requirements and constraints of the project.

  • Can the standard be implemented in the proposed architecture? 
  • Does the conceptual architecture make any assumptions about synchronous or asynchronous communication? 
  • Does the architecture support HTTP and/or MLLP? 
  • Are there any interdependencies with other aspects of the architecture that would make it difficult to implement the standard (e.g. there are no places to express SAML bindings in HTTP without SOAP, making it difficult to integrate RESTful interfaces with the existing security system and therefore difficult to implement HL7 FHIR)?

Likelihood of Adoption

Standards that are adopted and incorporated into solutions, either by vendors or other implementers, have better staying power in the market and offer a greater opportunity to provide value back to implementers through future integrations.  Implementation costs associated with widely adopted standards may also be lower.

  • Have vendors already implemented this standard? How many?
  • What is the expected return on investment for a software vendor?
  • Do vendors have the necessary expertise to implement the standard? If not, is it realistic that they can gain or acquire that expertise in time to implement?
  • Will vendors provide support for sustained use and maintenance?

Appropriately supports coded, structured and free text content

Different standards support different data structures and levels of terminology coding.  These difference entail trade-offs to application developers and users. At one extreme, end users may resist using an application which forces them to enter data in too many fields, increasing their time/effort to produce documentation. At the other extreme, just capturing free text can contribute to privacy breaches and present a barrier to implementing decision support.  

  • Does the project implementing the standard expect to support automated processing such as: data aggregation comparison within decision support or analytics applications, standardization of data used to trigger process flows (i.e. presence or absence of terms), etc.
  • Do clinicians require assistance to exchange data/information for human readability?
  • Are there other forms of non-textual data/information (i.e. audio, video, images) that need to be exchanged?
  • Are there privacy concerns with exchanging free text?

 

Stewardship Criteria

Stewardship criteria support comparison of potential standard’s steward in terms of governance structure, licensing and intellectual property rights, and defined maintenance process.  These criteria are associated with total costs of ownership and the future availability of the standards selected.

Costs and Benefits of Implementation

Is the cost of implementation and disruption to current business affordable given the expected benefits?  It is important to consider not only the initial cost but the on-going cost for maintenance as well. Cost is not only measured in currency, but can be in time or effort as well.  

  • What is the cost of required tooling?
  • Will project implementation timelines be increased due to additional complexity introduced by the standard?
  • Will the standards investment help reduce the cost or risk of subsequent system implementations?
  • What is the number of people required for implementation?
  • Is there any existing commercial off the shelf software vs. custom software?
  • If properly implemented, would the standard enable downstream system or societal benefits such as improved decision support, clinical research or other data reuse?

Governance Structure

Well governed standards have processes in place for the voting, maintenance and changing of the standard amongst its various stakeholders; as well as certification and other capabilities of the standards development or maintenance organization. 

  • Are defined processes in place to facilitate decision making and issue resolution related to both standards content and processes?
  • Are the different communities who are responsible for and effected by the standard represented in the governance process?  Is it well balanced?
  • Are processes to add and remove members from the governance committees documented and compatible with project needs?

Intellectual Property and Licensing

Intellectual property policies, licencing terms and licensing costs can impact standards uptake.

  • Does the standard have licensing costs that are significant enough to inhibit uptake of the standard?
  • Is it likely the standard licensing costs will increase over time?
  • If a standard is currently free, are there other hidden conditions?
  • Do existing implementations provide network benefits which justify costs?
  • Does use of the standard offset other costs (such as maintenance)?
  • Does the standard have licensing restrictions that prevent it from being implemented by anyone?
  • What standards are being implemented in neighboring regions/countries due to differing licensing costs? Will these impact expectations in our country?

Maintenance Process

Business requirements and technology change over time and, as such, standards need to evolve to meet changing needs.  It is therefore important to ensure the selected standard has defined maintenance processes in order to evolve.  There are also costs and benefits associated with the frequency of maintenance cycles and content changes.

  • Are there defined processes in place to effectively manage changes to the standard?
  • Are there processes in place to manage and resolve stakeholder conflicts related to change processes?
  • Are the change processes responsive to stakeholder needs and feedback?
  • Is the frequency of updates sufficiently short to accommodate the addition of new codes and repairs quickly?
  • Is the versioning process clearly defined, documented and compatible with business requirements?
  • Is the maintenance body responsive to requests for assistance, maintenance, etc.? 

 

Quality Criteria

Standard quality criteria relate to the overall calibre of the standard not the standard content itself. They measure the level of interoperability, adaptability, and stability provided by the standard as well as the availability of supporting resources such as training, implementation guides, and software to support development, conformance testing, and maintenance of the standard.

Implementation Support and Education

Services which support implementers may help increase the rate of adoption and/or decrease the implementation and ownership costs for projects who would otherwise have to provide support services to adopters directly.  These services may be provided by the custodian or service providers for a fee or free of charge.

  • Does the standard’s custodian provide direct support to implementers?  Do others?  At what cost?
  • Is it easy to obtain education and/or training on the standard and supporting tools?  
  • Is the education or training offered in multiple formats (online training modules, books, in person, etc.)?

Enables Interoperability

Existing and new clinical solutions will need to exchange information in a seamless manner.  Standards which are complementary to – or strategically positioned to work with – existing interoperability standards implementations may offer greater flexibility or long term benefit.

(Note that greater flexibility can also increase solution complexity.  It is important to consider how different components of an integrated solution will interoperate with one another and requirements for supporting services when evaluating this criterion.)

  • Is the standard backwards compatible? (E.g. can implementers of previous versions keep their applications in tact in order to be compatible with a newer version?)
  • Does the standard have the ability to map to other terminology and classification standards?

Implementation and Maintenance Tools

The existence of tools and code libraries that are easily accessible or commercially available can reduce the effort and cost required to implement and maintain a particular standard.

  • Are there any existing code libraries and examples available to support implementers or would it be necessary to write all base level code from scratch?
  • Are there standard design tools to help implementers extend or constrain the standard? Are the standard design tools stable and usable?
  • Do the tools run on different operating systems? 
  • Are there tools that help implementers easily recognize differences between versions of the standard and/or localizations?
  • Are there any application program interfaces or development sandboxes for implementers?

Conformance Testing Methodologies and Tools

Conformance criteria and profiles are used to test whether a standard has been implemented correctly.  Process to measure solution conformance may already be in place and available to support the project. 

  • Are there conformance testing tools that first time implementers can easily access?
  • Can implementers re-use their conformance testing environments and processes?
  • Is there a vendor certification process in place?

Stability

Standards which are stable and widely implemented are more attractive for other implementers to adopt.  Less mature standards may be more volatile and fragile. 

  • Has the standard been implemented and tested previously?
  • Has the standard already been implemented by the project’s implementers?
  • Is the standard stable or is it in a draft status and subject to change?

Adaptability

Different standards allow implementers different levels of flexibility to make modifications to support local needs.  

(Note that greater flexibility can also increase solution complexity.  It is important to consider how different components of an integrated solution will interoperate with one another and requirements for supporting services when evaluating this criterion.)

  • Is the standard highly flexible with lots of optionality and minimal cardinality constraints?
  • Is the standard very strict with little to no optionality and strict cardinality constraints?
  • Does the standard custodian have defined processes and tools for registering local extensions? 
  • No labels