General Standards Screening Criteria

Criteria

Description

Rationale

Alignment to the eHealth Blueprint

The purpose of this criterion is to assess whether the standard supports the exchange of information between two or more components of the EHR. The standard should also be intended to be used in a federated approach to EHR architecture where multiple organizations are expected to build, operate and/or host the various applications.

The standard supports eHealth mandate in providing a single, harmonized, coherent province-wide EHR.

Existing Standard [RB2] 

This criterion assesses whether the standard is adopted/adapted from existing standards, and whether it is intended for international, pan- Canadian.

Re-use of existing standards allows implementers to re-use code (i.e. in point of service /Health Information Access Layer (HIAL) applications, conformance testing environments, etc.). It also allows reuse of existing tools/skills/knowledge sets to establish the standard.

Supports Business Requirements

This criterion assesses whether the standard expresses all information required by the business/clinical domain. While the exact business requirements are project specific, example questions may include:

• Does the messaging standard have all the messages to support the required functions for data exchange? 
• Do the messages have the appropriate fields to express the information required for the business functions?
• Does the document standard have all the sections to express the required business information?
• Does the terminology contain all of the concepts that need to be captured?
• If bi-lingual information needs to be exchanged, does the standard allow for English and French terms to be mapped to common codes? 
• Are there user interface data collection/display requirements that should be considered when choosing the interoperability standard?

To ensure that the standard addresses the health care business requirements.

The standard should be health care delivery setting independent to enable use across multiple health care delivery settings.

Supports Technical Requirements

This criterion assesses the degree to which the standard is feasible to adopt and implement according to the technical requirements of the project. While the exact technical requirements are project specific, example questions may include:

• Can the standard be implemented in the proposed architecture? 
• Does the conceptual architecture make any assumptions about synchronous or asynchronous communication? 
• Does the architecture support HTTP and/or MLLP? 
• Are there any interdependencies with other aspects of the architecture that would make it difficult to implement the standard (e.g. there are no places to express SAML bindings in HTTP without SOAP, making it difficult to integrate RESTful interfaces with the existing security system and therefore difficult to implement HL7 FHIR)?

To enable successful implementation of the standard.

Likely to be Adopted[Office3] 

This criterion assesses the likelihood that the selected standard will be adopted and maintained by vendors and other implementers. Some questions that may be asked when assessing the likelihood for adoption, include: 

• Have vendors already implemented this? If they haven’t, will they get a return on investment if they do?
• Do vendors have the necessary expertise to implement the standard? If not, is it realistic that they can gain that expertise in time to implement?
• Will vendors provide support for sustained use and maintenance?

To ensure success of the implementation. If the standard is not widely implemented or supported by vendors, it will result in increased time, cost, and effort. 

Supports coded and free text content.

This criterion assesses the degree to which coded data is required.

Some questions to ask when assessing whether there is a need for coded data versus free text, may include: 

• Does the project implementing the standard expect to support automated processing such as: data aggregation comparison within decision support or analytics applications, standardization of data used to trigger process flows (i.e. presence or absence of terms), etc[Office4] .[Office5] 
• Do clinicians require assistance to exchange data/information for human readability?
• Are there other forms of non-textual data/information (i.e. audio, video, images) that need to be exchanged?
• Are there privacy concerns with exchanging free text?

Different standards support different levels of data granularity and coding.  However, there can be trade-offs to application developers and users to provide granular and coded data.  At one extreme, end users may reject using the application because it forces them to enter data in too many fields, increasing their time/effort to do so. At the other extreme, just capturing free text can cause privacy breaches and be a barrier to implementing decision support.

Criteria

Description

Rationale

Costs and Benefits of Implementation

This criterion assesses whether the cost of implementation and disruption to current business is affordable given the benefits. It is important to consider not only the initial cost but the on-going cost for maintenance as well. Cost is not only measured in currency, but can be in time or effort as well. Some sample questions to ask when assessing the costs and benefits of implementation, include:

• What is the return on investment for the vendor?
• What is the cost of tools required?
• Will the implementation timelines be increased due to the complexity of the standard?
• Will cost or risk of subsequent system implementations be reduced?[Office1] 
• What is the number of people required for implementation?
• Is there any existing commercial off the shelf software vs. custom software?
• If properly implemented, would the standard provide system or societal benefits such as improved decision support or clinical research? 

To take into consideration any cost constraints when adopting a standard, as well as initial implementation and longer-term maintenance costs. 

Governance Structure

The purpose of this criterion is to assess whether or not the standard is governed appropriately.  When assessing if a governance structure is in place, it is important to consider whether there are: 

• Defined processes in place to facilitate decision making and issue resolution related to both standards content and processes.
• Balanced representation of communities responsible for and effected by the standard.
• Processes for adding and removing members on the governance committees. [Office2] 

To ensure proper processes are in place for the voting, maintenance and changing of the standard amongst its various stakeholders; as well as certification and other capabilities of the standards development or maintenance organization. 

Intellectual Property and Licensing Costs

This criterion assesses whether the intellectual property and licensing terms for the standard allows the wide use of the standard. Example questions may include:

• Does the standard have licensing costs that are so expensive that the costs prohibits uptake of the standard?
• Are there network benefits or economies of scale to be realized due to previous implementations of the standard?[Office3] 
• Is it likely the standard licensing costs will increase over time?
• If a standard is currently free, are there other hidden conditions?
• Does the standard have licensing restrictions that prevent it from being implemented by anyone?
• What standards are being implemented in neighbouring regions/countries due to differing licensing costs? Will these impact expectations in our country?

To ensure that the standard is vendor neutral and allows for the wide use of the standard. Openness and intellectual property policies affect the status, uptake and implementability of specifications, especially in the long term. 

Defined Maintenance Process

This criterion assesses whether or not the standard custodian has processes in place to accept requests for change to address errors or new needs in the standard. When assessing if a maintenance process is in place, it is important to consider the following:

• There are defined processes in place for changes to the standard?[Office4] 
• Are processes in place to manage and resolve conflicts related to change processes?
• The change processes are responsive to stakeholder needs.
• The frequency of updates should be sufficiently short to accommodate new codes and repairs quickly, as well as extensions or temporary (?hot?) fixes to resolve immediate needs in between full updates.
• Is the versioning process clearly defined, documented and compatible with business requirements?
• Is the maintenance body responsive to requests for assistance, maintenance, etc.?

Business requirements and technology change over time, as such, standards need to evolve to meet changing needs.  It is therefore important to ensure the selected standard has defined maintenance processes in order to evolve.

Criteria

Description

Rationale

Provides Implementation Support and Education[Office2] 

To assess the degree to which resources are able to assist with appropriate use and implementation of the standard. Some key questions to consider when assessing this criteria are:

• Does the standards development/maintenance organization (e.g. HL7, IHE, Infoway, etc.) provide support for implementation?
• Is it easy to obtain education and/or training on the standard and supporting tools?  
• Is the education or training offered in multiple formats (online training modules, books, in person, etc.)?

If there are organizations available to provide supporting services to help implement, it will help increase the number of adopters and decrease the costs for specific implementers who would otherwise have to provide those services. 

Enables Interoperability

The purpose of this criterion is to assess whether the standard is complementary to or strategically positioned to work with existing interoperability standards implementations. When assessing this criterion, the following should be considered:

• Is the standard backwards compatible? [Office3] (E.g. can implementers of previous versions keep their applications in tact in order to be compatible with a newer version?)
• Does the standard have the ability to map to other terminology and classification standards?
• Does the standard facilitate information exchange? 

Note: Consider how components of an integrated solution interact as well as any enabling services when assessing the costs, benefits and risks of the items above[Office4] .

To enable existing and new clinical solutions and provide the capability of exchanging information in a seamless manner. To ensure that previous data and legacy systems are compatible with the standard and can be upgraded if necessary.

Implementation and Maintenance Tools

This criterion assesses the availability of tools, code libraries, and/or COTS products to support implementation and maintenance of the standard. The following are questions that may be asked:

• Are there any existing code libraries available or would it be necessary to write the base level code from scratch?
• Are there standard design tools that help implementers to extend or constrain the standard?
• Are the standard design tools stable?
• Do the tools run on different operating systems? 
• Are there tools that help implementers compare 2 or more versions of the standard or localization?
• Are there any application program interfaces or development sandboxes for implementers?

The existence of tools that are easily accessible decreases the effort and cost to implement and maintain a particular standard.

Conformance Testing Methodologies and Tools

This criterion assesses if the standard has well defined conformance testing processes and supporting tools.  Examples questions may include:

• Are there conformance testing tools that first time implementers can easily access?
• Can implementers re-use their conformance testing environments and processes?

In order to test whether a standard has been implemented correctly, a set of conformance criteria and profiles against which a standard can be measured is important. Ideally, a process to measure conformance should be in place to publicly notify those vendors that are certified (meet testing criteria). 

Proven Stability [Office5] 

This criterion assesses the stability and level of adoption of the standard. Some key questions to consider are:

• Has the standard been implemented and tested previously?
• Is the standard already implemented by the project‘s implementers?
• Is the standard stable or still in draft?

If a standard is stable and widely implemented it is more attractive for other implementers to adopt it.

If a standard is still in inception, it is more volatile and fragile, which will likely lead to increased risks (e.g. increased implementation costs and timeline delays) for initial implementers.

Adaptable and customizable [Office6] 

This criterion assesses the extent to which the standard allows implementers to make modifications to meet local requirements. Example questions may include:

• Is the standard highly flexible with lots of optionality and minimal cardinality constraints?
• Is the standard very strict with little to no optionality and strict cardinality constraints?
• Does the standard custodian have defined processes and tools for registering local extensions?

For every interoperability standard, it is essential to describe how tightly the aspects defined in the standard constrain the options for the implementation, and which aspects are left for the implementation-specific or customer-specific conventions.