Validation and verification of examination procedures in medical laboratories: opinion of the EFLM Working Group Accreditation and ISO/CEN standards (WG-A/ISO) on dealing with ISO 15189:2012 demands for method verification and validation

Introduction

The scope of this paper is to provide guidance for drawing up local/national documents about validation and verification. In essence, this is of course described in ISO 15189:2012 [1], which was developed as a standard to serve as a fundamental tenet for the quality management systems in medical laboratories. The standard is mandatory on which procedures have to be in place and what aspects have to be covered by these procedures, but deliberately not on the specific way that these aspects have to be covered. The standard relies on risk management by laboratory professionals to ensure that the local demands are met under local circumstances. For validation and verification of examination procedures, this means that laboratory professionals use their professional assessment to judge which performance characteristics need what evaluation to make sure that the intended quality can be delivered. This opinion paper provides a structured approach for such assessment. It guides how to meet the requirements of the standard without becoming too prescriptive, which would limit the professional judgment. Although the differences between validation and verification were clarified in the revised version of 2012 (5.5.1.1-5.5.1.3), questions remain and differences in their interpretation between countries are quite large [2].

In some countries, for example France, an explanation of ISO 15189:2012 is issued by the National Accreditation Body, accompanied by recommended templates on how to perform a validation or verification [3], [4]. Often, supplementary demands are made on top of the requirements from ISO 15189:2012, thereby restricting the freedom that was originally offered. Additionally, some laboratories have published their own practice to fulfil the requirements as a useful example to others [5].

We propose to apply risk consideration (in other words, professional evaluation) to verification/validation procedures to find the right balance between regulatory requirements on one side and professional autonomy on the other side, both within the intentions and requirements of the standard. Practically, this means that we will not define numbers of samples required for verification and validation experiments or prescribe the performance of certain experiments at all. We recommend careful consideration in light of the intended use of the examination procedure, taking into account all available information. By documenting the line of thought, choices become traceable. This proposal has its roots in the Dutch guideline on verification and validation, as adopted by the national scientific society [6]. We describe in a point-by-point approach the subsequent steps needed to fulfil the requirements of the standard with respect to verification and validation. This sequence starts with the selection of an examination procedure and ends with its approval for implementation when fit for purpose. Closely related subjects like intended use, metrological traceability (ISO 15189:2012 5.3.1.4), and internal (ISO 15189:2012 5.6.2.3) and external quality control (ISO 15189:2012 5.6.3.4) are addressed within the framework of validation and verification.

This paper is primarily intended for examination procedures used for human diagnosis or follow-up in clinical chemistry laboratories. Measuring equipment such as analyzers are not taken into consideration, at least not where installation and technical-operational aspects of such measuring equipment are concerned. The focus essentially is on quantitative measuring of a measurand in a biological matrix; however, it may also (partly) apply to qualitative analysis. Furthermore, this paper may also be useful and relevant for other disciplines in laboratory medicine.

The introduction of a new examination procedure

In compliance with ISO 15189:2012, examination procedures shall be subjected to independent validation or verification by the medical laboratory prior to being introduced [1]. We define validation as the demonstration via objective evidence that a new or modified examination procedure from one’s own working environment (or laboratory) is appropriate for a specific intended use in medical diagnostics, and that it complies with the relevant acceptance criteria as described by the medical laboratory. A verification, by our definition, always concerns an already validated examination procedure, and its appropriateness should be confirmed. Which performance characteristics and acceptance criteria are relevant for validation or verification shall, depending on the intended application, be established in the validation or verification plan in order to objectively assess if the examination procedure is appropriate for the intended use. The “intended use” of a medical examination procedure comprises the clinical condition and/or the indication prompting the examination procedure, as well as the manner in which the examination procedure, including preparation, is to be carried out. The acceptance criteria and the intended use are connected. Changing the intended use for specific situations will possibly alter the acceptance criteria.

Recommendation I

The laboratory shall use only examination procedures that have been validated or verified by the laboratory for the intended use.

Validation or verification

In practice, there is frequent confusion about the choice between validation and verification. This depends on the availability of reliable and valid data on the performance characteristics of the envisioned examination procedure (Figure 1). The data can, for example, be supplied by the diagnostics supplier (CE- or FDA-approved) or be taken from peer-reviewed texts/journals or from validation data from other accredited laboratories.

Figure 1:

Schematic representation of the workflow for the validation and verification of examination procedures.

The laboratory collects reliable and valid data on the performance characteristics and examines the data to ascertain if the acceptance criteria have been met. If so, verification of relevant performance characteristics is sufficient. If the performance characteristics are either not available or do not meet the acceptance criteria, the laboratory shall collect its own data (validation). Therefore, for example, it may occur that for procedures validated elsewhere, verification can be sufficient for precision, trueness and decision limit procedures, but that supplemental validation of the sample’s stability will be necessary.

Validation of examination procedures

ISO 15189:2012 [1] emphatically states a number of categories of examination procedures to be validated. The term “methods” mentioned in ISO 15189:2012 [1] corresponds to the new term, “examination procedures”.

Recommendation II

The laboratory shall validate examination procedures derived from the following sources:

1. non-standard examination procedures

2. laboratory-designed or developed examination procedures

3. validated examination procedures used outside their intended use

4. validated examination procedures subsequently modified

Recommendation II further explained: The non-standard procedure named under (a) is named as such in ISO 15189:2012 [1] but in practice always refers to the procedure here listed under (b) or (c). In (c), a “validated examination procedure” is considered equivalent to the terminology “standard method” named in ISO 15189:2012 [1].

Part of the validation process is the objective establishment (through measurement) of the relevant performance characteristics. In the aforementioned situations, either the performance characteristics are not established according to ISO 15189:2012, or the examination procedure has been modified in such a way that these performance characteristics are, according to ISO 15189:2012, not automatically valid for the modified examination procedure [1]. If an examination procedure is modified after validation or verification, the effect of such a modification shall be taken into consideration and evaluated (documented motivation). A professional evaluation, for example of decreasing the sample volume in a test to make it suitable for pediatric samples, would take into account whether this decrease could change the reaction mixture conditions significantly and as a consequence which performance characteristics might be affected by such a change.

Recommendation III

If an examination procedure is modified, it shall be taken into consideration and documented if there are potential/relevant consequences for the performance characteristics and, if so, what these consequences are.

Recommendation IV

When an examination procedure is modified with respect to the documented motivation, a (supplemental) validation or verification shall be performed.

Verification of examination procedures

The performance characteristics of the examination procedure evaluated during the verification process shall be relevant for the intended use of the results of the examination. If all the relevant performance characteristics are both available and valid for one’s own laboratory, the examination procedure can be subject to verification instead of validation. Here, “valid” means that the data have been obtained under documented similar conditions as those for the intended use. The laboratory shall have information available from the manufacturer/developer of the examination procedure or examination results from reliable independent studies in order to be able to confirm the known performance characteristics of the examination procedure. When more than one device is used to measure the measurand, the correct operation of each individual device shall be verified.

Recommendation V

Examination procedures that, on the basis of available documentation, do not necessarily have to be validated shall at least be verified for the relevant performance characteristics.

Recommendation VI

If evidence from a validation performed elsewhere is incomplete, verification is insufficient and a supplemental validation in one’s own laboratory shall be necessary.

Recommendation VII

When using more than one analyzer for the same measurand, an appropriate verification of each individual device shall be performed applying the appropriate acceptance criteria.

Establishing a validation or a verification plan

For both validation and verification, performance characteristics shall be assessed against the acceptance criteria. These acceptance criteria are pre-established and justified in the validation and verification plans, respectively. Besides the acceptance criteria, the measurand, intended use and the examination procedure shall have to be established; furthermore, the investigators and the staff member with the appropriate authority, based on professional education and responsibility, shall be identified.

Particularly for validation of an (self-developed) examination procedure, it is essential to clearly define the measurand. Describing the measurand requires knowledge of the component or chemical entity (analyte), the matrix and the condition of the analyte, as well as the characteristics of the examination procedure used. This plays a crucial role, for example, in immunochemical examination procedures.

Recommendation VIII

Validation/verification shall take place according to a pre-established validation/verification plan, which is authorized by staff with the appropriate authority, comprising at least the following elements:

1. intended use of the examination procedure

2. documentation of the measurand

3. selection of the relevant performance characteristics

4. acceptance criteria valid for the intended use

5. examination method

6. identity of investigators and competent authorizer(s)

The laboratory shall document the procedure used for the validation and record the results obtained. A staff member with the appropriate authority shall assess validation results and compile the assessment report.

Performance characteristics

The following possible relevant characteristics for validation are proposed in ISO 15189:2012, section 5.5.1.3 (under NOTE): measurement trueness; measurement accuracy; measurement precision, including measurement repeatability and measurement intermediate precision; measurement uncertainty; analytical specificity, including interfering substances; analytical sensitivity, detection limit and quantitation limit; and measuring interval, diagnostic specificity and diagnostic sensitivity of the measurement [1]. Whether or not these performance characteristics have to be examined is left to the judgment of the staff member with appropriate authority; in addition, examination should be statistically sound. Professional evaluation with regard to its intended use is an important element in the selection of the performance characteristics to be considered. Results of preliminary studies of the intended method or specifications claimed by the manufacturer as well as medical views or guidelines concerning acceptance criteria could be used as the starting point and should be documented as such.

In validation and verification, the laboratory shall take into consideration all these performance characteristics. As accuracy is the combination of bias and imprecision, it can be omitted as an individual performance characteristic. As the lack of accuracy is defined as measurement uncertainty, satisfactory bias and imprecision will automatically lead to acceptable measurement uncertainty, which then satisfies ISO 15189:2012 section 5.5.1.4 that requires laboratories to define performance criteria for measurement uncertainty [1]. At the moment, measurement uncertainty is the subject of a separate ISO standard under development [7] and therefore will not be discussed further here.

At least for some performance characteristics, local analytical confirmation is inevitable. Imprecision is subject to local conditions and therefore must be assessed by analysis. Also, comparison to the previous method if applicable must be assessed under local conditions, as such information is essential to inform test requesters about changes to be expected.

Recommendation IX

In verification, the staff member with appropriate authority shall take the following performance characteristics into consideration: measurement precision, measurement trueness, detection limits, stability, reference interval, comparison to previous method if present, medical decision values and interferences. Such consideration should be based on professional evaluation in the light of the intended use. Imprecision and comparison to previous method must be assessed by local experiments.

Recommendation X

In validation, the staff member with appropriate authority shall take into consideration all the performance characteristics listed in ISO 15189:2012 5.5.1.3.

Recommendation XI

Whenever specific performance characteristics are neither applicable nor feasible in view of the nature of the examination procedure or the prevalence of pathology, this should be documented and motivated.

Common models for establishing acceptance criteria for precision and bias can be found in the Milan consensus statement on this topic [8].

Documentation

The documentation associated with a validation or verification consists of a validation/verification plan, the results, including the raw data and a validation/verification report, in which, besides the examination of the acceptance criteria, the implementation is also described.

Recommendation XII

The results obtained shall be documented and stored in a validation/verification report, at least for the period in which the examination procedure is in use.

Recommendation XIII

The results obtained shall be demonstrably evaluated using the acceptance criteria established in the validation/verification plan. The conclusion of the evaluation and whether or not it is appropriate for the intended use shall be established and archived in a validation/verification report. The validation/verification report shall be assessed and authorized by a staff member with the appropriate authority.

Note: the documentation is stored for the duration of use, extended with the (self-established) storage period of all other registrations pertaining to results from the laboratory.

Release, implementation and performance assurance

Discussion and conclusions

With this paper, we aim to offer guidance on validation and verification of examination procedures complying with ISO 15189. The described approach valorizes the competences of the specialist in laboratory medicine to assess the requirements and specifications needed for a new examination procedure in the medical laboratory and to carry out an appropriate verification or validation. The structured approach in professional evaluation and documentation provides a traceable line of thought in which all requirements of this part of the standard are addressed. This approach relies on the skills of laboratory professionals trained and experienced in the issues at hand. It is meant to balance efforts and outcomes, securing accurate results, and preserving resources to be used elsewhere in favor of effective healthcare.

When regulators (national or supranational) impose precise restrictive guidelines, these shall be applied. We would then hope that in the spirit of ISO 15189:2012, such guidelines come into being with the public participation of the scientific organizations and are subject to a continuous PDCA cycle, balancing efforts and outcomes ultimately to adjust regulations and to promote effective healthcare. Obviously, the outcome of vigilance and surveillance efforts, of audits by accrediting bodies, and of self-assessment by professional organizations can be used as input for this cycle.