Original article
Clinical endoscopy
The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists

https://doi.org/10.1016/j.gie.2011.09.053Get rights and content

Background

Practitioners increasingly need to be able to evidence the quality of their care and their clinical competence for purposes of recredentialing and relicensing. Although this may be accomplished by audit and performance data, detailed and robust assessments of competence may be valuable in certain circumstances.

Objective

To develop and evaluate a detailed assessment of performance of colonoscopy.

Design

Evaluation of a Direct Observation of Procedural Skills (DOPS) method developed by an expert group of colonoscopists and clinical educationalists.

Setting

English National Health Service National Bowel Cancer Screening Programme (BCSP).

Subjects and Methods

Aspirant colonoscopists wishing to participate in the BCSP were assessed by using the DOPS. Reliability was estimated by using generalizability theory (G), and the candidates' and assessors' perspectives on validity were evaluated by questionnaire.

Interventions

Grading of performance by 2 assessors over 2 consecutive real cases.

Main Outcome Measurements

DOPS grades, global expert evaluation, performance data, evaluation questionnaire scores.

Results

The assessment had high relative reliability: G = 0.81. The DOPS grades correlated highly with a global expert assessment. The candidates and assessors believed that the DOPS was a valid assessment of competence.

Limitations

Not guaranteed to assess therapeutic skills; evaluation questionnaire influenced by result of assessment.

Conclusions

This is the first evaluation of a DOPS assessment on independent practitioners. It performs well, with good levels of reliability and validity, and is sufficient to be used in a high-stakes assessment. Similar approaches should be considered for assessment of competence in other areas of clinical practice for relicensing or recredentialing.

Section snippets

Background

Several tools focusing on assessment of complex, integrated aspects of clinical practice in the workplace have been developed. These workplace-based assessments (WBAs) have facilitated a move away from the use of numbers-based experience and a broad overview of capability toward a more structured10 and evidenced approach based on competence. Centered on a framework of scoring over a finite number of items, they have been used in the UK Foundation doctors' program and have been trialed in senior

Development of the assessment

An expert group composed of physician, surgeon, and nurse colonoscopists and clinical educationalists was convened. They comprised the national and regional endoscopy training center leads, the chair of the JAG, and the education advisor to the National Endoscopy Program. All had a track record of high-quality endoscopy and experience in training trainees and practitioners in endoscopy, and some had formal educational qualifications. They developed the assessment via a series of iterations.

Assessments

One hundred ninety-three assessments were undertaken by 162 candidates and 28 assessors. Of those, 31 were repeated assessments and were excluded from analysis, as were 14 candidates not eligible for inclusion for the correlations because their polyp detection rates were less than 20%, and 1 candidate was excluded owing to influential outlying data (a lifetime colonoscopy number of over 25,000), leaving 147 candidates. Descriptive statistics for these variables are shown in Table 2.

DOPS validity: questionnaire data

There were

Discussion

The assessment appears to be reliable, valid, and useful for assessing skills in colonoscopy. What are the wider implications? The DOPS can be used in clinical practice to evaluate the skills of trainees and independent practitioners at colonoscopy. Detailed assessment of attending physicians or consultants is likely to meet significant resistance for a variety of reasons. In some circumstances, it could be very valuable: In confirming attainment of competence in trainees, in the assessment of

Conclusion

This is the first specific evaluation of an endoscopic DOPS assessment. It performs well, with good levels of reliability and validity, sufficient to be used in a high-stakes assessment. This approach could be adapted as a quality management process in other specialties.

Acknowledgments

The authors thank all the candidates, the assessors, the Accreditation Panel, and the BCSP for assisting with the development of this process. Particular thanks to the National Endoscopy Programme and the training centre leads for developing the DOPS assessment. Special thanks to Linda Beard for administrative support, Nicola Gardiner for data handling, and Dr Ron Hoogenboom for the statistical analysis.

This study was supported by the English NHS Bowel Cancer Screening Programme.

References (25)

  • D.J. Bjorkman et al.

    Measuring the quality of endoscopy

    Gastrointest Endosc

    (2006)
  • Trainee certification

  • T.C. Wang et al.

    AGA Institute Future Trends Committee report: the future of gastroenterology training programs in the United States

    Gastroenterology

    (2008)
  • Renewal of and proctoring for endoscopic privileges

  • Trust, Assurance and SafetyThe regulation of health professionals in the 21st century

    (2007)
  • C.J. Bowles et al.

    A prospective study of colonoscopy practice in the UK today: are we adequately prepared for national colorectal cancer screening tomorrow?

    Gut

    (2004)
  • Scoping our practiceThe 2004 Report of the National Confidential Enquiry into Patient Outcome and Death

    (2004)
  • P.L. Stillman et al.

    Ensuring the clinical competence of medical school graduates through standardized patients

    Arch Intern Med

    (1987)
  • G. Holsgrove

    Principles of assessment

  • G.L. Noel et al.

    How well do internal medicine faculty members evaluate the clinical skills of residents?

    Ann Intern Med

    (1992)
  • J.R. Wilkinson et al.

    Implementing workplace-based assessment across the medical specialties in the United Kingdom

    Med Educ

    (2008)
  • J.J. Norcini et al.

    The mini-CEX: a method for assessing clinical skills

    Ann Intern Med

    (2003)
  • Cited by (108)

    • Ensuring Quality in Endoscopic Training: Tools for the Educator and Trainee

      2022, Techniques and Innovations in Gastrointestinal Endoscopy
    View all citing articles on Scopus

    DISCLOSURE: All authors disclosed no financial relationships relevant to this publication.

    If you would like to chat with an author of this article, you may contact Dr Barton at [email protected].

    View full text