The timely completion of objective assessment tools for evaluation of technical skills

• Published in: The Laryngoscope Vol. 122; no. 11; pp. 2418 - 2421
• Main Authors: Laeeq, Kulsoom, Francis, Howard W., Varela, David A. Diaz Voss, Malik, Mohammad Umair, Cummings, Charles W., Bhatti, Nasir I.
• Format: Journal Article
• Language: English
• Published: Hoboken Wiley Subscription Services, Inc., A Wiley Company 01-11-2012; Wiley-Blackwell
• Subjects: Assessment; Biological and medical sciences; Checklist; Clinical Competence; Cross-Sectional Studies; Education, Medical, Graduate; Educational Measurement - methods; faculty buy-in; faculty development; formative feedback; Humans; Internship and Residency - standards; Level of Evidence: 1b; Medical sciences; objective assessment of surgical skills; Otolaryngology - education; Otolaryngology - standards; Otorhinolaryngology. Stomatology; Psychometrics; surgical skills; Task Performance and Analysis; Evaluation; formative feedback; Treatment; Feedback; Surgery; Assessment; objective assessment of surgical skills; ENT; faculty buy-in; surgical skills; faculty development
• ISSN: 0023-852X; 1531-4995
• DOI: 10.1002/lary.23494

Objectives/Hypothesis:
To this date the effect of the time taken to complete an evaluation on the psychometric properties of the instrument has not been reported. The goal of our study was to assess the effect of time taken to complete an evaluation on its validity.
Study Design:
Cross‐sectional validation study.
Methods:
The global and checklist parts of tonsillectomy, mastoidectomy, rigid bronchoscopy, and endoscopic sinus surgery were used in the operating room by the otolaryngology faculty to evaluate the surgical skills of the residents. We categorized evaluations into two groups depending on the time taken to complete an evaluation (group A ≤6 days, group B >6 days). Construct validity was calculated for both groups by comparing the mean global and checklist scores of the residents across advancing postgraduate year levels.
Results:
A total of 468 evaluations, consisting of global and checklist parts, were completed for 29 residents by 32 evaluators. Mean number of days taken to complete an evaluation was 7.7 days. For all the evaluations completed within a 6‐day time period, the construct validity was significant for both global and checklist parts of the four instruments. In cases of the evaluations completed after 6 days, the construct validity was significant for the tonsillectomy instrument only.
Conclusions:
Our results indicate that the time taken to complete an evaluation has a significant effect on the construct validity of the objective instrument. In the future, efforts should be focused on faculty development to ensure timely completion of the evaluation for a more valid assessment process. Laryngoscope, 2012