ASSESSING OBSERVER ACCURACY IN CONTINUOUS RECORDING OF RATE AND DURATION: THREE ALGORITHMS COMPARED

• Published in: Journal of applied behavior analysis Vol. 42; no. 3; pp. 527 - 539
• Main Authors: Mudford, Oliver C., Martin, Neil T., Hui, Jasmine K. Y., Taylor, Sarah Ann
• Format: Journal Article
• Language: English
• Published: Oxford, UK Blackwell Publishing Ltd 2009; Society for the Experimental Analysis of Behavior; The Society for the Experimental Analysis of Behavior
• Subjects: Accuracy; Adult; Adults; Algorithms; Applied Behavior Analysis; Behavior; Behavior Patterns; Biological and medical sciences; Computers; continuous recording; Data Analysis; Data collection; Female; Fundamental and applied biological sciences. Psychology; Handheld computers; Humans; interobserver agreement; Interrater Reliability; Male; Methods; Middle Aged; New Zealand; Numerical Analysis, Computer-Assisted; Observation; observational data; observer accuracy; Observer Variation; Psychological Patterns; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Psychometrics. Statistics. Methodology; recording and measurement; Research Assistants; Resistance (Psychology); Signal Processing, Computer-Assisted; Software; Statistics. Mathematics; Studies; Therapists; Time Factors; Time Factors (Learning); Undergraduate Students; Video Recording - methods; Video Technology; Young Adult; New Zealand; Human; Evaluation; Interrater agreement; Observation; interobserver agreement; continuous recording; observer accuracy; Test reliability; observational data; Algorithm; recording and measurement
• ISSN: 0021-8855; 1938-3703
• DOI: 10.1901/jaba.2009.42-527

The three algorithms most frequently selected by behavior‐analytic researchers to compute interobserver agreement with continuous recording were used to assess the accuracy of data recorded from video samples on handheld computers by 12 observers. Rate and duration of responding were recorded for three samples each. Data files were compared with criterion records to determine observer accuracy. Block‐by‐block and exact agreement algorithms were susceptible to inflated agreement and accuracy estimates at lower rates and durations. The exact agreement method appeared to be overly stringent for recording responding at higher rates (23.5 responses per minute) and for higher relative duration (72% of session). Time‐window analysis appeared to inflate accuracy assessment at relatively high but not at low response rate and duration (4.8 responses per minute and 8% of session, respectively).