interpreting kappa in observational research baserate matters.pptVIP

Interpreting Kappa in Observational Research: Baserate Matters Cornelia Taylor Bruckner Vanderbilt University Acknowledgements Paul Yoder Craig Kennedy Niels Waller Andrew Tomarken MRDD training grant KC Quant core Overview Agreement is a proxy for accuracy Agreement statistics 101 Chance agreement Agreement matrix Baserate Kappa and baserate, a paradox Estimating accuracy from kappa Applied example Framing as observational coding I will be framing the talk today within observational measurement but the concepts apply to many other situations e.g., Agreement between clinicians on diagnosis Agreement between reporters on child symptoms (e.g. mothers and fathers) “Rater accuracy”: A fictitious session Madeline Scientist writes a script for an interval coded observation session where the Presence or absence of target behavior in interval Two coders (Eager Beaver and Slack Jack), blind to the script, are asked to code the session. Accuracy of each coder with the script is calculated Accuracy of Eager Beaver (EB) with session (interval data) Who has the best accuracy? Eager Beaver of course. Slack Jack was not very accurate Notice that accuracy is about agreement with the occurrence and nonoccurrence of behavior. We don’t always know the truth It is great when we know the true occurrence and nonoccurrence of behaviors But, in the real world we deal with agreement between fallible observers Agreement between raters Point by point interobserver agreement is achieved when independent observers : see the same thing (behavior, event) at the same time Difference between agreement and accuracy Agreement can be directly measured. Accuracy can not be directly measured. We don’t know the “truth” of a session. However, agreement is used as a proxy for accuracy Accuracy can be estimated from agreement The method for this estimation is the focus of today’s talk Percent agreement Percent agreement The proportion of intervals that were agreed upon Agreements/agreemen