All Academic, Inc. Research Logo

Info/CitationFAQResearchAll Academic Inc.
Document

When to Use Scott’s π or Krippendorff's α, If Ever?
Unformatted Document Text:  When to Use Scott’s π or Krippendorff's α, If Ever? Page 6 of 36 Equation 9 Equation 10 Equation 9 and Equation 10 are equivalent to each other, for the same reason that Equation 2 and Equation 3 are equivalent to each other. The difference is in c a , expected chance agreement. To estimate c a , Scott (1955) multiplied average positive rate (M/N) by itself, and multiplied the average negative rate (W/N) by itself, as shown in Equation 8, which can be also expressed as: Equation 11 By contrast, Krippendorff (1980) subtracted 1 from the two multipliers’ numerators and denominators: Equation 12 Equation 9, Equation 10, and Equation 12 constitute Krippendorff's α for binary scale with two coders. With multiple coders and multi-category nominal scales, Krippendorff's α takes more complicated form (Hayes and Krippendorff, 2007; Krippendorff 2004a, 2004b). While this paper focuses on binary scale with two coders to outline the boundaries of legitimate application of the two indicators, these boundaries should also apply to more categories and more coders. III. Fourteen Paradoxes of π and α While Scott’s π and Krippendorff's α have been recommended and used as general indicators of reliability, their behavior often deviates from what’s expected from such an indicator. Here we list fourteen paradoxes. Paradox 1: High agreement rate, low π and α. Suppose two coders coded 1,000 magazine advertisements for cigarettes in the United States. Their task was to see whether the Surgeon General's warning had been inserted as required by law. Suppose each coder found 999 “yes” and one “no,” with

Authors: Zhao, XinShu.
first   previous   Page 6 of 36   next   last



background image
When to Use Scott’s π or Krippendorff's α, If Ever? 
Page 6 of 36 
Equation 
   
 
 
Equation 
10 
   
 
 
 
Equation 9 and Equation 10 are equivalent to each other, for the same reason that Equation 2 and 
Equation 3 are equivalent to each other. The difference is in c
a
, expected chance agreement.  To estimate 
c
a
, Scott (1955) multiplied average positive rate (M/N) by itself, and multiplied the average negative rate 
(W/N)
 
by itself, as shown in Equation 8, which can be also expressed as: 
Equation 11   
 
 
By contrast, Krippendorff (1980) subtracted 1 from the two multipliers’ numerators and 
denominators: 
Equation 12   
 
 
Equation 9, Equation 10, and Equation 12 constitute Krippendorff's α for binary scale with two 
coders.  With multiple coders and multi-category nominal scales, Krippendorff's α takes more 
complicated form (Hayes and Krippendorff, 2007; Krippendorff 2004a, 2004b).  While this paper focuses 
on binary scale with two coders to outline the boundaries of legitimate application of the two indicators, 
these boundaries should also apply to more categories and more coders. 
 
III. Fourteen Paradoxes of π and α 
 
While Scott’s π and Krippendorff's α have been recommended and used as general indicators of 
reliability, their behavior often deviates from what’s expected from such an indicator. Here we list 
fourteen paradoxes. 
Paradox 1High agreement rate, low π and α. Suppose two coders coded 1,000 magazine 
advertisements for cigarettes in the United States. Their task was to see whether the Surgeon General's 
warning had been inserted as required by law. Suppose each coder found 999 “yes” and one “no,” with 


Convention
Convention is an application service for managing large or small academic conferences, annual meetings, and other types of events!
Submission - Custom fields, multiple submission types, tracks, audio visual, multiple upload formats, automatic conversion to pdf.
Review - Peer Review, Bulk reviewer assignment, bulk emails, ranking, z-score statistics, and multiple worksheets!
Reports - Many standard and custom reports generated while you wait. Print programs with participant indexes, event grids, and more!
Scheduling - Flexible and convenient grid scheduling within rooms and buildings. Conflict checking and advanced filtering.
Communication - Bulk email tools to help your administrators send reminders and responses. Use form letters, a message center, and much more!
Management - Search tools, duplicate people management, editing tools, submission transfers, many tools to manage a variety of conference management headaches!
Click here for more information.

first   previous   Page 6 of 36   next   last

©2012 All Academic, Inc.