All Academic, Inc. Research Logo

Info/CitationFAQResearchAll Academic Inc.
Document

When to Use Scott’s π or Krippendorff's α, If Ever?
Unformatted Document Text:  When to Use Scott’s π or Krippendorff's α, If Ever? Page 25 of 36 indicators of reliabilities, nor would we have had the paradoxes. We would have many hypothetical chance agreements to be compared with observed agreement. Because there is an unlimited number of ways we could practice “random,” (at least the possible distribution, which is a continuum, is unlimited), there could be an infinite number of chance agreements as references for comparison. The methodologists did not do that. Instead, they recommended equations like Equation 2, Equation 3, Equation 9 or Equation 10 to “remove” or “correct for” chance agreement. Each of them chose a different scenario of maximum random, yet each believed he was recommending a or the general indicator of reliability. This created a gap between the conceptual understanding, which sees maximum random as hypothetical, and the actual computation, which treats maximum random as real. This gap is a major source of our confusions and paradoxes. We should close this gap by stop using π or α as a general indicator of reliability. VIII. Conclusion The eight assumptions and implications can resolve the 18 paradoxes, supporting our speculation that π or α does not measure general reliability. Scott's π measures reliability under special conditions when coders (1a) enforce conspired quota, (1b) exercise maximum random with replacement, and (1c) practice limited honesty. These assumptions imply that coders (1d) perform constrained task, (1e) require fixed distribution, (1f) entail variable benchmark, and (1g) conduct specified random. Krippendorff’s α measures reliability under special conditions when coders (2a) enforce conspired quota, (2b) exercise maximum random without replacement, and (2c) practice limited honesty. These assumptions imply that coders (2d) perform constrained task, (2e) require fixed distribution, (2f) entail variable benchmark, (2g) conduct specified random, and (2h) fix the number of cases. The main differences between π and α are in Conditions (b) and (h) — π assumes replacement and no fixed sample, while α assumes fixed sample and no replacement. Both differ from Cohen’s κ in one way: π and α assume a collective quota, while κ assumes individual quotas.

Authors: Zhao, XinShu.
first   previous   Page 25 of 36   next   last



background image
When to Use Scott’s π or Krippendorff's α, If Ever? 
Page 25 of 36 
indicators of reliabilities, nor would we have had the paradoxes.  We would have many hypothetical 
chance agreements to be compared with observed agreement.  Because there is an unlimited number of 
ways we could practice “random,” (at least the possible distribution, which is a continuum, is unlimited), 
there could be an infinite number of chance agreements as references for comparison. 
 
The methodologists did not do that.  Instead, they recommended equations like Equation 2, 
Equation 3, Equation 9 or Equation 10 to “remove” or “correct for” chance agreement.  Each of them 
chose a different scenario of maximum random, yet each believed he was recommending a or the general 
indicator of reliability. This created a gap between the conceptual understanding, which sees maximum 
random as hypothetical, and the actual computation, which treats maximum random as real.   This gap is a 
major source of our confusions and paradoxes. We should close this gap by stop using π or α as a general 
indicator of reliability.  
 
VIII. Conclusion 
 
 
The eight assumptions and implications can resolve the 18 paradoxes, supporting our speculation 
that π or α does not measure general reliability.  
 Scott's 
π measures reliability under special conditions when coders (1a) enforce conspired quota, 
(1b) exercise maximum random with replacement, and (1c) practice limited honesty. These assumptions 
imply that coders (1d) perform constrained task, (1e) require fixed distribution, (1f) entail variable 
benchmark, and (1g) conduct specified random.  
 Krippendorff’s 
α measures reliability under special conditions when coders (2a) enforce conspired 
quota, (2b) exercise maximum random without replacement, and (2c) practice limited honesty. These 
assumptions imply that coders (2d) perform constrained task, (2e) require fixed distribution, (2f) entail 
variable benchmark, (2g) conduct specified random, and (2h) fix the number of cases. 
  
The main differences between π and α are in Conditions (b) and (h) — π assumes replacement and 
no fixed sample, while α assumes fixed sample and no replacement.  Both differ from Cohen’s κ in one 
way: π and α assume a collective quota, while κ assumes individual quotas. 


Convention
Need a solution for abstract management? All Academic can help! Contact us today to find out how our system can help your annual meeting.
Submission - Custom fields, multiple submission types, tracks, audio visual, multiple upload formats, automatic conversion to pdf.
Review - Peer Review, Bulk reviewer assignment, bulk emails, ranking, z-score statistics, and multiple worksheets!
Reports - Many standard and custom reports generated while you wait. Print programs with participant indexes, event grids, and more!
Scheduling - Flexible and convenient grid scheduling within rooms and buildings. Conflict checking and advanced filtering.
Communication - Bulk email tools to help your administrators send reminders and responses. Use form letters, a message center, and much more!
Management - Search tools, duplicate people management, editing tools, submission transfers, many tools to manage a variety of conference management headaches!
Click here for more information.

first   previous   Page 25 of 36   next   last

©2012 All Academic, Inc.