All Academic, Inc. Research Logo

Info/CitationFAQResearchAll Academic Inc.
Document

When to Use Scott’s π or Krippendorff's α, If Ever?
Unformatted Document Text:  When to Use Scott’s π or Krippendorff's α, If Ever? Page 19 of 36 random drawing and coding, and report the remaining objects in such a way that both averages meet the quota (Assumption 1). If neither average has reached the quota, they go to Step 2g. 2g. The coders repeat Step 2b and the subsequent steps until all objects are "coded." The three assumptions have four implications for both π and α. Implication 1: Constrained task. Assumptions 1 and 3 imply that the purpose of a study is not to find out how many objects are in one category or another, which has been pre-decided by the quota. The only purpose is to place objects into appropriate categories under the quota. Implication 2: Fixed distributions. These assumptions also imply that the observed distribution do not change within a study when the coders improve their work, as their “work” is not to assess distribution between categories, but only to assign objects to categories according to the quotas. Implication 3: Variable benchmark. Assumptions 1 and 2 imply that chance agreement c a is a function of the marble distributions predetermined by the quota. When almost all marbles are of the same color, the coders have close to 100% probability agreeing by chance, and close to 0% opportunities to code honestly. Hence the bar, c a , should be set close to 100%. At the higher extreme, when the quota sets all marbles to the same color, the coders have no chance to do any honest coding, therefore c a =100%, and π and α are both undefined. Although a r would be 100%, all of it is deemed due to chance. The lower extreme is different between π and α. For π, when marbles are half black and half white, the coders have 50% chance agreeing by chance and 50% chance of coding honestly, therefore the bar is set at 50%, which is the lowest c a under π. Table 1a and Figures 1a, 2 & 3 display the entirety of the changing benchmark for π from various angles in various details. For α, the lower extreme varies according to the number of cases (N) coded. When N=2, Krippendorff’s c a is either 0% (when distribution is 50&50%) or 100% (when distribution is 0&100%). As N increases from 2 to approach infinity, this lower end of the bar increases from 0%; to approach 50%. Figure 1b, 2, 3, & 4 and Tables 1b & 2 display the changing pattern from different angles with various details.

Authors: Zhao, XinShu.
first   previous   Page 19 of 36   next   last



background image
When to Use Scott’s π or Krippendorff's α, If Ever? 
Page 19 of 36 
random drawing and coding, and report the remaining objects in such a way that both averages 
meet the quota (Assumption 1). If neither average has reached the quota, they go to Step 2g.  
2g. The coders repeat Step 2b and the subsequent steps until all objects are "coded." 
 
The three assumptions have four implications for both π and α. 
 
Implication 1Constrained task. Assumptions 1 and 3 imply that the purpose of a study is not to 
find out how many objects are in one category or another, which has been pre-decided by the quota.  The 
only purpose is to place objects into appropriate categories under the quota. 
Implication 2Fixed distributions. These assumptions also imply that the observed  distribution 
do not change within a study when the coders improve their work, as their “work” is not to assess 
distribution between categories, but only to assign objects to categories according to the quotas. 
Implication 3Variable benchmark. Assumptions 1 and 2 imply that chance agreement c
a
 is a 
function of the marble distributions predetermined by the quota. When almost all marbles are of the same 
color, the coders have close to 100% probability agreeing by chance, and close to 0% opportunities to 
code honestly.  Hence the bar, c
a
, should be set close to 100%.   At the higher extreme, when the quota 
sets all marbles to the same color, the coders have no chance to do any honest coding, therefore c
a
=100%, 
and π and α are both undefined. Although a
r
 would be 100%, all of it is deemed due to chance.   
The lower extreme is different between π and α.  For π, when marbles are half black and half 
white, the coders have 50% chance agreeing by chance and 50% chance of coding honestly, therefore the 
bar is set at 50%, which is the lowest c
a
 under π. Table 1a and Figures 1a, 2 & 3 display the entirety of the 
changing benchmark for π from various angles in various details.  
For α, the lower extreme varies according to the number of cases (N) coded.  When N=2, 
Krippendorff’s c
a
 is either 0% (when distribution is 50&50%) or 100% (when distribution is 0&100%).  
As increases from 2 to approach infinity, this lower end of the bar increases from 0%; to approach 50%.  
Figure 1b, 2, 3, & 4 and Tables 1b & 2 display the changing pattern from different angles with various 
details.   


Convention
All Academic Convention is the premier solution for your association's abstract management solutions needs.
Submission - Custom fields, multiple submission types, tracks, audio visual, multiple upload formats, automatic conversion to pdf.
Review - Peer Review, Bulk reviewer assignment, bulk emails, ranking, z-score statistics, and multiple worksheets!
Reports - Many standard and custom reports generated while you wait. Print programs with participant indexes, event grids, and more!
Scheduling - Flexible and convenient grid scheduling within rooms and buildings. Conflict checking and advanced filtering.
Communication - Bulk email tools to help your administrators send reminders and responses. Use form letters, a message center, and much more!
Management - Search tools, duplicate people management, editing tools, submission transfers, many tools to manage a variety of conference management headaches!
Click here for more information.

first   previous   Page 19 of 36   next   last

©2012 All Academic, Inc.