When to Use Scott’s π or Krippendorff's α, If Ever?

Page **19** of **36**

random drawing and coding, and report the remaining objects in such a way that both averages

meet the quota (Assumption 1). If neither average has reached the quota, they go to Step 2g.

2g. The coders repeat Step 2b and the subsequent steps until all objects are "coded."

The three assumptions have four implications for both π and α.

**Implication 1**: *Constrained task.* Assumptions 1 and 3 imply that the purpose of a study is not to

find out how many objects are in one category or another, which has been pre-decided by the quota. The

only purpose is to place objects into appropriate categories under the quota.

**Implication 2**: *Fixed distributions.* These assumptions also imply that the observed distribution

do not change within a study when the coders improve their work, as their “work” is not to assess

distribution between categories, but only to assign objects to categories according to the quotas.

**Implication 3**: *Variable benchmark.* Assumptions 1 and 2 imply that chance agreement *c*

*a*

is a

function of the marble distributions predetermined by the quota. When almost all marbles are of the same

color, the coders have close to 100% probability agreeing by chance, and close to 0% opportunities to

code honestly. Hence the bar, *c*

*a*

, should be set close to 100%. At the higher extreme, when the quota

sets *all* marbles to the same color, the coders have *no* chance to do any honest coding, therefore *c*

*a*

=100%,

and π and α are both undefined. Although *a*

*r*

would be 100%, all of it is deemed due to chance.

The lower extreme is different between π and α. For π, when marbles are half black and half

white, the coders have 50% chance agreeing by chance and 50% chance of coding honestly, therefore the

bar is set at 50%, which is the lowest *c*

*a*

under π. Table 1a and Figures 1a, 2 & 3 display the entirety of the

changing benchmark for π from various angles in various details.

For α, the lower extreme varies according to the number of cases (*N*) coded. When *N*=2,

Krippendorff’s *c*

*a*

is either 0% (when distribution is 50&50%) or 100% (when distribution is 0&100%).

As *N *increases from 2 to approach infinity, this lower end of the bar increases from 0%; to approach 50%.

Figure 1b, 2, 3, & 4 and Tables 1b & 2 display the changing pattern from different angles with various

details.