Mead 17
clarified coding instructions, simplified the coding scheme, used two coders rather than one, and had
them code the hearings sequentially.
The current results are much more robust.
The RAs first code a stage independently, following my instructions. I then calculate how
closely their codings agree on each variable and advise them in general terms where agreement is
low and high. For the second cut, they review their codings, especially for variables where
agreement was low, but still working independently. For the third cut, they work together, debating
codings and adopting the same ones where they agree, although they may still disagree.
The new coding effort began in 2001 and has run for four years, funded by private
foundations. The work has taken more time and money than expected, but otherwise there have
been no serious problems. The coders have been loyal and industrious. One of the RAs I hired in
2001 is still with the study. The other worked for two years, and his successor has done very similar
work.
Intercoder agreement is now high. Table 1 shows the results achieved in the hearings coded to
date. Agreement is highest about whether progressive, obligation, or opportunity issues are present.
It is somewhat lower about whether paternalism is present, apparently because this category is less
sharply defined than the others. In the five stages completed to date, the coders have typically
reached agreement on over 90 percent of cases on the third cut. Equally important, the differences
between coders appear random. That is, they do not arise from any systematic differences about the
meaning of the categories or how to apply them. For this reason, I take the third cut of coding as the
most reliable, and it is these results I report below.
Table 1: Intercoder agreement in welfare politics study (percent)
Stage
Cut
Issue present?
Average
Leading issue
SSA
1st cut
65-98
85
90
2nd cut
67-100
86
88
33
I consulted Youssef Cohen, a colleague in the Politics Department at NYU, and Gary King, a former
colleague here who is now in the Government Department, Harvard University.
34
Specifically, I report data from Kendal Elliott’s third cut, as she is the RA who has coded all stages
to date. However, her third cut is substantially a joint product with her partners, Ian Gold and Frank Ortiz.
Convention