Assignment to the time condition was determined at the beginning of the interview (but revealed to
the respondents only at the beginning of the knowledge sequence.) The completion rates in the two time
conditions were very similar. Eighty percent of the respondents assigned to the “one minute” condition
Hence, selection effects are
very unlikely. We consider experimental differences between respondents who completed the interview
as valid estimates of the true treatment effects. Later in this paper, we also test for selection effects more
explicitly by adding control variables to the analysis.
Two features of this survey make it a conservative test of our hypotheses. First, Knowledge
Networks informs its panelists by email when a new survey is waiting for them. They can then take the
survey at a time of their own choosing. Hence, even respondents in our control group are not literally
caught during dinner or at other inopportune moments and asked to answer the knowledge questions on
the spot. In fact, they even had the opportunity to pause the interview when they learned that they would
be asked political knowledge questions. (They could not pause once they saw the first knowledge
question.) Clearly, we do not capture the true inconvenience of a typical phone interview. Indirectly,
panelists also receive credit for taking a survey because Knowledge Networks pays for their WebTV unit
or an Internet connection to their PC. To be sure, this reward does not represent an incentive to answer
thoughtfully, but the conditions in our control group do not recreate the conditions of a typical phone
interview perfectly. Therefore, respondents in our experiment are more motivated and less
inconvenienced than respondents in the telephone surveys from which many claims about political
knowledge are derived.
The timing of the experiment also introduces a bias that, if anything, would suppress the effect of our
treatments. It was fielded two weeks before Election Day 2004 after a long campaign. If people do poorly
on survey-based knowledge tests when there is no immediate real-world reason to know something about
2
Seventeen respondents started the interview but never made it to the knowledge questions. Seven of them would
have been assigned to the pay condition, ten to the no pay condition.
7
Convention