This is the post I had been hinting at for three days--Chris
Are most live-interviewer, national polls inflating Hillary Clinton's national poll advantage? For two weeks in April, I spent a lot of time trying to answer that question through a series of posts on MyDD that I referred to as the
Inflated Clinton Poll Theory. Even though, at the end of the series, I concluded that there was no clear evidence to support the theory, there was no official test of the theory at that time and there were still reasons to think that there might be serious problems with national Democratic nomination preference polls. For one thing,
the number of households shifting to wireless only continues to rise, which poses a real danger to traditional, live-interviewer telephone polls. Also, that Rasmussen Reports, which has proven to be an accurate polling firm over the past few years and which employs an automated, IVR polling technique,
repeatedly shows different results from more traditional methods, and as such continues to raise eyebrows. Third, all national polls on Democratic primary preferences, no matter who those polls favor, are including an extremely wide net of Democrats and Democratic-leaning independents in their samples that is unrepresentative of the generally narrow Democratic primary and caucus electorate. Finally, there remains the possibility that live-interviewer telephone polls might create a sort of social pressure that alters results. Do people tell machines different things about their political preference than they tell live humans?
Taken together, do all of these concerns inflate Clinton's national poll lead? For over a month, I have sought out a polling firm that would test this theory. In their latest poll, Cook / RT strategies conducted just such a test, and the results are quite interesting. In fact, they support the Inflated Clinton Poll Theory. The following passage from
a memo produced by Thomas Riehle, which I just received over email:
Lots of experiments will take place in 2007 testing new methodologies to see if they can meet or exceed the standards for accuracy of the tried-and-true live interviewer methodology using alternative polling methodologies. One important test will aim to determine whether the internal breakdowns of alternative methodologies yield similar results and similar conclusions to what we learn from live interviewer polls. RT Strategies, which conducts live interviewer polls for Cook Political Report, cooperated with a leading online polling firm, YouGov America (formerly Polimetrix) to investigate.
YouGov America is the best choice for an online pollster with whom to compare results, because all "online polls" are not the same. YouGov America employs a sophisticated sampling and respondent-matching methodology, in concert with superior panel recruitment, to make it the only online poll that can deliver a true, randomly selected sample without resorting to the crushing expense of recruiting and then providing internet access to communities that are otherwise under-represented online, or by employing the statistical gymnastics of "propensity weighting" to mask that under-representation.
Two recent Cook Political Report/RT Strategies polls were combined to yield a healthy sample size of 790 Democrats and Democratic leaners from live interviewer polls conducted April 27-29 and May 11-13. During that same time span, and using a questionnaire identical to the questionnaire used in the Cook Political Report / RT Strategies polls, YouGov America interviewed a random selection of its online panelists that included 750 Democrats and Democratic leaners. The results follow.
- Hillary Clinton scores less well online than she does in live interviewer polling. Overall, Clinton gets 32% in the combined Cook Political Report/RT Strategies polls, just 24% in the YouGov America online poll. The big differences: Among Baby-boomers and in the West, the online panelists were less likely than respondents to a live interviewer poll to make Clinton their first choice for the nomination. Hispanics online are much less likely than respondents on the phone to support Clinton.
- Obama's scores are very similar online to what he scores in live interviewer polls. Overall, Obama gets 24% with live interviewers, 23% online. Obama tends to score as well or better online among younger voters, much worse among the oldest voters, and not as well online among African-Americans as he does under the live interviewer methodology.
You can
read the entire memo here (PDF), and examine
the full set of comparative crosstabs between the two polls here (large PDF). While I have not had the time to go over it in full detail yet, on the surface it seems to support the idea that Clinton is not ahead nationally by the amount that traditional, live-interviewer polls suggest. In the live interview poll, she led 32-24-12 over Obama and Edwards respectively, but in the YouGov online poll, her lead was significantly reduced to 24-23-15.
Could the difference be social pressure, where Democrats don't tell live-interviewers that they are currently leaning against Clinton? Rasmussen's numbers consistently back up that theory, but those produced by Harris do not. Could it be that traditional live-interview polls and newer polling methodologies sample different universes of voters, thus producing different results? Possibly, but even if that is the case, it is extremely difficult to say which group of polls is sampling a more representative universe right now, both because we don't know who will vote in the 2008 primaries and because few polling firms release comprehensive crosstabs and methodologies. Could it simply be that when it comes to the 2008 Democratic nomination, live-interview polls are growing less useful due to the rising wireless-only population and social pressure, or that newer techniques are not yet able to achieve the same level of accuracy as traditional methods? Both are possible, but neither can be confirmed at this time.
What I do know is that this poll was conducted using the best online polling techniques available, and took place simultaneously with a live-interview poll under the supervision of a single polling firm. In the end, the two polls showed differing results that were statistically significant, and that difference does not favor Clinton. While this is not enough evidence to clearly demonstrate the Inflated Clinton Poll Theory, right now it is pretty much the only up to date evidence available to test the theory, and that evidence suggests that the theory is correct. More testing is obviously needed in order to draw a more definitive conclusion. Still, until that time comes, I don't think that the theory can simply be dismissed anymore, and an asterisk might be needed when it comes to all national Democratic nomination preference polls, both live interview and non-live interview. The current state of public opinion among the Democratic primary and caucus electorate is by no means clear, and it cannot be definitively stated that Clinton holds a large lead. The campaign might very well be tied right now. That, certainly, is big news.