Much to my surprise, and with tremendous thanks to
MoveOn.org, the critiques
Steve Soto,
Ruy Teixeira and I have long been making about the Gallup poll are starting to receive some coverage. It even prompted Frank Newport, Editor in Chief of Gallup, to make a public defense of the poll on
CNN:
FRANK NEWPORT, EDITOR-IN-CHIEF, GALLUP: Well, it's not at all unusual to find people criticizing polls or other information that they don't like, they disagree with the results of. Gallup's had that from both sides through the years when we show results that look like they're favoring one side or the other.
But certainly, this ad seems to have more misstatements of fact and misunderstanding than most. And it's good for us to be able to respond to it.
OK. Let's look at these misstatements of fact. They include:
They seemingly come up with some data, we're not sure from where, saying, oh, the average seems to suggest right now that Bush has just a three-point lead over Kerry. That doesn't seem to be the case.
Most observers now say it's a six to eight-point lead. That's what recent polls are showing.
You would think that someone who runs a polling organization would have bothered to do an actual poll of polling firms before making this statement. Had he done so, this is
what he would have found:
The numbers in the ad, which are quite eye-opening, are rock-solid. The ad says Gallup's average LV lead for Bush this month has been 10 points, while the average of all other LV polls has been 4 (they're clearly referring to 3-way LV results--which are by far the most numerous LV results--based on other data in the ad). That's correct. Even taking into account data released since 9/26 (the end-date for the ad's analysis), Gallup this month has averaged a 10 point lead for Bush among LVs in 3-way trial heats, while the other 27 3-way LV trial heats taken this month have averaged a 4 point Bush lead.
Similarly, the ad says polls released since 9/12 (that is, two weeks before the end-date of the ad's analysis), excluding Gallup, have averaged a 3 point lead for Bush in 3-way LV trial heats. Correct again, even adding in polls released since 9/26. In the 17 3-way trial heats released since 9/12 by polling organizations whose names are not "Gallup", Bush is averaging just a 3 point lead.
A polling firm's credibility is not aided if, when a representative appears on national TV to defend the accuracy of the polls, that representative makes an inaccurate, unresearched guesstimation about what other recent polling firms are showing. For crying out loud, Newport can't even accurately weight a sample with 27 responses, or at least doesn't even care to try.
In a USA Today article by Mark Memmont on the subject, Gallup offers a defense for why it does not weight by Party ID, the issue at the center of this now month-long debate over polling methodology:
The polling firm does not adjust its "pool" of voters to add or subtract Republicans or Democrats in an effort to mirror those parties' estimated make-ups.
Among the reasons Gallup doesn't try to do that:
- It believes there are no reliable data on which to estimate exactly how many Republicans or Democrats there are in the country. Some states, for example, don't require voters to register by party affiliation. Basing an adjustment on previous year's exit polls, "means you're 'weighting' one poll based on the results of another poll, which has its own built-in sampling error," Newport said.
- It believes party affiliation "is an attitude, not a demographic trait" and that voters can change their minds about which party they identify with more than once during an election year, Newport said. That would explain, he said, why the number of people who identified themselves as Republicans went down during this year's Democratic primaries -- when Kerry and his competitors were in the news.
The first reason is bizarre. Of course you can't "estimate exactly," that is why it is called an estimation. But if Gallup has a problem with inexact estimations, then why do
they estimate voter turnout in their likely voter model, which also cannot be known for certain?
Gallup asks each [RV] respondent seven LV screening questions, and gives each person an LV score of 0 to 7. [Assuming a turnout of 55 percent], the top 55% are classified as likely voters.
Gallup has no problem assuming a turnout of 55% of registered voters, which is absurdly low. According to Dave Leip, the lowest turnout ever among registered voters was
66.4% in 1996. So, Gallup has no problem ignoring history and estimating, at a crucial point in their methodology, mid-term level turnout in a Presidential year, but they refuse to estimate Party ID? This is, I suppose, because Party ID swings so wildly over time:
Pew
Harris:
Year Rep Dem Gap
2003 28 33 5
2002 31 34 3
2001 29 36 5
2000 29 37 8
1999 29 36 7
1998 28 37 9
1997 29 37 8
1996 30 38 8
1995 31 36 5
1994 32 37 5
1993 29 38 9
1992 30 36 6
1991 32 37 5
1990 33 38 5
The Pew poll is of 19,000 voters a year, while the Harris polls is of 6,000 voters a year. Gallup claims they can't make a good estimation from these numbers, and that this isn't a stable demographic trait? Quite frankly, after seeing these numbers, anyone could offer an estimation of next year's Party ID numbers that would be accurate within two points. While it is impossible to know exactly what Party ID looks like, it is quite easy to come pretty darn close.
That last point is why I have cast my lot with the firms who weight by Party ID, and why I can sort of understand if some other polling firms do not. Weighting results based on estimated Party ID is not properly scientific, and so I can see why some hardliners would reject it (of course it is also not properly scientific to claim that a poll with a margin or error of plus or minus four points shows a candidate up by eight). However, since it is so easy to come so close to nailing Party ID, and since having a proper model of Party ID turnout in a poll virtually guarantees that poll will be a very close reflection of the election results, why wouldn't a public pollster want to weight by Party ID? If nothing else it will save embarrassment and/or raise political junkie street cred. The two firms which weighted by Party ID in both 2000 and 1996, Harris and Zogby, were the most accurate in both elections.
At least the Party ID issue in polling has received some press now, and there is somewhat more public recognition that thee are two opposing camps when it comes to this issue. Mark up another small victory for the lefty political Blogosphere.
