Gallup's Shame

Gallup, even without their new poll, is without question the top outlying polling organization in this election. Since they began doing state polls on the 2004 campaign, one twelve occasions Gallup has had a poll in the field for at least one day when at least one other non-partisan polling firm has had a poll in the field. On eleven of those twelve occasions, Gallup's results where the most pro-Bush of the other non-partisan operations. On the other occasion, Gallup was actually the pro-Kerry outlier: (source)
(* = three way trial heat):

Florida, 7/19-7/22
	 Bush  Kerry   Date
Gallup	  50	 46    7/22
M-D*	  48	 46    7/21
Zogby*	  48.5	 48.4  7/23
IA	  46	 46    7/20
LAT	  45	 45    7/21
R2000	  44	 49    7/19

Florida, 8/20-8/22
	 Bush  Kerry   Date
Gallup*   48   46      8/22
Zogby*	 49.0  49.6    8/21

Minnesota, 9/11-9/14
	 Bush  Kerry   Date
Gallup	 48	 46    9/14
M-D	 46	 45    9/14
Star	 41	 50    9/13

Missouri, 9/3-9/6
	 Bush  Kerry   Date
Gallup	 55	41	9/6
Rasm	 48	42	9/3
Zogby*	 48.5	48.9	9/3

North Carolina, 7/9-7/11
	 Bush  Kerry   Date
Gallup	 54	39	7/11
R2000	 49	44	7/10-4

Ohio, 9/4-7
	 Bush  Kerry   Date
Gallup	  52	 43	9/7
SUSA	  50	 47	9/8

Ohio, 8/13-9/15
	 Bush  Kerry   Date
Gallup	  45	 47    8/15
Cincy	  46	 48    8/11-7

Ohio, 7/19-7/22
	 Bush  Kerry   Date
CD	  47	 44	7/23
Zogby	  48.1	 46.8	7/23
ARG	  45	 47	7/22
Gallup	  45	 51	7/22

Pennsylvania, 9/4-9/7
	 Bush  Kerry   Date
Gallup	  48	 47	9/7
Rasm	  46	 48	9/6-8
SUSA	  47	 49	9/7-9

Pennsylvania, 8/23-8/26
	 Bush  Kerry   Date
Gallup	  48	47     8/26
Rasm	  45	49     8/26

Washington, 9/3-9/6
	 Bush  Kerry   Date
Gallup*   44	 52	9/6
Zogby*	  44.2	 52.7	9/3

Wisconsin, 9/9-9/12
	 Bush  Kerry   Date
Gallup	  52	44     9/12
Rasm	  49	47     9/12
No other polling outfit, not even Zogby or Strategic Vision, is even close to being in favor of one candidate in state polls as frequently as Gallup has favored Bush during this election. Even the one time Gallup favored Kerry, it was still an outlier.

Looking at national polls, one finds almost exactly the same story. Since late July, Gallup has, save once, been the outlying poll. Here's how the last four Gallup national polls stack up against other non-partisan polls taken during the same time period (source):

National, 7/30-8/1
	Bush	Kerry	 Date
Gallup	 51	 47	  8/1
ARG	 46	 49	  8/1
Rasm	 45	 48	  8/2
CBS	 43	 49	  8/1
News	 44	 52	  7/30

National, 8/23-9/25
	Bush	Kerry	 Date
Gallup	 50	 47	 8/25
LAT	 49	 46	 8/24
Rasm	 46	 46	 8/24-6
Time	 46	 46	 8/24-6
Fox	 44	 45	 8/25

National, 9/3-9/5
	Bush	Kerry	 Date
News	 54	 43	  9/3
Gallup	 52	 45	  9/5
Rasm	 47.6	 46.5	  9/5

National, 9/13-9/15
	Bush	Kerry	 Date
Gallup	 54	 40	 9/15
Rasm	 49.3	 44.7	 9/15
Econ	 47	 46	 9/15
Pew	 46	 46	 9/14
Harris	 47	 48	 9/13
This last one is the most profound outlier of any national poll this year, leaving Newsweek and Time in the dust (maybe that is what they were trying to accomplish). Then again, maybe they aren't leaving Time in the dust, since CNN and Time are affiliated news organizations and frequently do joint polls.

It is pathetic and unacceptable for a "non-partisan" polling firm to be produce the outlying poll in favor of Bush in fourteen of its last sixteen polls. The odds of this happening at random are around one in 14,000. Considering those odds, the far more likely explanation for all these outliers is that Gallup's polling methodology is inherently structured in favor of Bush. Whether or not it is intentional, I do not know. However, I do know that Gallup's polls are connected to the largest news outlets in America of any poll, both in terms of print (USA Today is the largest circulation newspaper in the country) and cable news (CNN has more viewers than Fox, they just watch for shorter periods of time). I also know that sensational headlines sell. I further know that Gallup's chairman is a Republican donor.

This is a shameful state for the oldest and most respected polling organiztion in the country. Shame on you Gallup.


Display:


Gallup's Bush bias (none / 0)

Clearly it isn't random, but maybe Gallup is not deliberately partisan, just incompetent. Gallup's likely voter model differs from those of other pollsters, and their sampling and weighting approach may also differ. They may not be keeping up with changes in the electorate's phone accessibility and they are probably assuming that the turnout pattern in the last election or two, which favored Republicans, will continue.
by EvanstonDem on Fri Sep 17, 2004 at 12:44:40 AM EST

MN poll (Strib) (none / 0)

You have the Star Tribune poll backwards.  Just thought you might like to know.  There was an earlier article (was it by you? I don't remember) about the respondent bias in the Gallup poll (something like 53-41 Bush voters in 2000).  What a strange world we live in...
by paul orwin on Fri Sep 17, 2004 at 01:52:57 AM EST

Re: MN poll (Strib) (none / 0)

Thanks. Fixed.
by Chris Bowers on Fri Sep 17, 2004 at 02:33:58 AM EST
[ Parent ]

Poll bias (none / 0)

I've been tracking poll bias for national polls for several months. You commented on an earlier version of this graph a while back, which comes from this page. It's been very odd seeing how the Gallup poll has been moving up and up -- a few months ago it was in the lower right of the graph, now it's in the upper right and still climbing.
by Robert Chung on Fri Sep 17, 2004 at 07:24:34 AM EST

Electoral Vote (none / 0)

Because it is so close to the election date, I am more concerned with the Electoral vote - For at least the last couple of days, Frontline on C-Span has been claiming over 300 electoral votes for Bush - Is Frontline a dependable poll?
by Dorothy Ligon on Fri Sep 17, 2004 at 09:13:01 AM EST

Re: Electoral Vote (none / 0)

There a are slew of states where Bush leads by one point. that is the cause.
by Chris Bowers on Fri Sep 17, 2004 at 11:50:57 AM EST
[ Parent ]

Al Hunt in WSJ Analysis of Polls 9/17/04 (none / 0)

Here is Al Hunt's analysis of polling in today's (September 17) Wall Street Journal:

What If the Polls Are Wrong?

Election Surveys That Screen Out
'Unlikely' Voters Might Be Outdated
September 17, 2004

Presidential elections are poll-driven. The candidate ahead in the surveys usually gets better coverage, and the results energize supporters. The one behind often comes across as doing little right, and campaigns and constituencies lose confidence.

But what if the polls are wrong, and we aren't surveying the real likely electorate?

This might be more than an academic issue. A number of polls this presidential race show a gap in the preferences of registered voters vs. likely voters. In these models, the president usually does better with likely voters, the figure most news organizations emphasize. To get to likely voters, all polling organizations use what is called a "screen," asking questions to determine who is likely to actually turn out on election day.

These screens differ greatly, as there is no consensus among experts on what works best. "This is an art, not a science," says Peter Hart, the prominent Democratic polltaker who has helped conduct The Wall Street Journal/NBC News survey for 15 years.

This controversy will be fueled by today's just-released Gallup poll that shows George Bush with a 13-point lead over John Kerry. That is at variance with other surveys this week, which suggest a tight race with a much smaller Bush tilt. But the likely voters margin also is considerably larger than the eight-point advantage in Gallup's registered voters in this survey. The likely voters match-up invariably gets more attention.

Gallup explains it has what it considers a time-tested formula for determining most likely voters. It asks eight questions, such as current intensity of interest, past voting behavior and interest, and whether you know where your voting place is.

"We've discovered that if we ask a set of more indirect questions, we can better predict who is or is not likely to vote," Frank Newport, editor in chief of the Gallup Poll, has said.

But there is reason to suspect those criteria are outdated, especially in an election where both sides say the intensity level is much higher than four years ago and get-out-the-vote organizations are considerably better than ever -- few people on Nov. 2 will be in the dark on where the voting polls are.

"A formula that made sense years ago may not recognize all the changes in society," notes Mr. Hart. "It gives more credence to past behavior and too little to current interest."

"For low-turnout elections those old models work well," suggests Bill McInturff, a Republican, and the other WSJ/NBC News pollster. "But in today's presidential election those models tend to [tilt to] a little older, a little more white, a little more affluent and a little more Republican voters. They may miss some of the extraordinary activity going on in African-American and Latino communities."

The registered-likely voters dichotomy also is evident in some of Gallup's state surveys including last week's Ohio results." Among registered voters in the Buckeye State, Bush-Cheney had a 48%-to- 47% edge, a dead heat. Among likely voters, however, this poll had the Republicans up 52%-44%; that garnered all the attention, followed by a spate of stories suggesting this key battleground state was moving to the president.

Curiously, the Gallup poll in the similar state of Pennsylvania at the same time showed a virtually even race among both registered and likely voters. And occasionally, the screen favors the Democrats; a Marist survey this week of New York state showed Sen. Kerry 11 points ahead among likely voters, but only seven points ahead among registered voters.

But most of the time the screen for likely voters tilts Republican. In 2000, Gallup's election eve survey showed George Bush ahead by two points among its likely voters; he trailed Al Gore by a point among registered voters, very close to the final outcome.

In 2000, the next to last WSJ/NBC poll before the election showed Republicans doing three points better among likely voters than registered voters. The election eve survey showed Bush up three points among likely voters, but failed to tally registered voters and didn't predict Al Gore's victory in the popular vote.

The Wall Street Journal and NBC News have settled on one question to screen likely turnout. Registered voters are asked their interest level in the election on a scale of 1-10, and those that respond 9 or 10 are considered likely voters.

Both camps expect an increase in the 105 million Americans who voted last time; the Bush camp looks for abut 111-112 million while the Kerry campaign projects 116-118 million; nobody can be sure exactly who those additional voters might be.

The probable outlook: Polls will vary and conflict if this race remains tight. Also, poll watchers must remember that the best survey has a three or four-point margin of error; that means if it shows the race even, one or the other candidate actually could be up by a half-dozen. Here's a final guide: if almost all the election eve polls show one candidate up four or five points or more, take it to the bank. But if most show the race within a couple of points, plan on staying up late election night.

by drplaud on Fri Sep 17, 2004 at 09:17:03 AM EST

Re: Al Hunt in WSJ Analysis of Polls 9/17/04 (none / 0)

Interesting gem:
Both camps expect an increase in the 105 million Americans who voted last time; the Bush camp looks for abut 111-112 million while the Kerry campaign projects 116-118 million; nobody can be sure exactly who those additional voters might be.
It tells us a lot about the differing strategies.
by Jerome Armstrong on Fri Sep 17, 2004 at 10:23:01 AM EST
[ Parent ]

Why Gallup Is Worrying (none / 0)

Hint: It's NOT because of the LV model, which is outdated and clearly inaccurate.

It's because of how their LV screen works. Unlike most outfits, they actually ask voters how excited they are about the election (more or less). It worries me that, somehow, that kind of screen produces the worst results for Kerry. I thought the Dem electorate was fired up?

by AS on Fri Sep 17, 2004 at 10:19:26 AM EST

I believe one of the most important aspects... (none / 0)

...of the Gallup screen is "Did you vote in the last election?"  Obviously, there are a lot of pissed off people, who are voting for Kerry, who DID NOT vote in the last election.

But the Gallup poll is fucked up for other reasons-I believe that Wall Street article claimed they had Bush is up by 8 in Registered Voters, which is still wrong.

by Geotpf on Fri Sep 17, 2004 at 12:43:58 PM EST
[ Parent ]

% Undecided (none / 0)

Notice that Gallup almost always has the lowest percentage of undecideds/Nader voters (highest percentage of Bush+Kerry).  This holds true until the most recent poll.  Gallup probably pushes the undecideds harder to get them to report which way they're leaning than the other polls.  

I believe that leaners tend to break towards the incumbent in polls, even as they give Bush a low job rating, think the country is going in the wrong direction, and even if they break towards the challenger in actual voting.

by Chef Ragout on Fri Sep 17, 2004 at 11:00:43 AM EST

Found the partisan split (5.00 / 1)

Heavily Republican

http://www.theleftcoaster.com/archives/002806.html

by cfr on Fri Sep 17, 2004 at 03:44:25 PM EST

Gallup vs Ras (4.00 / 1)

I thought Ras was more accurate than Gallup, but today Ras has Bush 5 points up on the best to hanldle the economy issue. Is Ras out to lunch or is the electorate?
by gail on Fri Sep 17, 2004 at 03:52:29 PM EST

Re: Gallup vs Ras (5.00 / 1)

My sense was that Rasmussen makes 500 calls per day, but they lean Republican, so if they want to make the story go a certain way, they just call more Red States that day. But I don't know too much about the way they do their research.
by cfr on Fri Sep 17, 2004 at 04:15:05 PM EST
[ Parent ]

Re: Gallup vs Ras (5.00 / 1)

500 is such a small sample! Also if they are repub-leaning (I know Scott Ras is) maybe this is the way they'd accomplish their agenda - below the radar - by having Bush inch up rather than leap up as Gallup did.
by gail on Fri Sep 17, 2004 at 04:26:54 PM EST
[ Parent ]

Wrong... (5.00 / 1)

...post-labour day, they have been making ONE THOUSAND calls a day, and the numbers reflect three days.  So that's THREE THOUSAND CALLS in each report, which is a HUGE sample.

Their current numbers undersample Jews (who only voted 19% for Bush in 2000), because Sept. 15th was a Jewish holiday-they wouldn't be home, or take a telephone survey if they would be.  Same goes for Gallup and the other polls.  So they will have a pro-Bush bias until Sept. 19th, when the Sept. 15th data falls out of the sample.  Bush gained two points and Kerry lost 1.7 points because of this on Sept. 16th, the first day Sept. 15th's data was used.

by Geotpf on Fri Sep 17, 2004 at 10:58:56 PM EST
[ Parent ]

Re: Wrong... (5.00 / 1)

Let's remember this on the 25th - the undersampling of Jews, as it is another Jewish holiday. My relatives are Jewish and they would never answer the phones on holidays like these.
by gail on Fri Sep 17, 2004 at 11:16:23 PM EST
[ Parent ]

Oh Well (3.00 / 1)

Bush is going to win because every major television outlet is for him.

The majority of voters don't venture beyond TV for their news.

If they ran anti-Bush stories, and he won, they wouldn't get any leaked stories during the next administration.

Money, Money, Money...

by JoshNarins on Fri Sep 17, 2004 at 03:53:27 PM EST


You are not logged in.

In order to post a comment, you must be logged in. If you have a member account, please log in to comment.

If not, you can make an account right here. It's quick and free.