polling matters

Mark Schulman, from Schulman, Ronca & Bucucvalas, Inc.(SRBI), which has done polling for Newsweek for many years, responds to Alan Reifman's discussion of poll weighting.Alan, your web site presents a very insightful discussion of the weighting issue, which often accounts for major differences among poll findings.  I'm on my way out of the country at the moment, so the following are just some quick observations. As you know, we do the Time Magazine Poll, and I've received a number of queries over the past few weeks about this issue. It not only needs to be addressed, but weighting by party ID can result in serious distortion of the horserace numbers. Here's why:

1.  As an observer of party identification tallies day after day on our election surveys, it's clear that we're not measuring a constant factor. It varies day by day, week by week.  

2. Why does it vary? Most polls place the party ID question near the end of the questionnaire, so that it does not interact or contaminate the horserace measure and any other head-to-head candidate comparisons. The horserace always takes priority, since that's the topline number we report. As a result, respondents may tend to bring their party ID in line with their partisan choice, particularly after having gone through an extensive battery of election items. It's simply "cognitive consistency." Hence, a Bush surge, for example, might elevate the number of voters later in the survey identifying themselves as Republicans.

3. Since party ID is a "variable" and not an enduring constant, as is age or gender, it varies!

4. Voting behavior literature (Campbell, Converse, Miller and Stokes, The American Voter, for example), used to posit party ID as anchoring partisan choice, as if it were a constant. It's likely that party ID was a more enduring "constant" in the 1950's, but, that was then, and this is now! Voters are just not as tied to party as in the past. Let's get over this likely out of date notion that party ID is a constant that anchors the vote. The causal arrows here are unclear, that is, what influences what.

5. Hence, weighting by party ID, and its party ID, not party registration, can seriously distort the horserace data. Weighting by party ID would damp down the Bush surge over the past few weeks. Yes, there may be some "at home" selection bias when we interview during party convention periods. However, not all that many folks watch the conventions and the networks provide little convention coverage.

Finally, my choice, and the choice of most the major media polls, is to weight by factors that we know are real, such as age, gender, region, education, number of adults in household, number of voice phone lines, etc. While you can argue about the reliability of Census data, I'll place my bets with the Census rather than party ID.

David Moore has a good discussion of this issue as well on the Gallup web site.

That's the short version of my views. I really do believe that we need to put this issue to rest and stop pretending that there's legitimacy to party ID weighting. I look forward to further comment!

It can't be ignored that media vehicles such as Newsweek and Time live for things like "the Bush surge", as Mark puts it. They want a horserace that makes news & sells magazines. Regardless of the argument, SRBI's treatment of party ID as being somewhat fluid (and therefore not a factor to weight for) satisfys that need. Rasmussen 2004, Zogby, and SUSA might not be as flashy in their ups and downs, but their accuracy is just fine, or better, and arguably due to their Party ID weighting. I'll put the responses in the extended entry.

My Dream Poll

I have definitely been one of the loudest voices bitching about polls recently. However, I am finding such arguments both personally tiring and "professionally" repetitive. As such, I'd to set the bar and make clear what type of poll that I would like to see.

Polling firms should increase the sample size to around 2,500-3,000 RVs, and produce separate results for each of the three major Party ID subsets. That is, make the top sheet numbers where registered Democrat identifiers stand, where registered Republican identifiers stand, and where registered Independent identifiers stand. The advantages of such an approach would be a complete lack of assumption on what turnout will look like. Deviation from these results, as determined by exit polls and final results, will demonstrate who won the GOTV battle.

If only there was someone conducting this as a tracking poll. Right now, Rasmussen is the closest. How would your dream poll be conducted?

More Bad Poll Weighting

From Ruy Teixeira:The Gallup poll's internals also show that Kerry continues to lead among independents (49-46) and that both parties' partisans are equally polarized for their respective candidates (90-7) Considering this, why did the Gallup poll show Bush up one one among RV's? The only possible answer is that they included more Republicans in their sample than Democrats, even though Democrats have a national lead in party ID. In fact, had the Gallup results been weighted according to 2000 turnout (39D-35R-26I), Kerry would have led among registered voters 50.3-46.2.

Combine this with the new ICR poll, and the last two days of Rasmussen tracking, I am starting to seriously wonder if Bush is really ahead at this point.

Update: Wow--I think I was actually right. Check out this post by Ruy, quoting Alan I. Abramowitz of Emory University a few hours after mine (emphasis mine):

Among registered voters, Gallup shows Bush leading by one point overall, with Kerry leading 90-7 among Democrats, Bush leading 90-7 among Republicans, and Kerry leading 49-46 among independents. This means that Gallup's sample of registered voters includes more Republican identifiers than Democratic identifiers. But in 2000, according to the VNS national exit poll (which hits the overall percentages for Bush and Gore right on the nose), Democrats made up 40.3 percent of the electorate while Republicans made up only 36.5 percent of the electorate. If you apply Gallup's trial heat results among Democrats, independents, and Republicans to the VNS 2000 electorate, Kerry comes out with with a four point lead: 50.3 percent to Bush's 46.4 percent. See? I'm not an idiot.

Voter Turnout Demographics, Realities and Myths

Using data from the 2000 census and comparing it with exit polls from the 2000 election, I have produced some figures on voter turnout by demographic group.

There's more...

Incumbent Rule Research Update

Lately, I have been spouting off about how undecideds tend to break about 60% for the challenger and 40% for the incumbent in the final week of the campaign. I was basing this off of a quick estimate from research I had done five weeks ago. However, I had never actually sat down and crunched the numbers to truly determine the average undecided swing in the final week of a campaign in elections involving an incumbent. Now I have, and that is one Friday afternoon and evening that I will never get back.

Using my own research as well as research sent to me by Nick Panagakis of the National Council on Public Polls, I have gone through 451 poll results since 1976. In all 451 cases, the poll was in the field for at least one day that was within seven days of the election. In every case, it was the final poll taken by the polling firm for the campaign in question. Also, I do not believe that any internal or partisan poll results were used. Unfortunately, outside of the Presidential race, I was severely lacking in data from 1996.

My methodology worked as follows:

  • 1. Subtract the final poll result for the incumbent from the actual election result for the incumbent to determine the total number of points the incumbent gained from the final poll until the election. Total all 451 of these results to determine the total number of points all incumbents gained from all final polls until the election.
  • 2. Subtract the final poll result for the challenger from the actual election result for the challenger to determine the total number of points the challenger gained from the final poll until the election. Total all 451 of these results to determine the total number of points all challenges gained from all final polls until the election.
  • 3. Take the total from step #1 and add it to the total from step #2. Divide this total into both the result from step #1 and step #2 to determine the relative gain for the incumbent and the challenger.
The results were as follows:
Year	  Polls    Und.    Inc.  Chal. 
President   28	  2.4	 14%	86%
1976-88    155	 11.8	 20%	80%
1994	  101	 11.2	 35%	65%
1998	   76	 10.1	 27%	73%
2000	   31	  8.6	 40%	60%
2002-4	   60	  7.5	 42%	58%
1992-04    283	  8.9	 34%	66%
Total	  451	  9.7	 28%	72%
Here are my first thoughts on this data:
  • Is the incumbent rule weakening? With the exception of 1998, where Democrats kicked ass on GOTV, it has been a slow progression toward parity. Then again, this chart has holes, such as a complete lack of polls from 1996, which prevents one from drawing such a conclusion with confidence.

  • The number of undecided voters is clearly going down. This year, I see no reason not to expect more of the same. As the two parties slowly become primarily ideological, rather than regional and ethnic coalitions, the difference to voters is becoming starker. As a result of this, more people have made up their minds going into the booth. We really are becoming a polarized nation--that isn't just pundit bullshit.

  • The Presidential sample stands out for its extremely small movement from final polls until election night. Even though undecideds break overwhelmingly--better than 6 to 1--in favor of the challenger in a Presidential race, pollsters seem particularly adept at national trial heats in Presidential races. Probably because of the extreme amount of national attention given to the Presidential race, far more people have made up their minds going into the booth than in other elections. While we should not expect significant movement from the final polls on November 1 to the final results on November 2, whatever small movement there is will be almost entirely for Kerry.

  • Considering how accurate polling firms tend to be in national elections, in my presidential projections I will now include likely voter models from polling firms that do not include registered voter models. Where available, I will still use the registered voter models until the final week, however. Also, I will only include trial heats that push leaners, and give 80% of the remaining undecided to Kerry. There is simply no way that even 5% of the country will be truly undecided between Bush and Kerry going into the booth.

  • Should the 1992-2004 average of 66 to 34 in favor of the challenger, or 2 to 1, be considered a benchmark at which point GOTV efforts make the rest of the difference? At least for now, I'm going to run with this possibility.
It seems like it would be fine to use these numbers as a means of estimating how undecideds will break in any given non-presidential election, as long as it is sold purely as an estimation. These figures are averages and invariably there will be results that break sharply from that average. 66-34 is where the smart money, the house money, should be. You will win over the long term betting on that sort of a split, but in any given wager you might get burned.

Diaries

Advertise Blogads