Exit Poll Madness

Kevin Drum at the Washington Monthly has a look at the U of Penn professor's paper on the exit polls:

EXIT POLL MADNESS....Should I pander to the masses? Or remain a responsible voice of calm and reason?

Ah, what the hell. Let's pander. A few days ago I promised that if any credible evidence of voter fraud came my way, I'd post about it. So here it is. I'll explain later why I don't actually believe there was widespread voter fraud this year, but the following analysis is credible, so I'm going to pass it along as promised.

The table below comes from a paper written by Steven Freeman of the University of Pennsylvania. The paper is called "The Unexplained Exit Poll Discrepancy" and it's about the now famous fact that the exit polls showed John Kerry leading in practically every battleground state but then losing in the final tally. Here's the key data:



What does this mean? Take a look at the top line for Colorado. The first colored column shows what the exit polls predicted: Bush would win by 1.8%. The second colored column shows the actual results: Bush won by 5.2%. The third colored column shows the difference between the two: Bush won by 3.4 percentage points more than the exit poll predicted.

In fact, Bush won 10 out of 11 battleground states by more than the exit polls predicted. The odds of this happening by chance are essentially zero.

Now, Freeman's paper corrects a couple of the problems I've seen in earlier efforts along these same lines. First, he uses final (raw) exit poll data, not early afternoon data. Second, he provides a levelheaded discussion of the various reasons on offer for why the exit polls might be off — although in the end he finds them unconvincing.

However, Freeman doesn't allege voter fraud, and neither do I. There are several good reasons for this:


All Freeman demonstrates is that these results couldn't have happened simply by chance. However, they still might have been produced by systematic problems of some kind in the polling methodology.


Freeman knows this, so he then takes a look at possible sources of systematic error. He isn't persuaded that any of them stand up to scrutiny, but he also admits that there's just not enough data to say for sure. What's more, there might be some other source of error he's not aware of.


I've seen several indications that there really are some weird anomolies with the exit poll data. I mentioned one last night, and there's another one you can see right in the table above: the total national sample was about 13,000 voters, but the state level polls all claim to include about 2,000 voters. Obviously something is screwy there.


Finally, you have to posit a way in which the Bush campaign was able to rig the results in every single battleground state. Let's face it: it's a lot easier to think of ways in which the exit polls might be screwed up than to think up credible ways in which the Bushies could have pulled that off.


So why am I posting about this at all if I don't believe it? Mainly because I'm getting progressively more hissed off about the exit polls with every passing day. The folks who ran them have actually encouraged rumor mongering by refusing to publicly explain what happened to us benighted masses. So far, all they've done is write a confidential report that apparently didn't even acknowledge the systemic errors in the final results and instead laid the blame on those irresponsible bloggers who got everyone riled up by posting early results. Meanwhile, their defenders in the media were practically apoplectic about the gall of non-experts using data they can't possibly understand in order to advance their own bizarre conspiracy theories. Which would be fair enough if they'd step up to the plate and give us the benefit of their expertise.

But they haven't. They've declined to talk to reporters, they've released no serious findings of their own, and they haven't made their raw data available even to qualified researchers. Freeman's numbers might well be wrong, but there's no way for him to find out. Nature abhors a vacuum, and the experts are the ones who have created the vacuum in which rumors thrive.

So that's why I'm posting this: because I want to put some pressure on them to come out from their caves and tell us what they think — and to debunk guys like Freeman if they can. After all, there's probably a perfectly plausible explanation for all this. In fact, I'm sure there is. But until they tell us what it is, the conspiracy theories aren't going to go away.