"Dedicated to exposing the lies and impeachable offenses of George W. Bush"

Grading the Pollsters
WSJ
November 16, 2006

Political polling has come a long way since "Dewey Defeats Truman."

Pollsters earned high marks in last week's elections: Surveys correctly predicted that the Democrats would win control of the House of Representatives, that the Senate would be closely contested and that Democratic gubernatorial candidates would sweep into statehouses.

That doesn't mean that some pollsters didn't do better than others, however, and firms haven't hesitated to trumpet their results. "Rasmussen Reports is the winner in polling accuracy!" read a news release from the Asbury Park, N.J., polling firm. The headline from a Zogby International release said the Utica, N.Y., pollster batted "10 for 10" in its telephone polls of competitive Senate races, and asserted the results from its online polls showed its "interactive polling model shines."

I crunched the results from pollsters on dozens of races, and found that some did indeed appear to be more accurate than others. I'll share my results later in the column, but before I do, I want to explain why I'm more reluctant to crown a winner than some of the polling firms were in their PR. The science of evaluating polls remains very much a work in progress.

"A small cottage industry has looked at these issues, but not a large number of people have bothered with it," said Charles Franklin, a professor of political science at the University of Wisconsin who blogs about polls at Political Arithmetik.

Until recently, measuring poll accuracy was time consuming, and analyses often followed long after the election. Jay H. Leve, founder and chief executive of Verona, N.J.-based SurveyUSA -- one of the companies I evaluated -- said it took seven employees of his firm a full month to analyze the 2004 polls.

But two months ago, the Web site Pollster.com launched, compiling public reports from several polling firms. Some individual firms have also been quicker to post their own results on their Web sites.

While data are now easier to obtain, there remains disagreement about how to calculate accuracy. Is picking the winner enough? Most experts agree it isn't, and focus instead on measuring how close the predictions were to the actual spread -- the difference in percentage points between the two candidates. But what if a poll accurately predicts an election will be decided within a one-percentage-point margin, but incorrectly identified the victor?

Today's preferred methods haven't changed much since political polling's nadir, in 1948, when erroneous pre-election surveys were relied on by the credulous press to predict a victory for Republican Thomas Dewey over Harry Truman. In one of its editions printed on election night, the Chicago Daily Tribune ran the infamous headline, "Dewey Defeats Truman."

To assess what went wrong and to regain credibility, the nonprofit Social Science Research Council outlined ways to measure poll accuracy. Among eight methods the researchers considered, they chose to measure the difference between the predicted percentage vote for the leading candidate, and the actual share for that candidate. However, this method doesn't spot big errors in predicted victory margins, and can be skewed by independent candidates. As it happens, in 1948, Strom Thurmond and Henry Wallace influenced both the polls and President Truman's victory.

More recently, a study of 2002 polls by Prof. Franklin showed that the holder of a five-point lead in a poll won only 60% to 65% of the time, and polls showing the leader ahead by 10 points or more were dead wrong 10% of the time.

Pollsters expected a rout by incumbent Bill Clinton in the 1996 presidential election, but he beat Republican Bob Dole by just 8.5 percentage points in the popular vote. Political scientist Everett Carll Ladd Jr. criticized the polls in the press, calling them worse than the 1948 fiasco, and prompting veteran pollster Warren Mitofsky to analyze the results.

Mr. Mitofsky found that "no standard metric for gauging poll accuracy had been adopted by the polling community." So he went back to the SSRC's work after the 1948 election, and found another method he preferred: comparing the predicted victory margin to the actual margin, or the error on the margin. If a poll finds Candidate A ahead by three points and he wins by 10 points, that's an error on the margin of seven points. If Candidate B wins by five points, that's an error of eight points. There's no penalty for picking the wrong winner; pollsters are graded only on their error margin. (Mr. Mitofsky, a pioneer of exit polling and a helpful source for my Numbers Guy columns, died Sept. 1 at age 71. He was remembered online by several polling organizations.)

I opted to use Mr. Mitofsky's method in my own number crunching. I looked at five pollsters that were among the most prolific: Rasmussen, SurveyUSA, Zogby (which releases separate telephone and online polls) and Washington, D.C.-based Mason-Dixon. For all but the latter, I used the numbers posted on the organizations' own Web sites. For Mason-Dixon, which keeps some of its poll data behind a subscriber wall, I used Pollster.com to find polls from the two weeks before the election. I checked the results against vote counts as of this Tuesday.

(I chose the firms because they were active in many statewide elections. More familiar names such as Gallup and CBS News/New York Times polled in fewer races, focusing more on national attitudes such as approval ratings for Congress.)

There were some interesting trends: Phone polls tended to be better than online surveys, and companies that used recorded voices rather than live humans in their surveys were standouts. Nearly everyone had some big misses, though, such as predicting that races would be too close to call when in fact they were won by healthy margins. Also, I found that being loyal to a particular polling outfit may not be wise. Taking an average of the five most recent polls for a given state, regardless of the author -- a measure compiled by Pollster.com -- yielded a higher accuracy rate than most individual pollsters.

On to the results: In the Senate races, the average error on the margin of victory was tightly bunched for all the phone polls. Rasmussen (25 races) and Mason-Dixon (15) each were off by an average of fewer than four points on the margin. Zogby's phone polls (10) and SurveyUSA (18) each missed by slightly more than four points. Just four of the 68 phone polls missed by 10 points or more, with the widest miss at 18 points.

But the performance of Zogby Interactive, the unit that conducts surveys online, demonstrates the dubious value of judging polls only by whether they pick winners correctly. As Zogby noted in a press release, its online polls identified 18 of 19 Senate winners correctly. But its predictions missed by an average of 8.6 percentage points in those polls -- at least twice the average miss of four other polling operations I examined. Zogby predicted a nine-point win for Democrat Herb Kohl in Wisconsin; he won by 37 points. Democrat Maria Cantwell was expected to win by four points in Washington; she won by 17. (Zogby cooperated with WSJ.com on an online polling project that tracked some Senate and gubernatorial races.)

The picture was similar in the gubernatorial races (where Zogby polled only online, not by phone). Mason-Dixon's average error was under 3.4 points in 14 races. Rasmussen missed by an average of 3.8 points in 30 races; SurveyUSA was off by 4.4 points, on average, in 18 races. But Zogby's online poll missed by an average of 8.3 points, erring on six races by more than 15 points.

Zogby's online polls "just blew it" in Colorado and Arkansas governor races, Chief Executive John Zogby told me. (See Zogby's scorecard.) In other races, such as the two Senate races I mentioned, "we had the right direction but a closer race than the final." One explanation, he said, may be that Zogby's final online polls collected responses one to two weeks before the election, whereas other polling firms were active until the final week. "We have more work to do" to improve online polling, Mr. Zogby said, but he added, "we believe it's not only the wave of the future, but the future is very close to now."

"I don't think there's anyone out there who did better than us," Rasmussen Reports President Scott Rasmussen told me. (Read its own scorecard for its Senate and gubernatorial polls.)

Larry Harris, a principal with Mason-Dixon, said of his firm's numbers: "We had another good year."

Meanwhile, SurveyUSA has posted a detailed scorecard on its Web site, listing how its results compared with those of competitors in different states. "We won't be satisfied until we have an error of zero across our entire body of work," Mr. Leve said. "We're obviously proud of what we did. But I don't think any pollster whose error is greater than zero should be bragging, and we're not."

Remember, my analysis didn't penalize pollsters for picking the wrong winner, which happened nine times, though that's something that other evaluators might take into account. I ignored it partly because some races were decided by a few points or less -- a gap smaller than the statistical margin of error on these polls. For example, Rasmussen's final poll in the Minnesota governor's election showed Democrat Mike Hatch leading Republican incumbent Tim Pawlenty by two points. Mr. Pawlenty won by one point. Rasmussen also missed the Missouri Senate race by a small margin, with the loser, Republican Jim Talent, leading by one point in its final poll.

* * *

Write to Carl Bialik at numbersguy@wsj.com

Original Text

Commentary: