Jay Bookman’s Margin of Error

AJC Liberal columnist Jay Bookman writes this morning,

“Among voters 49 and younger, Democrat Michelle Nunn has a nine-point advantage over Republican David Perdue. Among those 50 and older, Perdue has an 11-point margin. If you’re a GOP strategist in this state, those numbers have to alarm you about the future. ”

But put down the Pepto-Bismol, my fellow Republican strategists. The news isn’t nearly as dire, in fact it doesn’t even really qualify as news. Because Jay Bookman’s margin of error is huge.

We’ll just deal with the SurveyUSA poll for 11Alive WXIA-TV, the first he cites, and we’ll discuss only the 18-34 group for simplicity. Among those voters, 40% said they would vote for David Perdue, the Republican and 45% for Michelle Nunn, the Democrat. We’ll call that margin “Nunn +5”

GaPundit readers should know that the first thing I look at is the “Statement of Methodology,” usually, as here, the last part of the published report. Two things stand out. First,

This poll was conducted by telephone in the voice of a professional announcer…. All respondents heard the questions asked identically

That means this was a robopoll. One issue with Robopolls is that they tend to systematically produce a sample that has lower participation from younger voters. I can say that typically we see 2-5% of voters in the youngest category of our robopolls. To get them to the 15% seen in the SurveyUSA results, that requires significant weighting, which can amplify your error.

Second, this statement,

Where necessary, respondents were weighted using the most recent US Census estimates for age, gender, ethnic origin and region, to align the sample to the population.

My first response to the weighting scenario is that I prefer to weight to known and projected composition of the Georgia voter poll, which can be very different from the census numbers. But setting that aside, we must consider the sample size of the sub-sample that is these younger voters.

With a reported sample size of 550 voters, of whom 15% are in the 18-34 age group, our effective sample size when discussing those voters is 82.5 voters. We’ll be generous and round that up to 83. Checking my handy-dandy Margin of Error calculator, we see that the margin of error for a sample size of 83 respondents is a whopping +/-10.76 points.

In a scenario where Michelle Nunn has a lead of +5 among 18-34 voters, a +/-10.76 margin of error renders that 5-point lead meaningless. And that’s giving SurveyUSA the benefit of the doubt that their repondent pool includes close to 15% of 18-34 year olds. If, as I suspect is the case, they got more like 5% respondents in that age group, their actual sample size of 18-34 year olds is more like 27.5 voters, and the Margin of Error among that group is more like +/- 18.52 points.

This is where knowing the importance of Margin of Error calculations comes in handy, and where understanding survey methodology and analysis can prevent you from drawing embarrassing wrong conclusions. It’s also why when I publish results of a robopoll, you’re more likely to see a sample size in the range of 1500+ respondents.

This is a great example of where a little knowledge and a poll can be dangerous.