not a beautiful or unique snowflake (nothings) wrote,
not a beautiful or unique snowflake
nothings

An article in Salon (subscription/day-pass required) comments that most presidential election polls are doubtful, in large part because of the methods used to judge "likely voters", who are weighted more heavily.

It also mentions briefly that the pollsters only get a 30% response rate these days--from only 30% of the phone numbers they call do they get an answer; the answer of the other 70% is unknown. What does this say about the error margin? The problem is these results start to approach those of self-selection; it may well be the case that the answers of people willing to answer the poll are not representative of the rest of the population.

Worst case, the entire 70% is voting one way or the other; so if the poll says, say, 52% Bush, 45% Kerry, that's only 30% of the voters. Say there are 1000 voters; then 700 didn't answer, and of the 300 who did, 156 said Bush, 135 said Kerry. Suppose the other 700 were all for Kerry; that would make it 835 to 156, or 84% to 16% Kerry. Suppose the other 700 were all for Bush; that would make it 856 to 135, or 86% to 14% Bush. So the margin of error, worst case, on the 52% to 45% is about +- 30%. Unsurprisingly, pollsters ignore this, but if you were running a physics experiment, that's how you'd have to evaluate it.

Ok, that's probably too extreme. How can we choose a reasonable distribution for the uncounted votes? Well, we can't. But we can hypothesize about some presumably truely independent variables: what percentage of people who are going to vote for each candidate are actually willing to talk to pollsters? (This is the thing we imagine might actually be inconsistent from one set of supporters to another; sort of shades of the infamous Dewey/Truman poll.) We could suppose that people for (just to pick one for this hypothetical, no bias intended) Kerry are unwilling to talk to pollsters 72% of the time, and people for Bush are unwilling to talk to pollsters 68% of the time--so that on average only 70% will talk to them. (I'm leaving out independents to simplify things here, but I will consistently assume that polls always return 5% undecided/independent. Also, it's not actually correct to expect the 72% and 68% to average to 70% unless the vote is dead even, but the details aren't important here.)

There's some gruesome math to work backwards here: from poll results and that response likelihood to the actual distribution of votes. In fact, when I tried to work it, I ended up with a pair of simultaneous equations that had no solution. Obviously, I messed up, but rather than spend more time on it, I figured it's easier to just run the results forward and produce some tables. Given the likelihood of responding to pollsters ("talkiness") below, and actual voting inclination ("actual votes"), we can easily compute the effective polling (plus the arbitrary 5% thrown in as mentioned). [Note that "talkiness" doesn't mean their willingness in talking to a pollster and being willing to admit their vote versus claiming undecided; nontalkiness is hanging up on the pollster or never answering the phone.]

I've produced that table below, and for each distribution of actual votes (the truth of who would get elected), boldfaced the line which has polls closest to Bush 50%, Kerry 45%. (Please note that I don't mean to assert anything by this choice; I just want to choose a slightly unbalanced result, and we could use any two names here to get the same result--even Dewey & Truman.)

Actual votes: Bush 60, Kerry 35
Bush talkiness: 20 | Kerry talkiness: 40 | Poll Bush: 43 | Poll Kerry: 51
Bush talkiness: 22 | Kerry talkiness: 38 | Poll Bush: 47 | Poll Kerry: 47
Bush talkiness: 24 | Kerry talkiness: 36 | Poll Bush: 50 | Poll Kerry: 44
Bush talkiness: 26 | Kerry talkiness: 34 | Poll Bush: 53 | Poll Kerry: 41
Bush talkiness: 28 | Kerry talkiness: 32 | Poll Bush: 56 | Poll Kerry: 37
Bush talkiness: 30 | Kerry talkiness: 30 | Poll Bush: 60 | Poll Kerry: 35
Bush talkiness: 32 | Kerry talkiness: 28 | Poll Bush: 62 | Poll Kerry: 32
Bush talkiness: 34 | Kerry talkiness: 26 | Poll Bush: 65 | Poll Kerry: 29
Bush talkiness: 36 | Kerry talkiness: 24 | Poll Bush: 68 | Poll Kerry: 26
Bush talkiness: 38 | Kerry talkiness: 22 | Poll Bush: 71 | Poll Kerry: 23
Bush talkiness: 40 | Kerry talkiness: 20 | Poll Bush: 73 | Poll Kerry: 21
Actual votes: Bush 55, Kerry 40
Bush talkiness: 20 | Kerry talkiness: 40 | Poll Bush: 38 | Poll Kerry: 56
Bush talkiness: 22 | Kerry talkiness: 38 | Poll Bush: 42 | Poll Kerry: 52
Bush talkiness: 24 | Kerry talkiness: 36 | Poll Bush: 45 | Poll Kerry: 49
Bush talkiness: 26 | Kerry talkiness: 34 | Poll Bush: 48 | Poll Kerry: 46
Bush talkiness: 28 | Kerry talkiness: 32 | Poll Bush: 51 | Poll Kerry: 43
Bush talkiness: 30 | Kerry talkiness: 30 | Poll Bush: 55 | Poll Kerry: 40
Bush talkiness: 32 | Kerry talkiness: 28 | Poll Bush: 58 | Poll Kerry: 36
Bush talkiness: 34 | Kerry talkiness: 26 | Poll Bush: 61 | Poll Kerry: 33
Bush talkiness: 36 | Kerry talkiness: 24 | Poll Bush: 63 | Poll Kerry: 31
Bush talkiness: 38 | Kerry talkiness: 22 | Poll Bush: 66 | Poll Kerry: 28
Bush talkiness: 40 | Kerry talkiness: 20 | Poll Bush: 69 | Poll Kerry: 25
Actual votes: Bush 50, Kerry 45
Bush talkiness: 20 | Kerry talkiness: 40 | Poll Bush: 33 | Poll Kerry: 61
Bush talkiness: 22 | Kerry talkiness: 38 | Poll Bush: 37 | Poll Kerry: 57
Bush talkiness: 24 | Kerry talkiness: 36 | Poll Bush: 40 | Poll Kerry: 54
Bush talkiness: 26 | Kerry talkiness: 34 | Poll Bush: 43 | Poll Kerry: 51
Bush talkiness: 28 | Kerry talkiness: 32 | Poll Bush: 46 | Poll Kerry: 48
Bush talkiness: 30 | Kerry talkiness: 30 | Poll Bush: 50 | Poll Kerry: 45
Bush talkiness: 32 | Kerry talkiness: 28 | Poll Bush: 53 | Poll Kerry: 41
Bush talkiness: 34 | Kerry talkiness: 26 | Poll Bush: 56 | Poll Kerry: 38
Bush talkiness: 36 | Kerry talkiness: 24 | Poll Bush: 59 | Poll Kerry: 35
Bush talkiness: 38 | Kerry talkiness: 22 | Poll Bush: 62 | Poll Kerry: 32
Bush talkiness: 40 | Kerry talkiness: 20 | Poll Bush: 65 | Poll Kerry: 29
Actual votes: Bush 45, Kerry 50
Bush talkiness: 20 | Kerry talkiness: 40 | Poll Bush: 29 | Poll Kerry: 65
Bush talkiness: 22 | Kerry talkiness: 38 | Poll Bush: 32 | Poll Kerry: 62
Bush talkiness: 24 | Kerry talkiness: 36 | Poll Bush: 35 | Poll Kerry: 59
Bush talkiness: 26 | Kerry talkiness: 34 | Poll Bush: 38 | Poll Kerry: 56
Bush talkiness: 28 | Kerry talkiness: 32 | Poll Bush: 41 | Poll Kerry: 53
Bush talkiness: 30 | Kerry talkiness: 30 | Poll Bush: 44 | Poll Kerry: 50
Bush talkiness: 32 | Kerry talkiness: 28 | Poll Bush: 48 | Poll Kerry: 46
Bush talkiness: 34 | Kerry talkiness: 26 | Poll Bush: 51 | Poll Kerry: 43
Bush talkiness: 36 | Kerry talkiness: 24 | Poll Bush: 54 | Poll Kerry: 40
Bush talkiness: 38 | Kerry talkiness: 22 | Poll Bush: 57 | Poll Kerry: 37
Bush talkiness: 40 | Kerry talkiness: 20 | Poll Bush: 61 | Poll Kerry: 33
Actual votes: Bush 40, Kerry 55
Bush talkiness: 20 | Kerry talkiness: 40 | Poll Bush: 25 | Poll Kerry: 69
Bush talkiness: 22 | Kerry talkiness: 38 | Poll Bush: 28 | Poll Kerry: 66
Bush talkiness: 24 | Kerry talkiness: 36 | Poll Bush: 31 | Poll Kerry: 63
Bush talkiness: 26 | Kerry talkiness: 34 | Poll Bush: 33 | Poll Kerry: 61
Bush talkiness: 28 | Kerry talkiness: 32 | Poll Bush: 36 | Poll Kerry: 58
Bush talkiness: 30 | Kerry talkiness: 30 | Poll Bush: 40 | Poll Kerry: 55
Bush talkiness: 32 | Kerry talkiness: 28 | Poll Bush: 43 | Poll Kerry: 51
Bush talkiness: 34 | Kerry talkiness: 26 | Poll Bush: 46 | Poll Kerry: 48
Bush talkiness: 36 | Kerry talkiness: 24 | Poll Bush: 49 | Poll Kerry: 45
Bush talkiness: 38 | Kerry talkiness: 22 | Poll Bush: 52 | Poll Kerry: 42
Bush talkiness: 40 | Kerry talkiness: 20 | Poll Bush: 56 | Poll Kerry: 38
Actual votes: Bush 35, Kerry 60
Bush talkiness: 20 | Kerry talkiness: 40 | Poll Bush: 21 | Poll Kerry: 73
Bush talkiness: 22 | Kerry talkiness: 38 | Poll Bush: 23 | Poll Kerry: 71
Bush talkiness: 24 | Kerry talkiness: 36 | Poll Bush: 26 | Poll Kerry: 68
Bush talkiness: 26 | Kerry talkiness: 34 | Poll Bush: 29 | Poll Kerry: 65
Bush talkiness: 28 | Kerry talkiness: 32 | Poll Bush: 32 | Poll Kerry: 62
Bush talkiness: 30 | Kerry talkiness: 30 | Poll Bush: 34 | Poll Kerry: 60
Bush talkiness: 32 | Kerry talkiness: 28 | Poll Bush: 37 | Poll Kerry: 57
Bush talkiness: 34 | Kerry talkiness: 26 | Poll Bush: 41 | Poll Kerry: 53
Bush talkiness: 36 | Kerry talkiness: 24 | Poll Bush: 44 | Poll Kerry: 50
Bush talkiness: 38 | Kerry talkiness: 22 | Poll Bush: 47 | Poll Kerry: 47
Bush talkiness: 40 | Kerry talkiness: 20 | Poll Bush: 51 | Poll Kerry: 43


So what this table says, if the actual election were to go 60% Bush, 35% Kerry, but Bush supporters only told their votes to pollsters 24% of the time and Kerry supporters did 36% of the time, then we'd get a poll result of 50% to 46% (assuming 5% of all polling was undecided/independent). Similarly, if the actual election were 55% Kerry, 40% Bush, polls would produce the same results with talkiness reversed.

Here is just the results for the poll results producing 50% Bush, 45% Kerry, computed slightly more accurately than the table above. "Votes" is the actual distribution of voters; "talkiness" again is the likelihood of giving any opinion to a poll.

Votes: Bush 60, Kerry 35 | Talkiness: Bush 23.5, Kerry: 36.5
Votes: Bush 55, Kerry 40 | Talkiness: Bush 27.0, Kerry: 33.0
Votes: Bush 50, Kerry 45 | Talkiness: Bush 30.0, Kerry: 30.0
Votes: Bush 45, Kerry 50 | Talkiness: Bush 33.0, Kerry: 27.0
Votes: Bush 40, Kerry 55 | Talkiness: Bush 36.0, Kerry: 24.0
Votes: Bush 35, Kerry 60 | Talkiness: Bush 39.5, Kerry: 20.5


Note that it takes a "smaller" shift in talkiness to compensate for a shift in votes; changing from 55/40 to 50/45 actual votes is a 10-percent change (5 percent on both sides), which requires a 6 percent change (3 percent on both sides) in talkiness. However, since the talkiness numbers don't add to 100, it's hard to really meaningfully compare these; if we start from the 30-30 case, and change each of Bush and Kerry's talkiness by 10% of their own value (33/27), we change the votes by 10% of their own value as well.

Of course, all this is just meaningless speculation; we have no idea whether there are any biases like this. But the point here is that with such a large non-reporting sample--70%--any significant bias in response rate produces a significant change in the polled result. This would not be true if the response rate was high; with a 90% response rate, a 10% bias in response rate would have much less effect. [At a 90% total response rate, larger response biases aren't even possible in close races; if person A's supporters had only a 50% response rate, yet the total poll got a 90% response rate, then person A can't have that more than 20% of the actual vote anyway.]

Unfortunately, there's no way to measure this response rate inconsistency, since the details of who isn't responding to polls isn't open to polling. With no clue whether there are biases in the response rate--and no good reason to think there aren't--there's really no reason to take polls with such low response rates seriously. Of course, a polling company doesn't really have much interest in pointing out how useless their numbers are, so they're not going to challenge themselves on this. Welcome to America. Go capitalism.
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 5 comments