What are the polls really showing in Sudbury?

As by-elections go, the one in the provincial riding of Sudbury in Ontario has been pretty dramatic. Accusations of skulduggery, a Liberal-turned-independent candidate mixing it up, and a floor-crosser that managed to ford both the ideological gap between two parties as well as the federal/provincial line.

Some of the polls have been showing a close race between Liberal candidate Glenn Thibeault and the NDP's Suzanne Shawbonquit, with independent Andrew Olivier earning a big chunk of the vote. Other polls have shown Thibeault with a relatively comfortable lead. What gives?

At first glance, it is very puzzling that the polls by Forum Research and Oraclepoll Research can differ so much. Both firms have been in the field twice at exactly the same time. Oraclepoll was in for three to four days, with the Forum polls right in the middle of those sampling periods. So timing should have nothing to do with it.

Forum's first poll on January 13 put the gap at just two points, with Shawbonquit narrowly ahead. Olivier scored just 1%. Then on January 21, the gap was three points for Thibeault as he dropped seven points and Shawbonquit fell 12. Olivier made a miraculous 21-point gain, which Forum called a surge.

Meanwhile, Oraclepoll gave Thibeault a lead of 13 points on January 12-15 at 39% to 26% for Shawbonquit, with Olivier at 19%. On January 20-22, the gap was 16 points (42% to 26%), with Olivier at 20%. In other words, while Forum was showing a major shift, Oraclepoll was showing stability.

So who's right?

There are many reasons to trust Oraclepoll's numbers more than Forum's. Here's why.

Firstly, Forum is based out of Toronto and Oraclepoll is based out of Sudbury. Right out of the gate, the local firm has an advantage both in terms of local knowledge and a local area code showing up on the caller ID.

Secondly, and most importantly, Oraclepoll is conducting a superior survey. Forum is doing its polls via IVR, as it always does. It jumps into the field for a few hours on one evening, gathers its 500 to 800 responses, and reports the numbers a day or two later (response rates being what they are for Forum, it probably needs to call 25,000 to 80,000 households to get even those small samples).

Though it doesn't say explicitly, we can assume that Forum is not calling cell phones. This is because, unlike land lines, it is not possible to know for certain where a respondent on a cell phone is living. This is a problem in by-elections, as ridings do not entirely occupy an area code. This is not an issue in provincial or national polling. Cell phones can be included there without issue, and usually are.

Update: Commenter DL (see below) points out that targeting cell phones in a billing area is straight-forward, so Forum may indeed be calling cell phones. The inability to call cell phones pinpointed to a specific riding would be more of a problem in a city like Toronto where there are multiple ridings within one municipality.

Oraclepoll, on the other hand, is doing its polls with live-callers and it explicitly says that it is including cell phones in its sample (perhaps it has a list of local cell phones to call from). It is also in the field for several days, and according to its press release is calling back numbers where respondents did not answer up to five times before giving up. That is how a poll is supposed to be done.

Forum is getting responses from whoever is home between, say, 7 PM and 9 PM on January 13 and January 21 and is willing to answer a survey. Oraclepoll is doing everything that is reasonably possible to reach everyone it is calling. The potential for a biased sample is far lower.

Lastly, Forum's 1% result for Olivier on January 13 defies logic. At the time the poll was released, people were very surprised that Olivier, who had a relatively high profile, was scoring just 1%. That his number jumped from 1% to 22% in eight days makes little sense, particularly when Oraclepoll was showing his support levels to be steady at between 19% and 20% over the same period. While it is conceivable that the discrepancies in the results for the Liberals and NDP can be explained away by sampling issues and statistical probability, the discrepancy for Olivier's numbers are virtually impossible.

I don't know what happened in Forum's January 13th poll. But it simply doesn't make sense.

In that survey, Forum inquired as to whether Sudburians (Sudburites? Sudburers?) approved of the candidates on offer. The responses there just don't line up with the voting intentions numbers it recorded.

At the time, the poll found that 95% of respondents were aware or had an opinion of Thibeault, compared to 75% for PC candidate Paula Peroni and just 61% for Shawbonquit. But Olivier, who had just 1% support in the same poll, had 88% recognition - better than either the NDP or PC candidates.

That poll also suggested that 48% of all respondents (including undecideds and those who did not know him) approved of Thibeault, indicating that he was converting about 80% of sympathizers into supporters. Shawbonquit, with 41% approval, was converting an incredible 99% of sympathizers.

Peroni's approval was just 28%, as she was unknown to many respondents, but she was converting just 45% of sympathizers into supporters.

Olivier, though, had an approval rating of 60% of all of those sampled - and this is including those who did not have an opinion or did not know him. That was higher than any other candidate. According to the poll, we are thus supposed to believe that Olivier was converting just 2% of his sympathizers into supporters. If you can believe that, I have a bridge that spans the width of Lake Huron to sell you.

I think we can reasonably conclude that the January 13 poll by Forum can be tossed aside and discarded. Forum's January 21 poll has no similar problems, and support levels for Olivier and Peroni are similar to those recorded by Oraclepoll. Considering the sample sizes, Shawbonquit's support is also within the margin of error of these two polls, and Thibeault's is only slightly outside of it.

Due to the advantages that Oraclepoll's survey has over Forum's (not to mention the poor record Forum has in by-elections outside of Toronto), the benefit of the doubt should probably go to Oraclepoll. And that's even with the smaller sample size (the margin of error of decided voters would be just under +/- 6%).

If we use that margin of error to estimate support ranges, we'd get Thibeault at between 36% and 47%, Shawbonquit at between 21% and 31%, Olivier between 16% and 25%, and Peroni between 5% and 11%. That is probably as close as we can get to the truth at this stage of the campaign, which will come to a merciful end on Thursday.

Breaking down the seat numbers

The current federal projection puts both the Liberals and Conservatives in range of winning a plurality of seats, but neither appears capable to assemble the numbers for a majority government at this point. Let's take a closer look at this.

First, though, I invite you to check out my article today for the CBC. I've delved more deeply into the numbers for British Columbia, an interesting three-way race with multiple sub-regional contests, and Alberta, where some surprises could be in store for 2015.

Despite being behind in the vote projection by 1.3 points, the Conservatives are currently being awarded 136 seats. That puts them 34 seats short of an outright majority (169 seats is the threshold for 50%, but unless a sitting government wants to have the Speaker be from one of the opposition parties, 170 seats are needed).

Conservative ranges

If we look more broadly at the likely ranges, the Conservatives have between 117 and 155 seats. That still puts them 15 short of a majority government, a big gap to overcome. The polls would need to be off by a fair bit, and a lot of the close races would have to fall in their favour, for a majority government to be won at this stage of the game.

The chart on the left breaks down the Conservative ranges over the last three weeks. I've removed the Liberals and NDP ranges for clarity. Below I've done the same for the Liberal and NDP numbers.

If we extend things even further to the maximum ranges, the Conservatives are in play in 189 seats. That gives them a cushion of 19 for a majority government, but would require the polls to be significantly off the mark. The maximum projection, for instance, is based on national support of just over 41%.

The Conservatives' hopes of a majority government really come down to Ontario and British Columbia. Even in the best of circumstances, where the party takes 41% of the vote, their seat total would only be increased by 16 outside of these two provinces. That isn't enough. Instead, it is British Columbia, where they could boost their haul by nine seats, and particularly Ontario, where they could take 24 more seats, where a majority government will be won or lost.

However, that is not the situation the party currently finds itself in, well short of a majority and neck-and-neck with the Liberals in the seat ranges. Even if the party does manage to end up at the high end of their current range, or 155 seats, the Liberals and New Democrats would likely end up with a majority between them. Would a Conservative minority government really last very long?

Liberal ranges

The Liberals, currently at 126, are well short of a majority and are even short of the Conservatives for a plurality. But their likely range puts them between 107 and 144 seats, and so capable of placing first in the seat count. A majority government does not appear in the cards at the moment - the maximum for the party is 165, five short of the mark.

For the Liberals, whether they end up on the high end of the range depends almost entirely on Ontario and Quebec. They are currently close to their maximum in British Columbia, Alberta, the Prairies, and Atlantic Canada. Even if the party ends up at the maximum range of their likely support, they can only pick-up six extra seats in these regions.

These regions are actually more in danger of being disappointing. If things go badly, they could finish with 18 fewer seats in these regions than they are currently slated to take.

Ontario is the big prize, though, with the current range being between 45 and 63 seats. That accounts for roughly half of their national range.

There is slightly less potential for a breakthrough in Quebec, but it is nevertheless important as the party currently sits at between 20 and 30 seats there. The maximum goes up to 35. What this means is that if the Liberals are slightly under-performing in Ontario and Quebec, they cannot make up those losses with better performances in the rest of the country. Their election chances live and die in Ontario and Quebec.

For the New Democrats, placing first or second is currently not envisioned. Their overall take is slated at 72 seats at the moment. While that is down 31 seats from what they managed in 2011, it would still rank as their second-best performance in party history. But their sights are set higher than that.

NDP ranges

The ranges suggest the party is unlikely to return to pre-2011 levels, with between 54 and 87 seats. The lower end of the scale, however, would lead to more gains for the Bloc Québécois. This has the potential to complicate the arithmetic in a minority legislature. With the Bloc at a half-dozen seats or less, the NDP and Liberals can form a majority on their own. Once the party starts creeping closer to 10 seats, that becomes less likely. And the only route for a stronger Bloc is through the NDP.

The maximum ranges for the party do suggest it could end up with as few as 30 seats, but that assumes a major collapse, particularly in Quebec. Their higher range of 109 seats is more interesting, as it would likely lead them to finishing second in the seat count. Though the NDP's highest range is higher than the lowest range for the Liberals and Conservatives, it is not conceivable that all three can occur at the same time. If the NDP is as high as 26%, for instance, the Liberals are likely down quite a bit - which means the Conservatives win more seats. At this stage, an outright NDP victory does not seem plausible.

Quebec really is where everything will be decided for the NDP. Stronger performances outside of the province would only add six seats to the current projection of 72. A strong performance in Quebec could add nine seats alone (or 16 if things go very well). A poor performance in Quebec could drop the party by nine seats. A really bad performance could drop them by 22.

Ontario is also important for the NDP, but more in terms of avoiding a serious drop in seats. They are considered to be in play in just 18 ridings there at the moment, but could drop to eight with even a slight under-performance. Ontario is more about salvaging things for the NDP than making gains.

That about sums up the lay of the land at this point, with the likely ranges serving as a good guide to where things could potentially go in the coming months if no major shifts take place. The maximum ranges tell us what could happen if those major shifts take place. If that happens, of course, the numbers will be re-calculated to see where things could go from that new vantage point. It will be an interesting year.