This blog has been updated on the election night. Its clear there was no election-altering gain to Democrats so this blog is moot. I will leave the words here but am overstiking them, as no longer relevant.
The Communities Dominate blog is the best source for politics-related data systems in use. It has the only published case study comparing the old method and the new, tested against each other, with all the metrics. Its about the only reason anyone in politics might have heard of the CDB blog or ‘that Tomi Ahonen’. But everybody in the data room of Hillary’s team and the few data guys on Trump’s team knows me and has read this blog. this is the definitive blog for data systems used in politics. The analysis of data systems in politics, written on this blog, are referenced in many published books about politics!
You are right now reading some utter bullshit by people from one side, and getting total radio silence from the other side. Those willing to speak are throwing about big words trying to impress like ‘Big Data’ and 'psychographics' and '100 point propensity scores'.
They are BULLSHITTING YOU. This is THE DEFINITIVE BLOG about those topics, at least where it relates to political campaigns. So this blog article on 4 November is just a brief commentary now, before Election Day, to clarify some basics. Let me start with an analogy
Putting bigger sails onto a sailship does not magically convert that ship into a steamship. It would need radical redesign, including adding a big heavy engine and its fuel, and punching a hole into the boat, through which the propeller shaft has to come, to put the propeller onto your ship. But a steam ship not only can move faster than fastest sailships, a steamship can do what sailships cannot, such as steam directly against he wind, and steam at full speed even when there is no wind at all.
Trump has put bigger sails on a sailship. But in the last election, Obama introduced the world’s first steamship and it utterly crushed Romney’s sailship (that was the fastest sailship ever made). And now, in the interim four years, Hillary has put a bigger ENGINE in Obama’s steamship. That is the data wars. Trump used an obsolete design but put bigger sails on it. Hillary took the bleeding edge design, and put an even better engine into it. This is not going to be close. The data wars will contribute 4% to Hillary’s victory margin on November 8.
ITS AN ARMS RACE
The data system wars are an ‘arms race’. Both sides are building their systems (or WERE building it, now four days from the election, they are just running them). We saw the systems used to powerful effect in 2004 (Howard Dean campaign). Then to DEVASTATING effect in 2008 (Obama vs McCain). The data wars DECIDED the race in 2012. (Obama vs Romney). The data system disadvantage by Republicans was so immense and decisive, it was listed as one of the most important things the GOP has to change for 2016, or else it could not win the election.
This is not what some obscure tech author of 12 books tells you on his silly blog. This what Reince Priebus wrote in the 2012 Republican Party ‘Autopsy’ report of Mitt Romney’s loss. They had been OUTCLASSED by Obama’s data machine. A machine that we calculated on this blog in 2012, right after the election - that it delivered 4 out of every 5 votes that Obama won. And Obama’s margin of victory was 5%. But the LAST POLLING of the weekend and Monday just before election day, said it was a 1% race. The WHOLE ELECTION MARGIN of victory for Obama - was generated with the data machine. The whole election margin! The Data machine pushed a 1% election nail-biter, into a clear dominating crushing victory of 5% for Obama. The data machine did that. I reported it on this blog, calculated all the math. The Republican Autopsy said they were outclassed and lost because of the data wars. The US campaigns KNOW this. The Ted Cruz campaign said the advantage of a modern data system is between 2.5% and 5% on election day. They essentially CONFIRMED MY MATH. All data engineers in US politics know this. The data nerds. And they all have read my blog.
TWO RIVAL TECHNOLOGIES
The two systems are not built on the same principle. Its like comparing a car to an airplane. Both have an engine. Both can move people. Except one is far faster and can do things the other cannot. A car cannot drive over the seas. An airplane can fly over the ocean exactly as easily as flying over a corn field. What Romney had, was a good car. What Obama had was the first airplane.
What Trump team now tries to tell us, is that they have a ‘Big Data’ system. BULLSHIT. Its a traditional psychographic scoring model. Like all database systems of the past. It makes guesses about us. It does that very well (they bought the service from a British company called Cambridge Analytica, paying at least 5 million for their data and analytics). What Trump has is a car with an airplane engine. It can't fly.
The Trump system is LARGE. That is not what ‘Big’ means in Big Data. Big Data is a revolutionary database methodology by which every indivdual consumer is individually measured, and typically CONTACTED. Cambridge Analytica has 230 US million voters in its database. It has NOT contacted them. It has NOT MEASURED their performance. It has run 100,000 surveys of voters to get profiles. And then assigns profiles and segments this database based on their magazine subscriptions and cable TV subscriptions and credit card scores and their address, age, educational background and voter registration etc information. That is not Big Data. That is traditional database of a consumer population. Its just very large.
Big Data contacts EACH VOTER INDIVIDUALLY. Yes. Each voter individually. That is what Obama did, that was so radical in 2012. Not that they had 100 point scores for each voter. That was a tech detail, like yes, a car has an engine and an airplane has an engine, and both engines have horsepower. You could run a car on an airplane engine or an airplane on a car engine (back in the early days of propeller-driven planes obviously, not jet engines). Both have an engine. But if you put an airplane engine in a car, it will not suddely FLY. You have to BUILD an AIRPLANE to fly, with wings, etc.
So the METHOD that gets us to ‘Big Data’ is precise technical issue, to do something on consumer customer base ever attempted before 2012. To go through the trouble of contacting every consumer (ie voter for an election). Seeing what they said on Facebook, did they download the YouTube video, did they forward it. To call that voter, talk to them. To go visit their home, talk to them. To send emails and SMS text messages to those voters. CONTACT THEM PERSONALLY. That is Big Data.
That was the bleeding edge of 2012. Nobody has this. Facebook, Google, Amazon, they did not bave Big Data until 2012, the Obama team decided to create this system (now many bleeding edge companies are doing similar systems, obviously).
Romney built a traditional database (like Trump now). It was built on the best data insights of the Republican party, and the known performance of the 2008 Obama system. It was built with a huge data staff of 30 data professionals, cost 50 million dollars, built with Microsoft. It was a rush-project done in six months. It was finished so close to election day, they didn’t have time to test it (the system failed in North Carolina on Election Day where they had to run blind).
Obama built a bleeding-edge data concept on something never done in politics before. Even though they WON the data wars of 2008. They decided to do something radical. They spent 100 million dollars and had a staff that reached 120 data scientists and built a whole array of systems over 18 months. The Obama system was built with Google, Facebook and Amazon as tech partners. It has a massively parallel database system spread over HP servers and resulted in one of the 20 largest databases ON THE PLANET. Just built for one election.
We measured the performance of the two on this blog. The Romney system DID PERFORM. It BOOSTED Republican turnout over what John McCain had in 2008. The Romney team did believe they had won the election, powered by this powerful new data machine. Except it was a total useless toy effort, compared with what Obama’s machine. If Romney had managed to build a better Bow-and-Arrow weapon, Obama had introduced the rifle. If Romney had built a faster propeller-driven airplane, Obama introduced the jet engine. If Romney built a faster sailship, Obama introduced the steamship. If Romney made a stronger explosive dynamite, Obama detonated the first atom bomb. There is NO CONTEST between these two methods.
The two are night-and-day. The Obama system is 4.5 TIMES BETTER. Not 4.5 PERCENT better (in a year when the election was decided by 5%). It was not 45% better. the Obama machine is 4.5 TIMES BETTER. 350% better. Understand. One side has something that is not 100% better or 200% better than your system. Its something that is 350% better than yours. When you spend a million dollars on a TV ad campaign that boosts your voter turnout by 10,000 votes. And the other side uses their system to run a better TV ad campaign - that also costs 1 million dollars, but they get 45,000 votes!!! 4.5 time better! 350% better!.
That is what Obama built in 2012. That is what was called ‘Narwhal’. That was using the bleeding-edge tech called Big Data. I wrote several blogs about it then and did a total analysis of the two systems compared head-to-head, written for MARKETING people, in other industries than politics, that I published here in early 2015. This is THE DEFINITIVE article about what is Big Data and why its the new era in databases. Why it makes psychographics like used by Romney (and now Trump) obsolete.
Obsolete.
VERY VERY SIMPLE EXPLANATION
Lets take a very simple example for you. I am a 56 year old white Finnish man. So, if you use Trump’s method, psychographics, you will find from my demographic data that I am a man. Thus you won’t need to target ads for women, like selling tampons, to me. It does work. Its far better than nothing. If also will find out that I am unmarried. No need to send ads about babyfood to me. And yes, there is very smart analytics that would detect, out of my purchase or web browsing behavior if suddenly I was expecting a baby haha (if say I was expecting suddenly to become a first-time father). Yes. the predictive modeling can be remarkably accurate.
Now the problems. I’m a 56 year old white man. From Finland. If you profile me, against any other 56 year old white Finnish MEN, anywhere on the planet, you will find they wear blue jeans. Finnish men hate to dress up ‘formally’. We are notorious about that. Some of the sloppiest dressers of Europe (some would not be so kind, and just say THE most sloppiest dressers - Finnish men, the women are gorgeous but us Finnish men, gosh). Ok. So he’s older, pretty affluent, lets target Tomi’s clothing ads as premium-brand blue jeans (and T-shirts and casual wear). This is what psychographics gets you. Those who have seen me, they know. Tomi doesn’t OWN blue jeans. I wear tailored suits and tailored WHITE shirts with loud (some say ugly) silk ties. Even on holidays and on vacations. I am so not-Finnish-male about this. Your psychogrphic model fails you on the EXCEPTIONS to the rules. Like in this case, what I happen to like to wear. For MOST men it will score them correctly. Me? They get wrong every time.
You think that's an isolated example? Take my music. I’m 56 years old, white guy, from Finland. What is my music taste. It HAS to be rock music. Hard rock, heavy metal. ZZ Top, Rolling Stones, Status Quo, Led Zeppelin, etc. Has to be. Or it could be classical music or jazz. But very likely rock. Heavy metal rock music. Essentially all of my peers of my age like that kind of music. Me? My fave music is rap music. Its ‘black’ music that is for YOUNGER generations. There are plenty of rap music fans (and a rap music scene) in Finland but they are all younger. Nobody age 56 in Finland ‘likes’ rap music! But I do. And I fell in love with rap BEFORE I moved to live in New York City. I was a fan of the music from its birth when most said rap is not music. Again the psychographics systems work in most cases, but fail with the exceptions.
A FASTER SAILSHIP OR REVOLUTIONARY STEAMSHIP
Psychographics works. It is FAR better than nothing. Romney built the most powerful database system the Republicans had ever had. They used it for targeting Get-Out-The-Voter efforts in 2012, with the help of 30 data professionals and Microsoft - and they truly built the state-of-the-art of psychographics. And it worked.
Except it is obsolete. They are building the most streamlined oceanliner sailship, when Obama has introduced steamships that can forge straight against the wind, and even run full speed when there is a lull and no wind at all. The steamship killed the sailing ships as commercial vessels (but sailing then became a leisure activity). That is the kind of revolution we saw in 2012. And the Republican party admitted it. They knew they had been defeated and knew they had to build a system like Obama’s.
Well. What did Trump do? He said he doesn’t believe in data. He ran most of his primary race without data. After he won his nomination, the Cruz team showed Trump what all they could do, and when Manafort came in to provide some sane professional campaign elements to Trump, he did get convinced to do some datamining. So he hired Cambridge Analytica, who has what? A Big Data system? No. Cambridge Analytica is a psychographic database segmentation system exactly like Romney’s except its a bit more modern. It is very good. But Trump is again doing a faster sailship. It’s because Trump is a cheapskate and learned that he’d have to pay 100 million dollars to do anything near what Obama had and Hillary now has. So its far easier to just outsource that work to the British and get their insights.
What was it used for? It was used to target TV ads, online ads, and help in the fund-raising targeting (the Nigerian money scam type of emails that Trump sends to his gullible supporters). Yeah. It works for that.
BUT IN HILLARYLAND
Meanwhile. What has Hillary done? She has vastly expanded the Big Data based system she has, the old Obama 2012 system now far upgraded and expanded. It drives every activity they do. So one of the details. They run 1,000 simulations every night on their system of the election. Not one simulation every night. Not ten. Not one hundred. They run 1,000 simulations every night! They test EVERYTHING. Now that Melania talked about bullying, what is the impact to the voters without any response? What if we run the Trump is a bully ad? What if we run the Hillary is the woman standing up against bullies. What if we run Melania’s words on a new experimental ad against TRUMP. And so forth. They run 1,000 simulations every night. This is a campaign run on DATA. Not on intuition or gut feeling or what pops into Trump’s head any moment. Hillary’s team is the most professional campaign in history. Driven by data.
So how do they get their data into the system? They run a DAILY POLL that is MASSIVE. They run over 100,000 surveys EVERY DAY. They run over 10,000 interviews in EVERY BATTLEGROUND STATE every day. That is perfect precision on the voter feeling on any issue, in every state, to every conveivable voting demographic. But that is only the ‘thermometer’. Is there a problem. Out of the survey snapshots, every day, they run a deeper analysis on whatever is right now in the news, the FBI letter, Trump’s Goon Squad of Voter Intimidation, or Melania promising to end bullying or Trump and his Moscow money connection etc. They have DETAILED surveys every night on a fracition of the total survey size, at about 1 in 10. So they still get instant and statistically very accurate feedback on ALL issues in politics, daily.
While we’re on polls. Trump fired his pollster and isn’t paying his bills for polling work already done. He owes his pollster 750,000 dollars. That would get you about one survey in the 13 battlegound states of the kind that Hillary runs EVERY DAY. But the Trump internal poll told him the truth, he has lost the election (which is why Trump got so morose back when this dawned on him, and then his response is to kill the messenger so don’t pay the pollster).
EVERY VOTER IS INDIVIDUALLY CONTACTED
Sorry, back to Hillary. After the polling, they do IN-PERSON TALKS to every voter (in battleground states). Sometimes they knock on a door, they see the Trump-Pence yard sign, and the door is opened by an older man who says, “I’m voting for Trump and so is my wife, go away or I will call the police!” And thats the length of that discussion. They STILL CONTACTED the voter, PERSONALLY. Its absolutely vital for Big Data process, to know every consumer, whether they love you or hate you. (I talk about this in my big primer about this process).
So they visit some homes. And others they contact by telephone or email or Facebook or SMS text message. And then they constantly refine and expand that knowledge base. Who is a supporter, has forwarded YouTube videos (on EITHER SIDE)? Who attended their rally where Bernie Sanders spoke? Who is volunteering time, who has contributed money, etc etc etc? Not just voters, they also have ineligble voters who are supporters, like say kids too young to vote. Or a spouce who is not a US citizen, etc. Can still be a Hillary supporter. There was a story that they found women who were secretly Hillary supporters, who were married to Trump-supporting Republican husbands - who themselves were Republican women - but now will vote for Hillary. The Hillary team knows EVERY VOTER (in battleground states) including these hidden voters. In 2012 the Obama team contacted undecided voters on average 5 TIMES PERSONALLY in the battleground states! That is how you win elections!
On election day, Hillary knows every voter who is registered to vote. They know the registered voters who have already voted early. They know of registered voters who have not voted yet, if they are Hillary supporters. They know what those voters do, if they are decided, and what issues matter to them. The Hillary team even knows which methods of contact that given individual voter prefers for CONTACT. Do they want to be contacted by phone and talk to someone or contacted via email or Facebook or - most popular of course, is SMS text messaging. Hillary’s process will not waste any calls or contacts on people who have already voted, or who are not registered, or who don’t like Hillary and would vote for Trump.
LIKE STEALING CANDY FROM A BABY
Trump has psychographic voter profiles and segments. They have decided that because Tomi Ahonen is a 56 year old white man who lives in a rich part of town, he is a likely Republican voter. And they target me for a get-out-the-voter effort. Even as (if I was a US citizen) they would then activate a voter FOR THE OTHER SIDE. The Hillary team knows NOT TO prod the enemy, don’t incite them to go and vote. If its a Trump supporter, don’t prod them. The Hillary team can target every single one of the 200 million registered voters in the USA, individually and personally. Yes. true target to the precision of one (in battleground states). They will not bother with that precision for practical reasons (a practical resolution of segmentation is 10,000 segments, it means that every single person is targeted so ‘uniquely’ that in their lifetime they are unlikely to get to know another person of that identical segment). If 130 million people vote, and say 70 million vote for Hillary, it means they will end up doing about granularity to individual segments of about 7,000 voters. Any TV or radio or Facebook or YouTube or Twitter campaign that gets to about 7,000 voters - IS WORTH DOING by the Hillary machine!
Trump does one standard TV ad in all 13 battleground states, that reaches 20 million voters. Hillary’s machine can get 2,900 SEPARATE marketing messages AUTOMATICALLY GENERATED to target 2,900 varieties of are you white or black or Hispanic or Asian-American; are you woman or man; are you gay or straight; are you Christian or Jewish or Mormon or Muslim or atheist or Hindu or Buddhist; are you in college or retired or working; are you unemployed, are you ex-military, are you without healthcare insurance, etc etc etc. The Hillary system is a radar. The Trump system is a binoculars. Yes you can aim your cannon better with binoculars than without but if you have radar, you will always win.
Ok. I wanted to put this blog out here for now. What Trump has is an updated version of what failed for Romney in 2012. What Hillary has is a VASTLY expanded and refined version of the new way to do political targeting. She will get AT LEAST 4% more votes because of this system on Tuesday, over what the last polls tell us. At least 4% more votes. If the average of polls for this weekend say, say its a 7% election for Hillary, then add 4% and Hillary will win by 11%. Take this to the bank. We have MEASURED it and if you don’t believe it, read these blogs. The math is all there.
Definitive Study of Obama vs Romney data wars 2012
Application of Big Data lesson of 2012 applied to modern marketing in 2015 (if you have not read about these matters, start here, its not too technical)
The Latest Insights and Updates to the Data Wars of 2012, written in late 2015
I will of course write a full analysis of this race after we get the details, after the election is over. The data machine is the biggest engine driving Hillary’s victory. It is a ‘secret weapon’. Some of the info of 2012 was so well hidden, the RESPONSE to Obama 2012 built by say Ted Cruz in 2015, was not strong enough to deal with what Hillary has now, because they hid part of their ability back then.
But this blog will get you all the data we can find, and crunch the numbers again, and we’ll see how big the impact of the data advantage was for the Democrats. Its about time the Republicans stop playing around with toys, and get serious and build a proper Big Data system, not a toy using psychographics. And what Cambridge Analytics has, is a large database yes, its not Big Data. Its nowhere close. Its just the old thing we saw with Romney, with a bit more precision. A better sailship. A faster propeller-driven airplane. What Trump is driving is children’s toy car, the battery-driven kind of ‘Ferrari’ the child can sit in and drive in the yard. Hillary has a proper RACE CAR. And its now been fine-tuned and has an even more powerful engine. Oh, and she has a TEAM of engineers to keep it in peak performance. And now these two will race each other. I am laughing so hard. Psychographics? That is what Trump bought? In 2016? After the Romney Autopsy? I am laughing hysterically...
Regards my mention of the supposed Right Wing media, here's an interesting essay.
http://www.alternet.org/election-2016/fox-news-creates-alternate-reality-then-cnn-perpetuates-it
Posted by: Wayne Borean | November 05, 2016 at 05:41 PM
More info on subject Winter posted:
The anti-Clinton insurgency at the FBI, explained
The recent series of FBI leaks are particularly worrisome because they raise the prospect of a state security agency equipped with the full resources and investigative might of the federal government working to interfere in the elections. The FBI is so powerful — it can, with court approval, issue subpoenas, tap phones, intercept emails and conduct round-the-clock surveillance — that even a small coterie of its agents can find ways of influencing the political process. That’s the kind of thing we normally see in autocracies like Egypt or Turkey, not here in the United States.
http://www.vox.com/world/2016/11/5/13525698/fbi-clinton-trump-leaks-server-email-scandal
Posted by: grouch | November 05, 2016 at 05:45 PM
For the first time in over 40 years, I am ignoring individual candidates in an election and voting strictly AGAINST the Republican party.
Comey should be indicted and the whole FBI should be investigated to find those who also violated the Hatch Act.
Posted by: grouch | November 05, 2016 at 05:53 PM
Why Vladimir Putin’s Russia Is Backing Donald Trump
By Kurt Eichenwald
http://www.newsweek.com/donald-trump-vladimir-putin-russia-hillary-clinton-united-states-europe-516895
Posted by: grouch | November 05, 2016 at 06:05 PM
The GOTV effort has been discussed quite thoroughly here. Still curious about an aspect: I recall that HRCs GOTV is deployed mostly in the battleground states. How much can it help in down ticket battles, which likely happen in many states, also other than the presidential battlegrounds?
Is it right to assume, now that presidential battlegrounds are moving to E.g. Arizona and even Texas, that HRCs campaign has the GOTV machine prepared for the voters in these states? Likely not, since volunteers need to be collected for a longer period, and if there is no action in the state there is no significant volunteer signup either.
Posted by: willebra | November 05, 2016 at 06:49 PM
Early voter surge in NV, FL, and now North Carolina:
"Meck County burned it up today where some lines had 500 people waiting still at 1:00 when polls were supposed to close."
"Voters STILL in [email protected]:20"
http://www.democraticunderground.com/12512588525
Posted by: Millard Filmore | November 05, 2016 at 09:04 PM
Maher slams mainstream media election coverage: ‘It’s not funny anymore. Media — do your f*cking job’
http://www.rawstory.com/2016/11/maher-slams-mainstream-media-election-coverage-its-not-funny-anymore-media-do-your-fcking-job/
Posted by: grouch | November 05, 2016 at 10:01 PM
[It's up to women, Hispanics and African-Americans to save us angry old white men from our self-destruction. It may be up to the international press to save us from our own greed-crazed incompetent media.]
Donald Trump: The unauthorized database of false things
The Star’s Washington Bureau Chief, Daniel Dale, has been following Donald Trump’s campaign for months. He has fact checked thousands of statements and found hundreds of falsehoods
By Daniel DaleWashington Bureau
Tanya TalagaGlobal Economics Reporter
The Star’s Daniel Dale has been fact checking Trump’s public statements on the U.S. election campaign trail since September 15. Below, find the complete list of the false statements Dale has found.
After that (very long) list, Tanya Talaga examines the errors, exaggerations and lies for patterns. Some remain hard to explain. Click here to jump directly to this analysis.
https://www.thestar.com/news/world/uselection/2016/11/04/donald-trump-the-unauthorized-database-of-false-things.html
Posted by: grouch | November 05, 2016 at 10:09 PM
Hi Everybody
Who is the best pollster? I've been doing some scoring of the pollsters and as we can expect possibly 19 final polls to come out for the Presidential election, national result, 4-way race, in the next 2 possibly 3 days, I decided to do a bit of math.
I evaluated all 166 polls on RCP of the 4-way race, against their contemporary rivals, to see who were accurate and who were wildly off. I measured each poll against the nearest 6 polls, 3 before and 3 after, and then measured how MANY of that given pollster's total polls were off - someone like Rasmussen is off 44% of the time vs its rivals at that same time - off by 3%. And get this, 17% of the time (one in six Rasmussen polls) is off by a massive 5% vs the rivals who report at that same time. (And to make things really rally nasty, Rasmussen has gradually introduced its polling error to be ever more friendly to Trump.. Yeah, right after the conventions they were essentially accurate, and every three weeks they pushed their polls out more and more to lie in favor of Trump and against Hillary. Pretty sneaky, eh? Good propagandists, Rasmussen, good propagandists. They'd get a good job if Trump ever became Dictator).
Anyway. I rated all pollsters and then looked how many have done multiple polls of the race so far and think that 19 pollsters are expected to release a poll on Sunday or Monday (or conceivably even as late as Tuesday morning). So what can we make of them? I decided to rank them for us, based on this year's performance, against their peers. Five are excellent, not one poll that was wrong vs peer average. In alphabetical order:
CBS News/NY Times
Economist/YouGov
GWU/Battleground
Pew
PPP
Those polls are likely to be within a point or two of reality. The rest had increasing degree of error. This second set had mostly good polls but a single or two outliers. Whichever way they have a systematic bias, I have indicated it as (D) for biased for Democrats/Hillary and (R) for biased in favor of Republicans/Trump.
McClatchy/Marist (D)
NBC News/SurveyMonkey (D)
Quinnipiac
Then we have the bad pollsters. These had severaral bad polls and could be far off from the mean. Take these pollsters less seriously:
Associated Press/Gfk (D)
Bloomberg (R)
CNN/ORC (R)
Gravis (R)
Monmouth (D)
NBC News/Wall Street Journal (D)
USA Today/Suffolk
And lastly we have the four tracking polls. These should definitely have their final trackers out Monday, possibly even Tuesday, and may give out numbers on Sunday as well. I've included their biases too
ABC News Tracker (wild, goes any which way)
IBD/TIPP (R)
Rasmussen (R)
Reuters/Ipsos (D)
So that should help us decypher the polls to know how much weight to give any one pollster's final number. The golden 5 this year are CBS News/NY Times; Economist/YouGov;
GWU/Battleground; Pew; PPP. I would say if at least 4 of these 5 give a number, we can take the average of this group as the best single set of numbers for the end. But even more accurate will be to take all national polls out Sunday and Monday, when all are out, and average them all. BUT its possible we get say Sunday a strongly R-balanced set of polls with D-balanced pollsters coming later on Monday, then take this 'golden 5' set as the best guide of where the race is. They've been the most accurate this year.
A final word. I am slightly hopeful that Gallup might surprise us with a final Presidential poll. To me, Gallup is synonymous with polling and we've always had Gallup polls in the past Presidential elections and it seems wrong not to have them polling this race this time. I am (slightly) hopeful they'll surprise us with a last poll now, just before the election. I think they stopped their polls last year because of the thing that polls were used to decide who got into the Presidential debates, if I remember correctly. I would love to see a Gallup poll, they usually were always very good. And they are polling on other election matters currently like the favorability of the candidates, and various issues. So its not like Gallup suddenly has stopped doing their core business haha. But if we're lucky, we might get Gallup tossed in there, as the 20th pollster and even with no track record, I'd add G to the Golden five, making it then the Golden Six.
(that has nothing to do with the fact I've been employed by the Gallup organization of Finland back in the day, haha, for several years as an interviewer and senior interviewer/test interviewer and then also in survey questionnaire design. Or maybe it has a bit haha).
Tomi Ahonen :-)
Posted by: Tomi T Ahonen | November 06, 2016 at 02:03 AM
Hi willebra
(Welcome to the discussion!)
Great question about GOTV. So first, on battleground vs non-battleground. The 'machine' is designed to work in battleground states and it is labor-intensive, needs human volunteers to make it work. Some of the work can be done remotely (telephone call from California to a voter in Nevada) but much of it has to be done locally (walk to a house, knock on the door, drive a voter to a voting location, monitor voting activity, etc). So the main benefit and gain is only in states where it is deployed. In nine states they used it last time and we know they now have deployed it too. They are Colorado, Florida, Iowa, Ohio, Nevada, New Hampshire, Pennsylvania, Virginia, Wisconsin.
North Carolina is certain to be added and has full deployment. Thats ten states where we know the GOTV effort will be fully in use for Hillary.
There was possibly/probably GOTV effort in New Mexico last time but not this time. Some residual effect may be that they can still benefit, I do not know. There is a possibility that this year's Republican hopes in Michigan and Minnesota may have had a Democratic effort built there, modestly, to defend, and Maine and Nebraska have their single voting district in both cases that is competitive, Maine for Republicans, Nebraska for Democrats, those may have some effort deployed. I don't think these are going to matter much in these states.
Arizona is different. Arizona was a late deployment but Hillary spent about half-level TV ad money in AZ vs the other 'proper' battleground states and she devoted roughly half-level staff there, and we could expect a significant, but reduced effect of GOTV in AZ.
In the other 'competitive' or pretend-competitive states like Texas, Utah, Missouri, Georgia and Indiana - the effect of the GOTV will be barely above that of the nation on the whole. And the nation, will not have much of an effect at all. To get the power, you have to have people doing the work so what they HAVE been doing, for example, is putting people on busses, and driving them from blue states to battleground states, to do volunteer work there.
I'd say full effect in 10 states: Colorado, Florida, Iowa, Ohio, Nevada, New Hampshire, North Carolina, Pennsylvania, Virginia, Wisconsin. And about 'half effect' in Arizona.
But on down-ticket races, gosh, its almost the same. If you can convice a random voter to go vote for Hillary on whatever reason - then you'll certainly also say, remember to vote for the down-ticket Democrats too, she needs them - and if the voter was activated by the GOTV, its very likely going to get them to also vote or the Democratic ticket in down-ballot races too. Note, many are motivated to go to vote simply out of civic duty or out of a passion on any possible people or issues, and if they had already voted, the Hillary GOTV effort will bypass that voter and not bug them... So there can be/will be plenty of voters who won't 'get the message' to vote the ticket, but those who are contacted by the Hillary GOTV effort, will definitely be pushed to do the down-ticket Democrat vote too. I think yes, very powerful on that. Differing from what the Republicans are doing.
What we will eventually see, is how they performed in the deployed battleground states vs non-battleground states. I measured from state-voter data, that the 2012 Obama effort pushed voter Registration up 2% more in states that the GOTV effort was deployed, and the final vote up 5% vs states where it was not deployed. This year should see somewhat better numbers first because they know how to do this, and secondly because the opposition is weaker on the GOTV effort this time.
Tomi Ahonen :-)
Posted by: Tomi T Ahonen | November 06, 2016 at 02:26 AM
Hi Brent
(welcome to the discussion)
Gosh, good question. Never thought of that! First off, part of the data entry comes via their field workers on election day, via their smartphones, and onto an app/browser based entry system. It could be hacked and flooded and denied info entry. That would act like a military radar jammer, that turns the radar screen full of false echoes, so there is a real airplane target there, among all the dots, but you have no idea which if the dots is the enemy airplane and the 500 other dots are just ghosts...
And with that, the reaction-ability of the GOTV effort would be at least disrupted (at least locally) and then would need to work on past historical data rather than live data. Would certainly diminish its power. I would guess, with the series of Russian hacking going on, that the Hillary data team has had time to prepare at least some levels of security and work-arounds (ie for example using the voice calls to telephone in some of the data if needed)
Secondly, the scripts. There is an advanced telephone-banking solution that is web based and links to the phone numbers, the profiles of the voters, and offers the suitable telephone script for the volunteer. This could be disrupted, but I would guess (I have no idea) that it might have data cached in nearby servers and not be subject to a single-point-of-failure. It could be, but unlikely to be. So then the disruption(s) could be local or regional but probably not national. And the internet being what it is, then some rerouting could get to secure sites. VPN etc...
A large part of the last-day work is going in the opposite direction, from the HQ to the voters, ie voter-activation reminders, SMS text messages, emails, Facebook reminders etc. And some are going to be guided phone calls from volunteers to voters. These would seem like less vulnerable to a denial-of-service hack but again, I'm not a security expert enough to really know.
We've had an increasing series of internet hacking type of disruptions worldwide, like several airlines have had massive fleet groundings etc. I would expect that the Hillary data team know this is their most critical tool and it will need to be up on that one day, that they'd have given it considerable protection (even more, haha, with Trump's invitation to get Russia to hack this election). That said, haha, wouldn't that be ironic to have a 100 million dollar superduper targeting machine, that is down all day on the one day in four years, that it was supposed to function...
(you're giving me nightmares!)
Thanks for the comment, I hope that helped a little bit. You no doubt are also knowledgable about that side of IT, how would you comment on it?
Tomi Ahonen :-)
Posted by: Tomi T Ahonen | November 06, 2016 at 02:40 AM
Hi Tomi
Regarding the possibility of a DDoS attack. It depends on the infrastructure on which Clinton's system has been deployed. If the system lives in a strong cloud, it is much more difficult to take it down. Maybe the attackers can slow down the system a bit, but it's very unlikely completely shut it down. In the cloud multiple servers get pooled and so they are able to share the load.
Posted by: cornelius | November 06, 2016 at 03:27 AM
Hi Everybody
Actually, just looked into it. Gallup said they won't do any general election polls, lets not expect one. And Pew said it won't do a final poll either. Both are being mocked by pollster-watchers as being too chicken to publish the one poll on which their accuracy can be measured. LAME...
So cut that down to 18 expected polls and only 4 of the 'gold standard' haha.. Too bad.
Tomi Ahonen :-)
Posted by: Tomi T Ahonen | November 06, 2016 at 03:36 AM
Here is a comforting read for those who are on the edge of their seats.
Nate Silver Is Unskewing Polls — All Of Them — In Trump’s Direction
http://www.huffingtonpost.com/entry/nate-silver-election-forecast_us_581e1c33e4b0d9ce6fbc6f7f
The gist of the article is that the election is not as close as some pundits say.
Posted by: cornelius | November 06, 2016 at 04:02 AM
Hi Tomi -- Love the blog. Your writing helps me calm down. Last night I was so worried about a fascist takeover that I downed a Percocet and watched "Triumph of the Will" on the a very large flatscreen, flat on my back, in a darkened room. Without subtitles. No, but seriously, this stuff is scary. There's also an excellent new series on Amazon Prime called "The Man in the High Castle," which is an "alternative history" show, in which the Nazis won the war and took over the US east of the Rockies, and the Japanese took over west of the Rockies. The show is set in 1962, and Hitler is still alive. The story line has to do with some sort of resistance movement afoot. It is very well done, incredibly brutal, and utterly disturbing.
About the race, all over cable today there was chatter about Michigan being "in play" and Clinton camp is "worried." However, I note that out of the whole rust belt in 2012, Obama won Michigan with the highest margin, 9 points, 54-45. 400,000 votes. More than WI, OH, PA. Just Trump desperation (and nothing better to do).
Remember in 2012 when all the Romney supporters were predicting victory? Nothing compared to Trump idiots. It's 10X. If you're on Twitter, you know what I'm talking about. They are foaming-at-the-mouth sure Trump will win. Every poll is a conspiracy to them. I think they know this is their last chance, they can see the demographics closing in on them. And a lot of them are very old, probably Goldwater supporters.
BTW, Tomi, very good writing on the "Big Data" HRC advantage. Nobody else even mentions that, but it is fascinating.
Posted by: Daktari | November 06, 2016 at 05:57 AM
@Daktari: "The story line has to do with some sort of resistance movement afoot. It is very well done, incredibly brutal, and utterly disturbing."
For another disturbing movie on that subject, check out the Russian title "Come and See"
https://en.wikipedia.org/wiki/Come_and_See
http://www.imdb.com/title/tt0091251/
Posted by: Millard Filmore | November 06, 2016 at 06:13 AM
@Daktari;
Maybe it's me, or maybe it's my age, or maybe it's just that I get /really/ impatient with someone trying to stretch a novel into anything longer than a miniseries. In any event, I still think the original novel by Phillip K Dick iss better than the series. Too bad nobody wants to read anything that's more than five years old anymore. :-(
That said, I would LOVE to see someone attempt Fritz Lieber's Fahfrd and the Grey Mouser as a TV series. Several novellas and short stories that could be squeezed into hour long segments, plenty of high adventure, lots of pretty girls, the heroes win and lose at every turn... what's not to like? ;-)
Posted by: sgtrock | November 06, 2016 at 06:31 AM
@sgtrock -- Thanks for letting me know. I will definitely read that. I do read! Also in that genre is the great Philip Roth novel "The Plot Against America."
As for adaptations, no one ever says "Man, that movie was so much better than the book!" Never happens. So, I figure I'll just watch the movie first. OK, sometimes.
Posted by: Daktari | November 06, 2016 at 06:45 AM
Wellll, maybe 2001. Great book, absolutely fantastic movie. :-)
Posted by: sgtrock | November 06, 2016 at 07:07 AM
@cornelius
Ryan Grim needs to learn the difference between random error and systematic error. Imagine 100 people measure how the length of shadows change with time to calculate when sunset will be (and all of them are using the correct formula!). Shadows have blurred edges. The people will have to use their judgement to decide where the shadow ends, so they will get different lengths. This is a random error, which should give a short answer about as often as a long answer, so using the average will give a better estimate of when the sun will set.
Now imagine I shorten the pendulum in a stopwatch so it gains 3 seconds per minute, and that all 100 people use that stopwatch. This is a systematic error, and all 100 people will be putting a time that is 5% too big into the equation. When sunset does happen, it will be 5% earlier (measured by an accurate grandfather clock) than the average prediction. The next day, we can repeat the experiment with the same equipment, allow for a 5% systematic error and get a good prediction for the time when the sun sets.
Nate Silver 'correcting' poll results based on how well different polls predicted previous elections is both good and bad. Firstly, there would have to be systematic error in each pollster's method. That would show up as each pollster being wrong by a consistent amount for each election. I have no idea if this really happened. The big assumption is that each pollster precisely repeats the method used during the last election. That is remotely possible, but I could easily believe each pollster 'corrected' his method in an attempt to avoid repeating the mistakes they made last time. Again, this would show up in previous results as each pollster being wrong by a random amount each time.
The next big assumption is that the pollsters selected a formula, made phone calls then calculated a result. Some pollsters could have selected a result, done some phone calls then calculated the required formula. There are plenty of other confounding factors. The candidates are different, so people could have different reasons for giving false answers over the phone. Perhaps the people making the phone calls are not following the script, or are entering results that do not match the conversation.
Correcting for systematic error is the right thing to do - if you have evidence that there really is a systematic error to correct.
Posted by: Isceald Glede | November 06, 2016 at 07:10 AM