This blog has been updated on the election night. Its clear there was no election-altering gain to Democrats so this blog is moot. I will leave the words here but am overstiking them, as no longer relevant.
The Communities Dominate blog is the best source for politics-related data systems in use. It has the only published case study comparing the old method and the new, tested against each other, with all the metrics. Its about the only reason anyone in politics might have heard of the CDB blog or ‘that Tomi Ahonen’. But everybody in the data room of Hillary’s team and the few data guys on Trump’s team knows me and has read this blog. this is the definitive blog for data systems used in politics. The analysis of data systems in politics, written on this blog, are referenced in many published books about politics!
You are right now reading some utter bullshit by people from one side, and getting total radio silence from the other side. Those willing to speak are throwing about big words trying to impress like ‘Big Data’ and 'psychographics' and '100 point propensity scores'.
They are BULLSHITTING YOU. This is THE DEFINITIVE BLOG about those topics, at least where it relates to political campaigns. So this blog article on 4 November is just a brief commentary now, before Election Day, to clarify some basics. Let me start with an analogy
Putting bigger sails onto a sailship does not magically convert that ship into a steamship. It would need radical redesign, including adding a big heavy engine and its fuel, and punching a hole into the boat, through which the propeller shaft has to come, to put the propeller onto your ship. But a steam ship not only can move faster than fastest sailships, a steamship can do what sailships cannot, such as steam directly against he wind, and steam at full speed even when there is no wind at all.
Trump has put bigger sails on a sailship. But in the last election, Obama introduced the world’s first steamship and it utterly crushed Romney’s sailship (that was the fastest sailship ever made). And now, in the interim four years, Hillary has put a bigger ENGINE in Obama’s steamship. That is the data wars. Trump used an obsolete design but put bigger sails on it. Hillary took the bleeding edge design, and put an even better engine into it. This is not going to be close. The data wars will contribute 4% to Hillary’s victory margin on November 8.
ITS AN ARMS RACE
The data system wars are an ‘arms race’. Both sides are building their systems (or WERE building it, now four days from the election, they are just running them). We saw the systems used to powerful effect in 2004 (Howard Dean campaign). Then to DEVASTATING effect in 2008 (Obama vs McCain). The data wars DECIDED the race in 2012. (Obama vs Romney). The data system disadvantage by Republicans was so immense and decisive, it was listed as one of the most important things the GOP has to change for 2016, or else it could not win the election.
This is not what some obscure tech author of 12 books tells you on his silly blog. This what Reince Priebus wrote in the 2012 Republican Party ‘Autopsy’ report of Mitt Romney’s loss. They had been OUTCLASSED by Obama’s data machine. A machine that we calculated on this blog in 2012, right after the election - that it delivered 4 out of every 5 votes that Obama won. And Obama’s margin of victory was 5%. But the LAST POLLING of the weekend and Monday just before election day, said it was a 1% race. The WHOLE ELECTION MARGIN of victory for Obama - was generated with the data machine. The whole election margin! The Data machine pushed a 1% election nail-biter, into a clear dominating crushing victory of 5% for Obama. The data machine did that. I reported it on this blog, calculated all the math. The Republican Autopsy said they were outclassed and lost because of the data wars. The US campaigns KNOW this. The Ted Cruz campaign said the advantage of a modern data system is between 2.5% and 5% on election day. They essentially CONFIRMED MY MATH. All data engineers in US politics know this. The data nerds. And they all have read my blog.
TWO RIVAL TECHNOLOGIES
The two systems are not built on the same principle. Its like comparing a car to an airplane. Both have an engine. Both can move people. Except one is far faster and can do things the other cannot. A car cannot drive over the seas. An airplane can fly over the ocean exactly as easily as flying over a corn field. What Romney had, was a good car. What Obama had was the first airplane.
What Trump team now tries to tell us, is that they have a ‘Big Data’ system. BULLSHIT. Its a traditional psychographic scoring model. Like all database systems of the past. It makes guesses about us. It does that very well (they bought the service from a British company called Cambridge Analytica, paying at least 5 million for their data and analytics). What Trump has is a car with an airplane engine. It can't fly.
The Trump system is LARGE. That is not what ‘Big’ means in Big Data. Big Data is a revolutionary database methodology by which every indivdual consumer is individually measured, and typically CONTACTED. Cambridge Analytica has 230 US million voters in its database. It has NOT contacted them. It has NOT MEASURED their performance. It has run 100,000 surveys of voters to get profiles. And then assigns profiles and segments this database based on their magazine subscriptions and cable TV subscriptions and credit card scores and their address, age, educational background and voter registration etc information. That is not Big Data. That is traditional database of a consumer population. Its just very large.
Big Data contacts EACH VOTER INDIVIDUALLY. Yes. Each voter individually. That is what Obama did, that was so radical in 2012. Not that they had 100 point scores for each voter. That was a tech detail, like yes, a car has an engine and an airplane has an engine, and both engines have horsepower. You could run a car on an airplane engine or an airplane on a car engine (back in the early days of propeller-driven planes obviously, not jet engines). Both have an engine. But if you put an airplane engine in a car, it will not suddely FLY. You have to BUILD an AIRPLANE to fly, with wings, etc.
So the METHOD that gets us to ‘Big Data’ is precise technical issue, to do something on consumer customer base ever attempted before 2012. To go through the trouble of contacting every consumer (ie voter for an election). Seeing what they said on Facebook, did they download the YouTube video, did they forward it. To call that voter, talk to them. To go visit their home, talk to them. To send emails and SMS text messages to those voters. CONTACT THEM PERSONALLY. That is Big Data.
That was the bleeding edge of 2012. Nobody has this. Facebook, Google, Amazon, they did not bave Big Data until 2012, the Obama team decided to create this system (now many bleeding edge companies are doing similar systems, obviously).
Romney built a traditional database (like Trump now). It was built on the best data insights of the Republican party, and the known performance of the 2008 Obama system. It was built with a huge data staff of 30 data professionals, cost 50 million dollars, built with Microsoft. It was a rush-project done in six months. It was finished so close to election day, they didn’t have time to test it (the system failed in North Carolina on Election Day where they had to run blind).
Obama built a bleeding-edge data concept on something never done in politics before. Even though they WON the data wars of 2008. They decided to do something radical. They spent 100 million dollars and had a staff that reached 120 data scientists and built a whole array of systems over 18 months. The Obama system was built with Google, Facebook and Amazon as tech partners. It has a massively parallel database system spread over HP servers and resulted in one of the 20 largest databases ON THE PLANET. Just built for one election.
We measured the performance of the two on this blog. The Romney system DID PERFORM. It BOOSTED Republican turnout over what John McCain had in 2008. The Romney team did believe they had won the election, powered by this powerful new data machine. Except it was a total useless toy effort, compared with what Obama’s machine. If Romney had managed to build a better Bow-and-Arrow weapon, Obama had introduced the rifle. If Romney had built a faster propeller-driven airplane, Obama introduced the jet engine. If Romney built a faster sailship, Obama introduced the steamship. If Romney made a stronger explosive dynamite, Obama detonated the first atom bomb. There is NO CONTEST between these two methods.
The two are night-and-day. The Obama system is 4.5 TIMES BETTER. Not 4.5 PERCENT better (in a year when the election was decided by 5%). It was not 45% better. the Obama machine is 4.5 TIMES BETTER. 350% better. Understand. One side has something that is not 100% better or 200% better than your system. Its something that is 350% better than yours. When you spend a million dollars on a TV ad campaign that boosts your voter turnout by 10,000 votes. And the other side uses their system to run a better TV ad campaign - that also costs 1 million dollars, but they get 45,000 votes!!! 4.5 time better! 350% better!.
That is what Obama built in 2012. That is what was called ‘Narwhal’. That was using the bleeding-edge tech called Big Data. I wrote several blogs about it then and did a total analysis of the two systems compared head-to-head, written for MARKETING people, in other industries than politics, that I published here in early 2015. This is THE DEFINITIVE article about what is Big Data and why its the new era in databases. Why it makes psychographics like used by Romney (and now Trump) obsolete.
Obsolete.
VERY VERY SIMPLE EXPLANATION
Lets take a very simple example for you. I am a 56 year old white Finnish man. So, if you use Trump’s method, psychographics, you will find from my demographic data that I am a man. Thus you won’t need to target ads for women, like selling tampons, to me. It does work. Its far better than nothing. If also will find out that I am unmarried. No need to send ads about babyfood to me. And yes, there is very smart analytics that would detect, out of my purchase or web browsing behavior if suddenly I was expecting a baby haha (if say I was expecting suddenly to become a first-time father). Yes. the predictive modeling can be remarkably accurate.
Now the problems. I’m a 56 year old white man. From Finland. If you profile me, against any other 56 year old white Finnish MEN, anywhere on the planet, you will find they wear blue jeans. Finnish men hate to dress up ‘formally’. We are notorious about that. Some of the sloppiest dressers of Europe (some would not be so kind, and just say THE most sloppiest dressers - Finnish men, the women are gorgeous but us Finnish men, gosh). Ok. So he’s older, pretty affluent, lets target Tomi’s clothing ads as premium-brand blue jeans (and T-shirts and casual wear). This is what psychographics gets you. Those who have seen me, they know. Tomi doesn’t OWN blue jeans. I wear tailored suits and tailored WHITE shirts with loud (some say ugly) silk ties. Even on holidays and on vacations. I am so not-Finnish-male about this. Your psychogrphic model fails you on the EXCEPTIONS to the rules. Like in this case, what I happen to like to wear. For MOST men it will score them correctly. Me? They get wrong every time.
You think that's an isolated example? Take my music. I’m 56 years old, white guy, from Finland. What is my music taste. It HAS to be rock music. Hard rock, heavy metal. ZZ Top, Rolling Stones, Status Quo, Led Zeppelin, etc. Has to be. Or it could be classical music or jazz. But very likely rock. Heavy metal rock music. Essentially all of my peers of my age like that kind of music. Me? My fave music is rap music. Its ‘black’ music that is for YOUNGER generations. There are plenty of rap music fans (and a rap music scene) in Finland but they are all younger. Nobody age 56 in Finland ‘likes’ rap music! But I do. And I fell in love with rap BEFORE I moved to live in New York City. I was a fan of the music from its birth when most said rap is not music. Again the psychographics systems work in most cases, but fail with the exceptions.
A FASTER SAILSHIP OR REVOLUTIONARY STEAMSHIP
Psychographics works. It is FAR better than nothing. Romney built the most powerful database system the Republicans had ever had. They used it for targeting Get-Out-The-Voter efforts in 2012, with the help of 30 data professionals and Microsoft - and they truly built the state-of-the-art of psychographics. And it worked.
Except it is obsolete. They are building the most streamlined oceanliner sailship, when Obama has introduced steamships that can forge straight against the wind, and even run full speed when there is a lull and no wind at all. The steamship killed the sailing ships as commercial vessels (but sailing then became a leisure activity). That is the kind of revolution we saw in 2012. And the Republican party admitted it. They knew they had been defeated and knew they had to build a system like Obama’s.
Well. What did Trump do? He said he doesn’t believe in data. He ran most of his primary race without data. After he won his nomination, the Cruz team showed Trump what all they could do, and when Manafort came in to provide some sane professional campaign elements to Trump, he did get convinced to do some datamining. So he hired Cambridge Analytica, who has what? A Big Data system? No. Cambridge Analytica is a psychographic database segmentation system exactly like Romney’s except its a bit more modern. It is very good. But Trump is again doing a faster sailship. It’s because Trump is a cheapskate and learned that he’d have to pay 100 million dollars to do anything near what Obama had and Hillary now has. So its far easier to just outsource that work to the British and get their insights.
What was it used for? It was used to target TV ads, online ads, and help in the fund-raising targeting (the Nigerian money scam type of emails that Trump sends to his gullible supporters). Yeah. It works for that.
BUT IN HILLARYLAND
Meanwhile. What has Hillary done? She has vastly expanded the Big Data based system she has, the old Obama 2012 system now far upgraded and expanded. It drives every activity they do. So one of the details. They run 1,000 simulations every night on their system of the election. Not one simulation every night. Not ten. Not one hundred. They run 1,000 simulations every night! They test EVERYTHING. Now that Melania talked about bullying, what is the impact to the voters without any response? What if we run the Trump is a bully ad? What if we run the Hillary is the woman standing up against bullies. What if we run Melania’s words on a new experimental ad against TRUMP. And so forth. They run 1,000 simulations every night. This is a campaign run on DATA. Not on intuition or gut feeling or what pops into Trump’s head any moment. Hillary’s team is the most professional campaign in history. Driven by data.
So how do they get their data into the system? They run a DAILY POLL that is MASSIVE. They run over 100,000 surveys EVERY DAY. They run over 10,000 interviews in EVERY BATTLEGROUND STATE every day. That is perfect precision on the voter feeling on any issue, in every state, to every conveivable voting demographic. But that is only the ‘thermometer’. Is there a problem. Out of the survey snapshots, every day, they run a deeper analysis on whatever is right now in the news, the FBI letter, Trump’s Goon Squad of Voter Intimidation, or Melania promising to end bullying or Trump and his Moscow money connection etc. They have DETAILED surveys every night on a fracition of the total survey size, at about 1 in 10. So they still get instant and statistically very accurate feedback on ALL issues in politics, daily.
While we’re on polls. Trump fired his pollster and isn’t paying his bills for polling work already done. He owes his pollster 750,000 dollars. That would get you about one survey in the 13 battlegound states of the kind that Hillary runs EVERY DAY. But the Trump internal poll told him the truth, he has lost the election (which is why Trump got so morose back when this dawned on him, and then his response is to kill the messenger so don’t pay the pollster).
EVERY VOTER IS INDIVIDUALLY CONTACTED
Sorry, back to Hillary. After the polling, they do IN-PERSON TALKS to every voter (in battleground states). Sometimes they knock on a door, they see the Trump-Pence yard sign, and the door is opened by an older man who says, “I’m voting for Trump and so is my wife, go away or I will call the police!” And thats the length of that discussion. They STILL CONTACTED the voter, PERSONALLY. Its absolutely vital for Big Data process, to know every consumer, whether they love you or hate you. (I talk about this in my big primer about this process).
So they visit some homes. And others they contact by telephone or email or Facebook or SMS text message. And then they constantly refine and expand that knowledge base. Who is a supporter, has forwarded YouTube videos (on EITHER SIDE)? Who attended their rally where Bernie Sanders spoke? Who is volunteering time, who has contributed money, etc etc etc? Not just voters, they also have ineligble voters who are supporters, like say kids too young to vote. Or a spouce who is not a US citizen, etc. Can still be a Hillary supporter. There was a story that they found women who were secretly Hillary supporters, who were married to Trump-supporting Republican husbands - who themselves were Republican women - but now will vote for Hillary. The Hillary team knows EVERY VOTER (in battleground states) including these hidden voters. In 2012 the Obama team contacted undecided voters on average 5 TIMES PERSONALLY in the battleground states! That is how you win elections!
On election day, Hillary knows every voter who is registered to vote. They know the registered voters who have already voted early. They know of registered voters who have not voted yet, if they are Hillary supporters. They know what those voters do, if they are decided, and what issues matter to them. The Hillary team even knows which methods of contact that given individual voter prefers for CONTACT. Do they want to be contacted by phone and talk to someone or contacted via email or Facebook or - most popular of course, is SMS text messaging. Hillary’s process will not waste any calls or contacts on people who have already voted, or who are not registered, or who don’t like Hillary and would vote for Trump.
LIKE STEALING CANDY FROM A BABY
Trump has psychographic voter profiles and segments. They have decided that because Tomi Ahonen is a 56 year old white man who lives in a rich part of town, he is a likely Republican voter. And they target me for a get-out-the-voter effort. Even as (if I was a US citizen) they would then activate a voter FOR THE OTHER SIDE. The Hillary team knows NOT TO prod the enemy, don’t incite them to go and vote. If its a Trump supporter, don’t prod them. The Hillary team can target every single one of the 200 million registered voters in the USA, individually and personally. Yes. true target to the precision of one (in battleground states). They will not bother with that precision for practical reasons (a practical resolution of segmentation is 10,000 segments, it means that every single person is targeted so ‘uniquely’ that in their lifetime they are unlikely to get to know another person of that identical segment). If 130 million people vote, and say 70 million vote for Hillary, it means they will end up doing about granularity to individual segments of about 7,000 voters. Any TV or radio or Facebook or YouTube or Twitter campaign that gets to about 7,000 voters - IS WORTH DOING by the Hillary machine!
Trump does one standard TV ad in all 13 battleground states, that reaches 20 million voters. Hillary’s machine can get 2,900 SEPARATE marketing messages AUTOMATICALLY GENERATED to target 2,900 varieties of are you white or black or Hispanic or Asian-American; are you woman or man; are you gay or straight; are you Christian or Jewish or Mormon or Muslim or atheist or Hindu or Buddhist; are you in college or retired or working; are you unemployed, are you ex-military, are you without healthcare insurance, etc etc etc. The Hillary system is a radar. The Trump system is a binoculars. Yes you can aim your cannon better with binoculars than without but if you have radar, you will always win.
Ok. I wanted to put this blog out here for now. What Trump has is an updated version of what failed for Romney in 2012. What Hillary has is a VASTLY expanded and refined version of the new way to do political targeting. She will get AT LEAST 4% more votes because of this system on Tuesday, over what the last polls tell us. At least 4% more votes. If the average of polls for this weekend say, say its a 7% election for Hillary, then add 4% and Hillary will win by 11%. Take this to the bank. We have MEASURED it and if you don’t believe it, read these blogs. The math is all there.
Definitive Study of Obama vs Romney data wars 2012
Application of Big Data lesson of 2012 applied to modern marketing in 2015 (if you have not read about these matters, start here, its not too technical)
The Latest Insights and Updates to the Data Wars of 2012, written in late 2015
I will of course write a full analysis of this race after we get the details, after the election is over. The data machine is the biggest engine driving Hillary’s victory. It is a ‘secret weapon’. Some of the info of 2012 was so well hidden, the RESPONSE to Obama 2012 built by say Ted Cruz in 2015, was not strong enough to deal with what Hillary has now, because they hid part of their ability back then.
But this blog will get you all the data we can find, and crunch the numbers again, and we’ll see how big the impact of the data advantage was for the Democrats. Its about time the Republicans stop playing around with toys, and get serious and build a proper Big Data system, not a toy using psychographics. And what Cambridge Analytics has, is a large database yes, its not Big Data. Its nowhere close. Its just the old thing we saw with Romney, with a bit more precision. A better sailship. A faster propeller-driven airplane. What Trump is driving is children’s toy car, the battery-driven kind of ‘Ferrari’ the child can sit in and drive in the yard. Hillary has a proper RACE CAR. And its now been fine-tuned and has an even more powerful engine. Oh, and she has a TEAM of engineers to keep it in peak performance. And now these two will race each other. I am laughing so hard. Psychographics? That is what Trump bought? In 2016? After the Romney Autopsy? I am laughing hysterically...
That doesn't sound too bad. :)
Without Florida Trump can try all he wants and still not win.
Posted by: Tester | November 07, 2016 at 07:03 PM
In the interests of full disclosure for the good of the country, I may or may not have sent one or more emails which included either or both of the words "Hillary" and "Clinton" at some time or times between the beginning of the Internet and now.
Should I expect a House investigative committee subpoena soon?
Posted by: grouch | November 07, 2016 at 07:32 PM
@grouch
Do they need a subpoena? It's much simpler to just call the Russian hackers and voila your e-mails!
Posted by: cornelius | November 07, 2016 at 07:47 PM
Putin news:
Montenegro Prosecutor: Russian Nationalists Behind Alleged Coup Attempt
Included plans to assassinate prime minister because of his government’s bid to join NATO
http://www.wsj.com/articles/montenegro-prosecutor-russian-nationalists-behind-alleged-coup-attempt-1478473032
Posted by: Winter | November 07, 2016 at 07:49 PM
Hi Everybody
We have first 9 final polls out. I have blogged about them. I'm making it a 'live blog' for Monday up to early Tuesday hours, to add any polls as they come in.
I have calculated the scenarios for you to allocate the undecided voters, so you can see the range of most-Trump positive scenario vs most-Hillary positive outcome and see where the race seems to be, as measured by the first 9 pollsters. And short version: they say it's a 5.6% race haha..
Tomi Ahonen :-)
Posted by: Tomi T Ahonen | November 07, 2016 at 08:21 PM
Hi Everybody
Remember when I said that after he's gone from office, when Hillary is in, the Republicans will start to pine for Obama how he was such a great President after all?
Didn't need to wait that long. Glenn Beck of all people, has started that song now, one day before Hillary's victory. Beck finds that Obama had influenced Beck to become a better person (granted, he was starting from so low, anything would do that)
Yeah.. Another, yet another, of my predictions starting to come true haha
Tomi Ahonen :-)
Posted by: Tomi T Ahonen | November 07, 2016 at 08:31 PM
More Pootin news:
Russian warplanes keep buzzing the Baltics. Here’s how NATO scrambles
https://www.washingtonpost.com/world/europe/russian-warplanes-keep-buzzing-the-baltics-heres-how-nato-scrambles/2016/11/06/15c1ea6e-9df4-11e6-b552-b1f85e484086_story.html
"By the end of the night, Finland and Estonia said their airspace had been violated — and in the sea below, a powerful nuclear-capable missile system was on its way to a Russian naval base in the enclave of Kaliningrad."
Posted by: cornelius | November 07, 2016 at 09:04 PM