Revisiting Best Big Data Case Study: We Have New Details about Narwhal vs Orca Case Study (of US Elections 2012 and world's largest Big Data project for which there is performance data) (Updated)

We have some new performance data on Big Data. And we return to the best case study there exists on Big Data as the only major case that offers some details of its performance. Yes, its time to revisit Orca vs Narwhal, the battle of election-specialist database systems of the 2012 Presidential election. I promised you that some more details will eventually come out and we'll revisit that topic when it does Today we find new data courtesy of Jim Messina.

NOTE - update on 15 Dec to this story based on fresh news from Cruz campaign use of Big Data in 2016 cycle (see end of this blog)

Most who use Big Data today are aware of how incredibly powerful it is, so Big Data is some of the most secret competitive advantages of those entities - which almost invariably are either business corporations or some secretive government agencies. So to get 'performance data' on Big Data is still very murky and rare. Except for one case study. What was three years ago the sixth largest database built on the planet and momentarily the most advanced customer relationship system ever made, is now obsolete and has been junked and is being replaced. It was President Obama's Narwhal, the data mining, voter insights and predictive modelling data system that turned a razor-thin election that the last polls said should be 1% either way, into a blowout victory of 5% for Obama against Romney. And note, Romney's team also built a rival system called Orca, so it was a battle of data mining systems. Romney team's Orca was evolution of past systems and methods, the most expensive database built by the Republicans and bigger and more powerful than what Obama did four years prior, in 2008. But Obama's team had 'supersized' their effort with Narwhal which was the most expensive political/election database system ever created. In two years, with 100 programmers, and assistance by Google, Amazon, Facebook, etc the system cost 400 million dollars to build and it utterly crushed the Republican Orca in the process. And brought Obama his clear decisive election victory of 5% rather than a race that was being counted still the following day.

So as the Obama Narwhal system is now literally obsolete and useless, it has been junked (imagine investing 400 million dollars into a database that you use once and then throw away as obsolete?). But as its no longer the system in use, there are occasional tidbits of details that come out about it. I wrote the definitive blog about the performance of the two systems against each other, shortly after the election of 2012. Anyone interested in the power of Big Data in elections should read that blog. I also discussed in an earlier blog many of the tactics and specifics used by both campaigns. These two are the 'political primer' into use of big data, social media, mobile and... SMS text messaging... in politics and especially political campaigns and elections. Note, I separately wrote a blog about the power of Big Data where I revisited in only summary form, that Narwhal vs Orca case study. This anyone intersted in Big Data should read, as it is again, the most detail and performance-based blog article anywhere about a real world-class leading-edge Big Data system in use. With results measured, opposing methods contrasted and all published into the public domain. My analysis from those three blog articles has been referenced in many published books already about how the 2012 election was won and there is nothing else in the public domain with more detail about the Big Data/datamining and voter insights systems than those three articles on this blog.

JIM MESSINA GIVES SOME NEW DETAILS

With that, we go to Jim Messina. Who is he? He was Obama's Deputy Chief of Staff at the White House. He then went to run the re-election campaign for the 2012 election as the Campaign Manager. He's now supporting Hillary Clinton's campaign. And one of Messina's pet projects was the use of high tech to win elections. What we now know as Big Data. The Narwhal project was Jim's baby. Every single month of the Obama campaign's spending, mysterious Project Narwhal took the largest expenditure. The campaign to re-elect Obama literally put its money where its mind was, it believed Narwhal would be the biggest factor to ensuring re-election - as it was. What should have been a nail-biter 1% election either way, turned into a blow-out of 5% victory for Obama. And the three top political advisors of Obama, David Axelrod, Jim Messina and David Plouffe, who were all asked for election-night predictions by Obama on what will happen - all three guessed the election correct within one state (Plouffe was perfect). They relied on Big Data rather than polling. They were correct. Big Data supercedes polling. To really understand, its a long read but read first my primer on Big Data. Then read the two Narwhal performance blogs, the Big Data final numbers performance, and the Big Data tactics blog. With that, lets go to Messina. He was at a forum with Karl Rove and discussed various political issues in length and revealed a few previously unknown data points from project Narwhal. Geek Wire reports the key details. But these are very relevant.

66,000 ELECTION SIMULATIONS

If you are into modern strategy theory you know fully the power of simulations. A strategist - military or civilian - can test out various scenarios and map out responses ie 'strategies' to deal with various possibilities. I have written and spoken about simulations occasionally including in the signature book of this blog, Communities Dominate Brands, because we used the most powerful simulations system in telecoms, when I was head of Nokia's Business Consulting back 14 years ago. It was one of the keys to Nokia's competitive advantage and we used also our telecoms competition simulation in the training provided to telecoms operators/carriers (which incidentially was also the most powerful of its kind, one simulation lasted two days and needed a minimum of 16 people plus two administrators just to run one simulation. Similarly to how Narwhal was a series of databases and simulators and models, ours at Nokia Consulting was also that way. The 'Arbitrage' telecoms competition simulator was built out of the same engine as Nokia's massive MDF Master Design Framework the most complex telecoms operator/carrier business simulator; which then was based partly on the network dimensioning engine that Nokia network engineers used to actually build cellular networks, etc etc etc). So yeah, been there done that. And if you are say an Air Force and flying say the F18 Hornet with a flyaway cost of 50 million dollars and the highly maneouverable jet only has one pilot.. and your Air Force only has 64 of the jets like say the Finnish Air Force, then you don't want your rookie Hornet pilot to crash that expensive weapon system just because he got a bit confused in the cockipt about which button did what. So what all modern air forces do, is they buy also a simulator for their main jets - and all pilots learn to fly the simulator first, typically for months or even a year, before they are allowed in the controls of an actual flying plane. They have to know all the controls instinctively and be trained for reacting to just about every conceivable emergency - before they are allowed to fly a real thing. 

So thats what air forces do. Its by the way what civilian jet pilots also do to learn. First fly the simulator. Its how submarine crews train and tank crews practise too Simulations. Not just in business haha, really the most demanding jobs - astronauts, Formula 1 drivers - they do most of their work inside the simulators to master their craft, not in the real super-expensive machine. And the beauty of using simulations in STRATEGY is that you can plan for disasters that in reality are impossible to try - like what if the enemy kills all your frontline soldiers with a sneak-attack, then what... etc etc etc.

So to Narwhal. Narwhal captured voter insights of all American voters (within the 'battleground states' it would have been prohibitively expensive in 2012 technology terms to try to do that for all Americans). They knew very intimately who was a Romney voter - no point to contact that voter, he or she will never change their mind, or it would be enormously costly in time and effort to try to change that mind. And they knew who were Obama supporters - no NEED to convince those to vote. The battle was for the undecided voters in the middle, who were 'persuadable'. But to the power of Narwhal. Once you know truly individually, Mr John Smith of Cincinnati Ohio is a registered Republican who votes reliably Republican and who LIKES Romney - at that level of detail EVERY voter in every 'swing state' (like Ohio). That gets nearly creepy level detail and that true insight... it then enables SCENARIOS. What if there is a snow storm. What if there is a terrorist attack. What if Obama has a heart attack, etc etc etc. Map out possibilities that could influence the election and see what happens.

Did they run simulations? Messina said they ran ... 66 THOUSAND full election scenarios. What if it rains in the West but its a nice day in the East and the South. What if there is a strike at a GM factory. What if the unemployment number goes up. What if what if what if.. sixty six THOUSAND times, over 18 months. On average Narwhal ran 120 actual full-national-election simulations PER DAY !!! Every single night 120 simulations were run for a year and a half !!! (the Narwhal system was used for other things - and a lot of new data entered - at daytime)  66,000 simulations does not guarantee that you win. 66,000 simulations guarantees that you know intimately every conceivable detail and variation of what MIGHT happen and how it would impact that election.66,000 simulations give you the best CHANCES to win, especially if the other side has NO SIMULATOR..

Why did Obama do the Dreamers decision in the summer of 2012? Because Narwhal said that if he did that then, when the locals in Nevada and Colorado were angry about Romney's anti-Hispanic rhetoric, before he had 'pivoted' to his general election position, that was the right thing to do. Nevada and Colorado were nominally 'battleground states' that both sides thought they could win. In reality, the Obama Narwhal team and top Obama advisors like Axelrod all knew that Nevada and Colorado had already been won - BEFORE THE CONVENTIONS !!! As it later emerged in the exit polls It was Romney's bad choices in his nomination fight - and how Obama capitalized on it early - that sealed two of only nine states that could decide the election. And that decision was driven by Narwhal. It knew. Big Data. And simulations. 66,000 simulations. 120 simulations run every single day. Thats the kind of power you can do if you have Big Data.

SEVEN PERCENT

Then the election. Messina said that because the US political system has become ever more polarized, what once was over 30% margin in the middle of 'undecided' voters has shrunk to only 7% in the 2012 election. So 93% of Americans knew who they would vote for, essentially regardless of what happened in the last weeks and months of the campaign. That 7% by the way was in the middle, so it could go either way. Messina then gave the performance metric. In the last four days of the election, in almost all recent elections where an incumbent was on the ticket (ie a re-election vote), the last 'undecided' voters break against the incumbent and for the challenger. If you haven't been convinced in the previous weeks and months that the sitting President has earned his re-election, you vote against him (no women Presidents yet). So this is how Gerald Ford, Jimmy Carter and George Bush 1 lost their re-elections (or in case of Ford, his first election but as incumbent after Nixon resigned in mid term). So it would normally be well under 50%. Meanwhile how did Obama do in 2012? The last four days, the electorate broke for Obama rather than Romney by a 76% margin !!! I calculated that means 4 Million Americans were undecided into the weekend before the election and by Tuesday 3 of the 4 million had decided to go for Obama instead of Romney. I think this is a very interesting number and Messina expanded on the thought. He said he didn't care about reaching all Americans, he cared about reaching those critical 7% and talking to them frequently, endlessly, with language they used, on issues that mattered only to those 7%. Very smart and very VERY powerful - and this is something the Romney team was utterly unable to do, because their system wasn't powerful enough to ISOLATE that 7%.

To know which is the critical 7%, you have to find them. And the only way to do that is to talk to ALL voters and eliminate those other 93%. In some cases its very easy. You visit their home and see they have a Romney sign on the yard. No need to even talk to that family, they will vote reliably Republican. Or the couple who donated two times to the Obama campaign - these clearly are loyal Democrats who will certainly vote for the President. What they they have to keep doing, is reach everybody in some way, to determine who are truly undecided - and mark those (into Narwhal) for future contacts. How much did this last-day focus bring to Obama? If we say it should have broken 66-34 against the incumbent that would have been a Romney win of 2.5% ie 51% - 49% with a bit of roundng-off to even numbers. But Narwhal converted those undecided 4 million to go 3 million to Obama and 1 million to Romney - so the swing was 5.5 million votes - 4.5% of the total electorate !!! A swing of 4.5% done with targeting by BIG DATA. This is the power we are talking about. And Messina gave one more detail, very very crucial.

COMMUNITIES DOMINATE BRANDS

Communities Dominate Brands is the signature book of this blog and is the foundation on which this blog was first created ten years ago. One of the items we identify in that seminal book (the world's first book on the business use of social media) is the power of social media - what your friends recommend or even a random stranger endorses - is more powerful than what advertising achieves. Communities do dominate brands. And Messina said the way the Obama team used Narwhal targeting was to have FRIENDS tell undecided voters why they should vote for Obama. Narwhal had identified not just who were undecided and how they wanted to be contacted and what issues resonated with them - Narwhal ALSO identified the social media peers, friends, influences who were most relevant to that given undecided voter. This is why the most powerful message ever sent, the one SMS text message that delivered the election victory was that one where Team Obama asked volunteers to make just one phone call on behalf of the President that election day. Obviously that call would be to an undecided voter on election day, by very sharp targeting - a nurse got a call from another nurse, a war veteran got a call from another war veteran etc  All this is power of Narwhal. Power of Big Data. The best case study with actual measured results that's ever been released into the public domain. And as Jim Messina says today, polling is obsolete. He will actually fire anyone working for him now who shows him a public poll. Its exactly as I've warned the day will come, when the data we get of ACTUAL behavior (ie Big Data) is more powerful and accurate than any polling and the traditional polling industry will become obsolete (and as I started way back before going to college, as a field interviewer for Finland's Gallup organization, this too was said with deliberate thought. It was a good life back then and a nice job, in marketing research). Big Data makes traditional polls and market reseach as obsolete as trying to fight a modern war wiproth bows and arrows. Yes, they were powerful weapons once, but not anymore. They are utterly replaced by guns and other more modern weapons. This is about to happen to public polling. And the polls will live for a while feeding the media before the Big Data capability becomes so ubiquitous and cheap that even major media houses can own the systems.

Now, if you're interested in the 2016 election, Jeb Bush was the one of the Republicans buildling the largest Big Data system that was a huge drain on his finances. As Jeb's campaign struggled they fired most of that team or reassigned them and the database is now adrift and not being developed. Meanwhile on Hillary Clinton's side, Jim Messina and gang have embarked on the largest and most powerful election Big Data system ever devised which will cost at least 50% more than what Narwhal did last time. And Hillary's is well funded to keep that project on schedule and amply awash in cash to hire the best datamining and programming experts plus the best companies with this competence.

UPDATE 15 DECEMBER - One of our regular readers, Catriona, found story about Ted Cruz's campaign use of Big Data in his campaign now in the primary season for 2016 election (see comments). The article by WaPo has excellent practical political examples of how Big Data is used daily in talking to potential supporters and voters, how voters can be categorized and targeted (segmented) and of course, that the Ted Cruz campaign had studied the 2008 and 2012 Obama use of Big Data and intends to capitalize on those lessons and evolve past that. And exactly like Obama in 2012, the team has noticed they can draw immense power out of the system even as it is being developed and deployed, as the Ted Cruz campaign already now is using its system to select and refine messages and even make decisions on where to send Ted to talk or meet voters etc. Exactly how Obama 2012 used Narwhal. Also note the WaPo article has excellent other tidbits on how to use geofencing for example - usually a hopeless useless marketing tactic that is 'subtractive' and suboptimal compared to almost any other uses of mobile in marketing - geofencing is almost antithetical to 'mobile' where 'mobility' means escaping physical locations (which is why location-based services are literally the worst-performing part of all of mobile services) but as always there are exceptions to the rule. Politics is one area, where geofencing 'faults' are actually 'features' or benefits. The ability to geofence a messsage to a limited location means, that a particularly divisive and extreme politician like Ted Cruz can deliver messages within a very tight narrow area, in a situation where only fans and friends are nearby, without much risk of the messages spreading to the general voting population. Geofencing is thus a defensive method to allow pandered messages to given voting groups, without much risk that message goes to the other voters, allowing tailored (and yes pandered) political messages. Clever even if perhaps dubious in terms of political ethics. But hey, politicians are not expected to be 100% truthful haha.

That article means almost certainly that someone from the Ted Cruz campaign has been reading those blog articles of mine about the Narwhal and Orca analysis. Hi Ted Cruz team! And welcome back! And likely also therefore.. some in the Ted Cruz team have been also reading some of the other elections-related blogs I've written - good luck with 2016 season. To the Cruz team, I am sure you know I rate your chances best at winning the GOP nomination in 2016 (and I was one of the first to do that calculation about the delegate math) but that I also think you'd then lose the general election to HIllary C linton. Sorry about that, haha, I call it as I see it. Your best chance would be for either Trump or Rubio to be the nominee this year, and Ted Cruz to build his reputation in the next years, to return into the race as the clear Republican front-runner in 2020, and hope that Hillary's Presidency has had a disasterous first four years and be vulnerable to you in the interim. Not unlike how Reagan failed at his first attempt but then came back, and defeated one-term Jimmy Carter. If that happens (not getting the nomination now for 2016), Cruz's best bet is to bone up on the foreign policy side of competence in the Senate, and in the negotiations for a friendly convention, try to fenagle a seat on some foreign policy or military related Senate subcommittee and then make sure to attend most meetings and be very visible there... And to evolve from the pure bomb-thrower in the Senate to build a few friendships there to get some mainstream Republican support for 2020, that woudl be priceless in a race against an incumbent President. So anyway, cheers.

For the Big Data peeps of the Cruz campaign, good luck and I hope to meet up with you at some point in the future maybe the next time I visit Houston, you're pushing the envelope and doing great work. I loved what I read on WaPo so keep up the thinking and innovations. Now, don't forget SMS and MMS. When we meet, your campaign will be over and I will want to hear all the juicy bits that you are able to talk about. The beers will be on me! Good luck!

PS PS - I thought of this when doing the Tweets about this addendum - note comparison to Jeb Bush campaign. Jeb has had biggest TV ad spend (the classic way of winning in US Politics is to outspend your rival - its how Romney finally destroyed his rivals in 2012 GOP race). Jeb's seen his support collapse. And Jeb also had the biggest expenditure in Big Data of the Republican side (Scott Walker was second bggest, Ted Cruz only third) and this obviously is the 'new way' to win in US politics, in how Obama defeated Romney. And Jeb ALSO had the biggest Big Data operation, yet his support has collapsed. Having the best tech or the best marketing cannot help you win if the product is crap. And if after biggest spend in TV ads AND biggest spend in Big Data, Jeb Bush only sees his support vanish - that does mean his candidacy is.. crap. Its dead. Its a stiff. Bereft of life, it rests in peace... But Ted Cruz? If his team is also the one using Big Data the best, expect only improvements and gains to that team, and at the expense of who? Ben Carson, Donald Trump for sure who are only winging it without the insights from Big Data...  I will be doing occasional updates to the Big Data angle to the 2016 race and hope to find details about Hillary Clinton's data operations too, at some point...

So those were the new tidbits. I hope still more will emerge with time about Narwhal and Orca. if you want to understand more about Big Data how it applies in business, marketing and yes, politics, start by reading this Primer to Big Data with case study Narwhal. So if you are Coca Cola or Pepsi, how do you use Big Data to win in that battle and so forth. No other blog has more actual performance data about Big Data than that blog article I wrote a couple of months ago. 

Meanwhile if you want to see whats coming in mobile and tech - can't have Big Data without mobile - my Mobile Forecast is available here. If your company needs a seminar or workshop around these matters (or anything in mobile, digital, media, social or tech) then please contact me. My email is tomi at tomiahonen dot com.

Comments

[Catriona]

Guess who's using data analytics in a big way:

https://www.washingtonpost.com/politics/cruz-campaign-credits-psychological-data-and-analytics-for-its-rising-success/2015/12/13/4cb0baf8-9dc5-11e5-bce4-708fe33e3288_story.html

[Tomi T Ahonen]

Hi Catriona

I missed that, thanks! Really good article, detailed and up-to-date. Very good. Lots of tidbits on the tactics and usage of the tool they are building at the Cruz campaign. I will add a PS to the above article and credit you of course. Thanks!

Tomi Ahonen :-)

[Wayne Borean]

Tomi,

So, 7%? That big break you've forecast for Hillary isn't going to be all that big!

Catriona,

Nice catch. I suspect many of the others are using Big Data systems too, maybe not as advanced.

All,

Curiously such an approach would be impractical in Canada. Due to Campiagn Finance restrictions, none of the campaigns have the money to spend. And they can't take 'Free' services either, any free service donated will get tacked onto campaign expenses. Assuming that the finance watchdog finds out about it, and the way things have broken recently not reporting something like this would be suicidal.

Just curious Tomi, would something like this be feasible in Finnland?

[Tomi T Ahonen]

Hi Wayne

First on the 7%.. This would be (in optimal case for Hillary) in ADDITION to the 12 point landslide she already has locked in... on TOP of it, not the extent of it haha.. But yes, if Cruz can develop a reasonably competitive solution then most of that 7% would be negated and its even possible that Messina's team is stuck in older ways and a newer fresh design could outperform that one ie with less money Cruz could theoretically produce something better - however thats a far less likelihood than that the experts who did it twice before, can now improve on their secret insights to make their third iteration now for Hillary even stronger than the best Cruz team can do starting from scratch.

On Finland, much like this is already in use on the telecoms targeting side (much of this theory in math and actual practical applications and patents for business came from Finland) but I honestly am not in any way tuned into the domestic Finnish politics to know if they are using such methods. I do know all the carrriers/operators have had microtargeting to segment sizes of 10,000 ie 'perfect targeting precision' in the theoretical view, for more than a decade IN USE. Japan was second country to do that. Most telcos don't do that level of customer detail yet. So for practical example, my brother, also MBA, who shares 95% of the same interests and likes as me, would get a DIFFERENT marketing targeting if he calls up the same carrier in Finland as I would. They can target to 10,000 segment precision meaning in our lifetime we will never meet another person who shares the exact same 'segment'. So for consumers feels genuinely unique personalized targeting. Its totally commonplace in Finland. Finland also had the world's first digital interactive customer service association for its marketing and media industries, that assoc is over a decade old already haha (I've delivered speeches for their audiences).

Etc etc etc.

Oh, if the systems is in place and multiple vendors offer competitive solutions, then you could rent the capacity for your campaign rather than buy and build from scratch. Hence costs far smaller. BUT the INPUTS to the system would need to be done by the given campaigns. How do we classify Mr Tomi Ahonen of Helsinki? What are my preferences and who do I support and what is my voting history etc. THAT legwork would need to be done regardless. Its not the physical mechanical database, its the political campaign application (I don't mean 'app' I mean 'usage').

Tomi Ahonen :-)

[Tomi T Ahonen]

To all in this thread

Just to let you know, I have found plenty more details about Narwhal vs Orca and have just published that in the new blog today.

Tomi Ahonen :-)