I hate bad reporting on stats. No, I HATE bad reporting on stats. HATE - (updated)

UPDATE 17 March 2010 - Mob4Hire has responded here at this blog, and at their blog, on Slideshare and on Twitter. More relevantly, they have made changes to the wording in their report, replacing some phrases of "mobile users" with "survey respondents" which should make it very clear to any readersof the report, what the findings are about. As I wrote yesterday, Mob4Hire are a good company that we have blogged about here long before this matter happened. The report itself contains very valuable research data on 5,000 heavily mobile-enthusiastic, mostly male, mobile application users in 111 countries.

Mob4Hire CEO Stephen King has written personally at their blog and kindly also visited us here and posted  a comment to this blog. His points are valid. His actions are professional. The most important point is that he tells us they have reposted the 8 page summary of their report, uploaded that to slideshare and Stephen King tells us

"So, if it was in any way unclear from us (which we don’t think it was, but that’s our opinion), you can get further analysis from Tomi. Tomi emailed me directly, so as an olive branch, we've made a few changes to the report (i.e. changed a few of the phrases "mobile users" to "survey respondents") and re-posted to Slideshare, so hopefully that's that."

The full open letter by Mob4Hire is here. In my mind, today, this matter is a case of unfortunate language in the summary version of a great and valuable survey. I see that Mob4Hire has acted very responsibly and rapidly in observing their language could be better, and fixed that. Obviously there was nothing wrong in the findings of the report itself. I consider the matter 'fixed' from my side.

WHAT OF DIGITAL REPUTATIONS?

But in social media, the story does not disappear. This is the Communities Dominate blog. We often report on developments in the digital reputations and the electronic echoes that now ripple throughout the blogosphere, on Twitter etc. We've discussed Dell Hell and the Sprint 1,000 fiasco and the crazy London Tube officer and other such incidents. Yesterday on Twitter and the blogs, we had another such incident with Mob4Hire. While the language of the summary report has been changed, there now exists a series of postings and follow-ups, which refer to this incident and obviously refer to this blog or my tweets etc.

Yesterday in my angry Twitter comments and on this blog I also made very hostile statements about the professionalism and competence of Dr Bob E Hayes, his company Business Over Broadway and Mob4Hire. I Today I am fully willing to accept that the use of unfortunate language in the report summary does not mean these three entities attributed with creating the report were incompetent or irresponsible, because they did fix the matter and correct the language. In fact, I am willing to accept as proof of their professionalism that they took immediate control of the situation, did not try to hide from it, and responded to it.

So, first of all, I apologize for the hostile language and I retract my claims of unprofessional, irresponsible, incompetent or shameful behavior by Dr Bob E Hayes, his company Business Over Broadway, and Mob4Hire.

Secondy, I am conflicted about how to handle the blog article and its electronic echoes. I am 'old school' among bloggers and believe in the blogosphere needing to have permanence and a record. However, I do feel that now as Mob4Hire has rapidly fixed what as not a bad report, but only a few unfortunate phrases, the damage to their reputations seems excessive. I know this blog is widely read and syndicated, I do think it is too harsh 'punishment' to leave the full text of my blog yesterday on this blog. But equally I cannot delete the posting as it should stand as a record of social media and there were many reactions to the story.

So I have edited the blog first now, to remove the accusations of unprofessionalism and any personal attacks.

Then I wrote this 'Update' section to the top of the blog, to make sure, that any readers on 17 March or thereafter will first read this update, before proceeding if they so wish, to read about the incident yesterday. I have also edited the headline of this blog to add the word 'updated'

I am keeping the comments section with Stephen King's comment. I am adding my response to Stephen also to it (next). I will also go visit Mob4Hire's blog and post my comment and update there. I will also go to Slideshare and post an updated comment there as well.

ORIGINAL BLOG POSTING FROM 16 MARCH BY TOMI T AHONEN

(note this blog posting has been edited on 17 March 2010 to remove some accusations by Tomi Ahonen on the lack of professionalism by those who produced this report. The rest of the language in this 'rant' response to what was bad language in the original report, has been left here as a historical note)

I just got on a mini-rant on Twitter. My good friends over at Mob4Hire (a very nice outfit out of Canada who do crowd-sourced mobile phone testing, we've even blogged about them here at Communities Dominate. Well, even great companies make mistakes.

So, Mob4Hire today released a report they proudly claim is "unprecedented global survey' that reveals 'role of mobile applications in the wider ecosystem of the users' mobile experience". They tell us that they surveyed 5,000 respondents in 111 countries. Sounds very very impressive, thorough and relevant finding? And then they give us their summary findings in alarming bullet point headlinens:

"75% of current wireless subscribers indicated that mobile appliations are important in their decision to select a mobile operator" Wow. Seems very significant number. 75% of current total mobile phone subscribers (on the planet, this was covering 111 countries)

They also reported that 32% of respondents "say mobile apps are extremely important (10 out of 10) when choosing their new operator."  (the word 'extremely' was underlined in the original). Wow. 10 out of 10. That is rare to find 32% in ANY survey, but these apps must be really really important now. One third of consumers on the planet will change operators based on this apparently. Wow.

And they then said 27% of the respondents were intending to switch operators within the next 12 months (that is within typical industry stats, actually below the number in many cases). They then listed what looks like three reasons to switch operators, and add again very alarmingly that "nearly 40% of likely switchers attributed their desire to switch to a lack of mobile applications". Wow again. This is dynamite stuff...

Later in the 8 page summary they also tell us that "on average people downloaded 5 to 7 free apps over the past 3 months" and at the same time they bought 1 paid app. again, sounds like very useful data on the industry on this unprecedented global survey of mobile user habits.

The findings then offer us a split of free apps downloaded per mobile phone brand and paid apps by the same breakdown. Finally they offer a bizarre matrix of what seems like quality index of who is good and who is bad among global operators in delivering apps absed on mobile app 'stability' and mobile app customer satisfaction. Apparently the Middle Eastern and Africa focused Zain is the world's best operator in achieving greatness in delivering mobile apps.

WHAT DO WE KNOW

The numbers they report include the following tidbits. 86% of the respondents to the survey "had downloaded at least 1 mobile application in the last 3 months" - what? yes. Bear in mind that only 13% of all phones on the planet are smartphones. Many operators around the world, in particular in the Emerging World markets, do not even have active apps stores yet - so there are plenty of smartphone users who have no store to go to get apps to download. But the 'unprecedented global survey' finds somehow that 86% of mobile phone users have downloaded an app. To what? voicemail?

No, actually it is worse. This unprecedented global research of 5,000 users across 111 countries actually found that 89% "of mobile users" report that apps crash on their phones! No, it did not find that perhaps 89% of application users, or of smartphone users, but rather that out of all mobile phone users on the planet - "of mobile users" - yes nine out of ten have apps that crash on their phones. Even if we somehow have no satisfied consumers with apps that have not crashed, this means that of the research survey population - 89% have an application on their phones! Thus 89% have apparently smartphones? Vs 13% of the world and roughly speaking 20% in the US and 25% in Western Europe. But this unprecedented global survey finds that in reality 89% of us - you and me and your parents, kids, grandparents even - nine out of ten have an app that has crashed.. Well. We see now where this is headed.

They are making outrageous claims. And yes, lets take just one more detail - gender split. How do you feel the mobile phone penetration is among women around your neck of the woods? About roughly the same as men? 50/50 perhaps. or maybe 55/45 in places like the Middle East or Africa. How did this research find its survey population. Yeah, maybe a bit of a male bias - yeah. 82% were men! Only 18% were women. And this is claimed to be globally relevant and appropriately reflects 'wireless subscribers' and 'mobile users' and 'customers' and 'people'. 

Clearly this research is aware of smartphones as it several times makes mention separately of "iPhone users" and other brand and smartphones. This further leads the reader to be assured that when they say 'mobile user' they do really mean that, not for example 'smartphone users'.

So, first we now know from their own data, that this is not a survey of overall mobile users. It is a skewed survey on two extreme dimensions. Where smartphones represent 13% of all mobile phones and thus 87% of all phones in use are not smartphones, this survey flips that statistic and found 89% with phones that had disappointing app experiences (at least 89% of the surveyed people thus had a smartphone). And rather than a normal distribution of men and women, they have 82% men and only 18% women.

WHAT THEY ACTUALLY FOUND

So lets do the corrections to the outrageous statements and claims. Here is what at best could be claimed form these findings. I have indicated what has been removed so you can compare the original and corrected text:

"75% of current wireless subscribers male smartphone users who have downloaded apps recently indicated that mobile appliations are important in their decision to select a mobile operator" Ok, this seems more 'reasonable' now. Note that this is less than half of smartphone users (half women) and not all with smartphones have downloaded apps, so we perhaps now look at 5% of all mobile phone owners on the planet. But yes, rather than the sensationalist headline that three out of four mobile phone owners say apps are important in selecting an operator (that would be astonishing) we now find that yes, only 3 out of 4 male users of smartphones, who actively download apps, think apps are important in selecting an operator. Fine. That sounds reasonable.
 
They also reported that 32% of respondents male smartphone users who have downloaded apps recently say mobile apps are extremely important (10 out of 10) when choosing their new operator."  (the word 'extremely' was underlined in the original). Yes, it is not that one in three mobile phone owners thinks apps are extremely important in choosing operators. No for 95% of people it is irrelevant. Only one third of male smartphone owners who actively download apps, feel so strongly about apps, that they say it is 'extemely' important in selecting a new operator.

They then listed what looks like three reasons to switch operators, and add again very alarmingly that "nearly 40% of likely switchers attributed their desire to switch to a lack of mobile applications".  No, that is not what the survey found. It found that the fifth most listed reason was this 'mobile apps'. The four more important reasons were 'poor value for price' and 'poor customer service' and 'poor network coverage' and 'unreliable service'. The survey allowed multiple responses ! Poor value got 64% of respondents, and poor customer service got 53% (thus the total will exceed 100%) etc. Apps came in 5th and had only 37% of the respondents.

But that is totally misleading to suggest that 37% felt their primary desire to switch was lack of apps - because the survey respondents were allowed multiple replies and the total percentages run not to 100% but to 298%. In other words, most of those who said 'apps' were one reason, would also have picked on average 2 other reasons ahead of apps - poor value, poor customer service, poor network and unreliable service. If the reasons were to be harmonized for their total relevance, then the real stats for SEVEN reasons listed, and their 'weighted importance' in deciding why to switch networks would be: Poor value 22%, poor customer service 18%, bad network 17%, unreliable service 16%, apps 13%, lack of variety of phones 9% and bad reputation 6%.  

So lets correct the statement, "nearly 40% of likely switchers 13% of reasons given by male active users of smartphone apps, who are likely to switch operators soon attribute their desire to switch to a lack of mobile applications".  Again, its not that 40% of all mobile phone owners intending to change operators think apps are relevant. No, out of all reasons given by male smartphone owners who already download apps, the lack of applications came in fifth among reasons, and out of all reasons given, was listed 13% of the time. Big deal. If one in four intends to change, and 5% of those are male smartphone owners who download apps, and out of those 13% of the reason to change (and fifth overall) is apps - thats oen tenth of one percent of churn. Please lets not overhype this ok. Not 40%. One tenth of one percent

And on the usage, "on average people downloaded 5 to 7 free apps over the past 3 months" - if this was true of all 4.6 Billion mobile phone users - that would be an annual level of free applications downloaded at 110 Billion apps. Yes, Apple did report about 2 Billion apps downloaded last year and the majority of them were free, but as Apple was the biggest app store, clearly this report is totally off again. These are not 'people' but these are rather smartphone users. Let correct the statements

"on average people male smartphone users who have downloaded apps recently downloaded 5 to 7 free apps over the past 3 months" and at the same time they bought 1 paid app. Yeah, if you have a smartphone, are male, probably like games, have already downloaded at least one app in the past 3 months, I think its very reasonable that the average is about 5 to 7 apps in a 3 month period for these users, and one paid app. Yeah, now its reasonable.

Clearly the total finding changes from alarmist to reasonable.

This is irresponsible and unacceptable. Its not enough that the authors of this report go and correct those blatant lies, but that they also take public responsibility for this pointless hysteria and admit they were at error. I will not stand by this kind of bad-mouthing of the mobile industry and will protest to all Mob4Hire execs and senior management and will Tweet and blog about it. This is unacceptable.

(TOMI ADDENDUM FROM 17 March 2010 - the above is the edited version from the full text of the blog originally posted at this blog on 16 March 2010. I have only removed language which attacked personally the professionalism of the three entities identified with the creation of the report. The rest of the original text remains as a record)

Comments

[Romain]

The origin of the bias is that they performed the survey on their own "testers" that they use for their "crowdsourced" testing services (pretty smart idea BTW). Obviously someone who volunteers to test mobile apps (even for a fee) is not representative of the vast majority of mobile owners.
Their error is to claim that their findings hold a general value.
On a side note, I really like some of their findings, like that "a majority of iPhone users [...] typically use one mobile application store".
Wow, breaking news, isn't it ?

[Samuel]

Speaking about being ill informed I'm shocked that you aren't aware that MOST mobile phones (not just smartphones) have supported application downloads for almost a decade.

[Tomi T Ahonen]

Hi Romain and Samuel

Thank you for the comments. Will respond individually

Romain - haha, yeah, 'surprising' that iPhone users tend to only use one app store, isn't it.. Yeah the sad part is that these Mob4Hire are nice guys, they did a big global survey of 5,000 app user globally and they paid for it from among their tester base. These are mobile enthusiast users. But someone went and then decided to write that all wireless subscribers want to do this and all mobile users want to do that, and totally messed up a good survey result, and are doing major damage to their reputation. A poor choice of terminology really. But that does not excuse it, we can't have these numbers picked up by the press, I can't imagine the hysteria that would generate among the Application-istas haha..

Samuel - thanks, you are slightly right and mostly wrong. Yes its true apps have been able to be downloaded for almost a year, when first Java and Brew enabled phones appeard in Japan and South Korea. And yes, today more than half of all phones do allow that. But its not true that 'most phones have supported app downloads for a decade'. The population of Java/Brew enabled phones passed the half point of total installed base of phones only in 2008. Not 'most for the decade' only 'majority' for two years...

That of course then is not respective of the actual enabled ability. Some phones are crippled, some networks do not allow this, the data plans mostly do nont allow this (usually only smartphone data plans tend to support apps downloads), most feature phones are not pre-configured to allow it, and most owners of those phones will not download apps to them. Some do. The world's first app store was launched in Japan 9 years ago.. So yeah, I do know. But even today your 'most' sounds impressive but its only 53% of all phones in use support Java or Brew or is a smartphone or a combination of the above. Just over half..

Thank you both for writing

Tomi Ahonen :-)

[Stephen King]

Tomi ... as always thanks for your passion. I have replied to your comments on a post on our blog: http://mob4hire.blogspot.com/2010/03/open-letter-to-tomi-ahonen-regarding.html

A quick summary:

We certainly did not mean to mislead anyone, nor did we intend to misrepresent our sample as being the global population at large. We thought it was clear: At the front of the report, page 1, we state “The survey respondents are primarily male (82%) with a median age of 25 years. A highly educated sample, 57% of the respondents have a 4-year degree or higher.” We also disclose that the 5,000 respondents are part of our Mob4Hire mobile testing community and are certainly mobile app enthusiasts, which we also disclose BEFORE we make any comments about the survey findings. So, any further statements in the report about “people” or “mobile users” should be taken in this context ... that's how I would read it, anyway. It’s an extremely valid look at mobile app early-adopters, don’t you think?

Secondly, we don’t say that mobile apps are the PRIMARY driver of network operator customer loyalty. Network coverage and service are still the most important dimensions to operator customer loyalty. See page 3 of the report … specifically the graph. We’re very clear on this point. But, mobile apps are become increasingly important, especially to the sample of early adopters that we fully disclose. I think that's a great finding.

Tomi, you made your point ... and as I communicated to you directly, and through some other posts, we've made some changes to the report (for example, changing "mobile users" in some statements to "survey respondents") to help make it clearer. Again, thanks for your passion.

But, maybe it'd been better to have left out the personal attacks? Let's stick to the data and interpretation ... that's where the fun stuff is.

Thanks,

Stephen King, CEO
Mob4hire.com

[Tomi T Ahonen]

Hi Stephen

Thank you for taking rapid action on this and of addressing the matter personally. I think that shows how professoinal Mob4Hire is as an organization.

I noted that you have posted a blog open letter, you have come here with the comment to our blog, you have also posted a reply at Slideshare and on Twitter.

I appreciate it that you feel the language was appropriate, so we clearly have a difference of opinion there. What I think shows great poise is that inspite of that difference of opinion, you have adjusted some language removing 'mobile users' and replacing it with 'survey respondents' to make the report less prone to misinterpretation.

I am fully willing to consider that responsible and professional behavior, and have therefore done an update to this blog. I have also visited your blog and posted a comment telling your readers that I have edited my blog. I have removed any hostile personal accusations to the producers of the report. And I have Tweeted these updates.

The report, as I say, is good and it would have been unfortunate it it would have been getting the wrong type of heat and attention by some who would have misinterpreted the summary statements.

Thank you for visiting our blog and my best to all at Mob4Hire

Tomi Ahonen :-)

[Laptop Batteries]

Speaking about being ill informed I'm shocked that you aren't aware that MOST mobile phones (not just smartphones) have supported application downloads for almost a decade.

[Polly]

I am fully willing to consider that responsible and professional behavior, and have therefore done an update to this blog. I have also visited your blog and posted a comment telling your readers that I have edited my blog. I have removed any hostile personal accusations to the producers of the report. And I have Tweeted these updates.

[white iphone 4]

There are certainly a lot of details like that to take into consideration.

[white iphone 4]

White iphone 4 Conversion Kit is now available. Though White iphone 4 Conversion Kit is suppose to release until next year, but now, you got a great chance to buy White iphone 4 Conversion Kit.

[HatfieldDARLENE19]

The loan are useful for people, which want to start their own company. As a fact, it's not very hard to receive a student loan.