Reading time: 3 minutes 30 seconds

Look at data from firms selling goods and services that involve advanced analytics, machine learning, deep neural networks, and other AI technologies. Do you trust all the numbers they use to pique your interest?

Don’t.

And don’t wake up on the naive side of the bed. Watch out for intentional (or uninformed) injections of sample bias and techniques that confuse and conflate current state numbers with future state intentions.

  1. Sample bias

There’s so much sample bias around that I’m convinced it’s a technique many analysts and consultants are required to use.

Obvious examples? Any studies that limit respondents to those more likely to over-report AI investment. If you want to subtly or bluntly convince people there’s more production AI investment going on right now, ensure your survey only includes data from

“Participants…knowledgeable about the business and technology aspects of ML or AI either currently deployed or in planning at their organizations.”

A major industry analyst firm

Sometimes the sample bias is even more explicit. Let this example be another literal example of hidden sample bias:

All respondents were required to be knowledgeable about their company’s use of cognitive technologies/artificial intelligence, and 90 percent have direct involvement with their company’s AI strategy, spending, implementation, and/or decision-making.

One of the “Big Four” accounting organizations.

With the 90 percent phrase, the writers’ are trying to be more convincing that their study represents the knowable truth. They’re actually proving to me that the deck is stacked with those most likely to skew the results.

Help protect others from sample bias: Send me other examples of sample bias you see in analyst and consulting firm reports. I’ll collect them and publish some of the best.

2. Confusing and conflating current state numbers with future state intentions

Sample bias pollutes current state numbers. Assuming anything of similar quality out of future state intentions?

“The road to hell is paved with good intentions.”

https://en.wikipedia.org/wiki/The_road_to_hell_is_paved_with_good_intentions

And what if the intentions are bad?

Various firms have projected that survey respondents’ predictions for future AI investments reflect future reality! Imagine that. How many of last years’ new years’ eve resolutions have you kept? What about the book “Dow 36,000: The New Strategy for Profiting from the Coming Rise in the Stock Market” — wasn’t it accurate? One of its reviewers wrote

“It may sound like headline-grabbing sensationalism, but the scholarly and punctilious authors make a persuasive case . . . the book is highly readable and witty.”

— Arthur M. Louis, San Francisco Chronicle

(That book was published in November 2000.)

I’ve found globally recognized, industry analyst firms mixing current states and future predictions and presuming the latter are as accurate as the former. Maybe they’re right. Maybe their current numbers are wishful thinking as well.

Comparing current year “actuals” versus future year “intentions” is a junky technique. The right way to do year over year comparisons is with longitudinal research, asking the same question every year for multiple years.

I happen to have enjoyed survey results published in 2018 and 2018 by Spiceworks.

In 2019, their survey respondents indicated that 13 percent “have adopted AI” with another 17 percent planning to do so in the following year.

But in 2019, only 10 percent indicated they had adopted AI! So much for the 17 percent projection. Spiceworks attempts to explain the difference by saying:

Although many companies intended to adopt AI, VR, and 3D printers last year, the schematics have changed, or at least plans have been delayed. Current adoption rates for VR and AI technology haven’t budged much, year over year, particularly in small companies.

This could be the result of some businesses initially being overly optimistic about future tech adoption, but later opting to focus on updating their infrastructure and software instead. After all, as we know from our recent State of IT Budgets report, the need to refresh outdated IT infrastructure is the number one reason IT budgets are increasing in 2019.

https://www.spiceworks.com/marketing/state-of-it-2019/future-tech/

Do not accept people’s projections of their future behavior as accurate! Too often, even the firmest predictions turn out to be no more accurate than New Years’ Eve resolutions. Ask President Hillary Clinton about that.

I’ll save for another blog post consideration of other ways data on “adopting AI” (whatever that means) is dramatically misleading. Until then:

  1. Beware of evident and hidden sample bias in market penetration studies reported by entities with a vested interest in convincing you to think, or continue to think, in a particular fashion, most often tied to their long term financial success.
  2. Back away from studies that confuse or conflate reported current states versus projections for future states.

As Mark Twain observed, “Figures don’t lie but liars figure.” Not all deceptions are intentional lies. Some reflect a lack of competence on the part of the source. Watch out.

For the record: I cited PWC’s CEO survey data on AI market penetration in this earlier article. Five percent (US enterprises) and ten percent (European) feel much closer to reality than most of the other estimates out there.

Disclaimer: This post is my own opinion.