Tom Davenport, writing for Harvard Business Review, created a stir several years ago when he declared data scientist the sexiest job of the 21st century. More recently, Glassdoor published the 25 best jobs in America. What job landed in first place? You guessed it: data scientist. It would seem this attention might create droves of candidates for open positions, but it hasn’t.
The qualifications are challenging. An effective data scientist is still created from the optimal mix of statistician, software developer, and communicator. Hilary Mason, the eminently qualified data scientist and founder of Fast Forward Labs, uses the chart below to visualize the magical combination of skills we seek.
The scarcity of this unique blend, which has led to high valuation and hyperbolic descriptions, still persists. Mommas may all be letting their babies grow up to be data scientists now, but we needed them yesterday. The talent shortage remains painful.
But wait—with Google’s AI solution AlphaGo defeating Lee Sodol, one of the world’s most talented players at the ancient game of Go, maybe the solution is software. We also have Watson smoking the world’s best “Jeopardy” players. Can’t we just create a data science robot and move on to the next problem? Maybe we’ll be able to speak to an upgrade of Siri and say, “Hey, girl, tell master whether we can improve sales of flying toasters next quarter by running ads on ‘Judge Judy.’”
Google Ventures and Goldman Sachs are pushing hard in this direction with Kensho, which they describe as the world’s first computational knowledge engine for the financial industry. They’re talking about replacing financial analysts with software, a goal that makes me wonder who is going to ask all of those pesky questions at earnings calls. (I know, that’s probably a concern that’s rooted in the past.)
My dream, though, is less about artificial intelligence (AI) and more about intelligence augmentation (IA). Data scientists still have to do too much heavy lifting before telling you whether A is better than B. The creativity behind the hypothesis is where we should be focusing our big human brains; instead, we are bogged down with data wrangling, data integration, data munging, etc. If you compare the state of data science to accounting, it seems we are still waiting for the invention of the spreadsheet. The basics are just too difficult.
After the man-versus-machine battle within the chess domain became uninteresting, with chess grandmasters routinely losing to supercomputers, a new freestyle chess tournament was formed in 2005 by PAL/CSS, with entrants being allowed to combine man and machine. Surprisingly, the winner was not a human grandmaster allied with a supercomputer. The contest was won by a couple of amateurs, strong players but previously undistinguished, and their laptops. They combined excellent intelligence augmentation software with a collaborative method for inventing and testing many strategies.
An IA vision for making data science easier changes Hilary’s diagram to something like this:
Augment the intelligence of our many analysts with a Data-Science-Made-Easy Magic Box, and we won’t have to wait for Siri to become proficient in imagining various hypotheses. Instead, we could rely on Larry or Jennifer or Rachel, who might even have human failings such as bad haircuts or coffee breath. We might even organize all three of them, in which case, asking our question about flying toasters might yield this exchange:
- Larry: “Everyone watching ‘Judge Judy’ already has a flying toaster, so don’t waste money there. However, running ads during ‘Hoarders’ influences a demographic that often doesn’t yet own but is predisposed to buy.”
- You: “Really?”
- Rachel: “Sales should increase 10%.”
- You: “Wow.”
- Jennifer: “And if we reshoot our ad to show our toasters piloted by cats wearing aviator goggles, sales will increase 20%.”
- You: “Seriously?”
- Rachel: “That’s what the data says.”
- You: “Get some cats. Let’s do this.”
In this IA scenario, we solve our data-science talent shortage while retooling people with business knowledge, enabling better decisions based on data.
That’s a future with lots and lots of awesome.