Deep and Wide Competence
Engineers talk of the mythical 10x engineer. Data scientists have their own fable: the “unicorn” who is simultaneously computer scientist, project manager, statistician and software engineer. These chimeric creatures exist, but they are not what their job descriptions would suggest. The data scientist is an ambiguously valuable professional hire, so it’s worth a little time trying to say where the value lies.
When I think of good data scientists I don’t think of a laundry list of software they claim to use, I think of accordion players. I think of the capacity to stretch across domains and go deep into any of them. This is not merely the classic T-shaped professional, with some breadth and a specialism. The ask for a data-scientist, like the statisticians before them, is to play in everyone’s backyard. To absorb domain context and go deep where your stakeholders can’t, derive insight and value, define the problem and build the solution… and then do it all over again in the next domain.
The repetitive twitch of the accordion player that pulls wide, presses deep and repeats, characterises good data science. Over time, the breadth of exposure pulls the data scientist across the business and into the thorniest areas with the least well-defined answers. Reliable discovery across that range is what makes them valuable.
Everyone’s Backyard
The best thing about being a statistician is that you get to play in everyone’s backyard. - John Tukey
Playing in everyone’s backyard sounds liberating, but the accordion motion didn’t emerge pure play. It was the product of a specific economic moment: employers sensed that data could transform their business, but they couldn’t articulate how. They hired malleable professionals and pointed them at everything. That same pattern is repeating now with AI agents. We have broad deployment, diffuse expectations, uneven results. The Big Data era is the right precedent for understanding what’s coming, because it already taught us the difference between collecting answers and discovering good questions.
The Paradox of Big Data: Demand for Data Science
In 2005 Hadoop was invented. In 2006 AWS started offering cloud compute. In 2012 Harvard Business Review declared data science the sexiest job of the 21st century. These events look precipitous, as if the work were truly new. It is not. Data science is as old as the attempt to parse insight from numeric scratching on cuneiform tablets.
Between then and now, we’ve added tooling, but the process of transcription, diligent analysis and communication has not changed. Like today, some changes in tooling made of this work easier. Scikit-learn was established early in the 2000s and went from strength to strength. Python script-kids were derisively dismissed, but they inherited the future from the Excel jockeys of the 1990s. Disputes raged across blogs about Python versus R, Jupyter notebooks versus production engineering. These petty spats about tooling were an identity crisis in disguise. In the era of Big Data, who was empowered to be a data scientist?
The ambiguity of the title opened doors. But equally employers were divided in what they wanted. Traditional titles like statistician or econometrician seemed archaic and expensive, the preserve of professionals with specific expertise. The world needed a cheaper, more malleable professional. The data scientist was born, and you could bootstrap yourself into the role if you could signal or demonstrate expansive competence. The promise of Big-Data spoke of revolutions in every domain, but the work asked of data-scientists was massively heterogenous.
Success for Diffuse Tasks was Unevenly Distributed
The data scientist role met with varied success — some spectacular, much of it modest to negative. Much of the success depended on where they were deployed. The data maturity of the organisation or the receptivity of the organisation to data-informed strategy. But everywhere we discovered that Big-Data was not enough.
Scaling data collection was not enough.
Fan, Han and Liu in “Challenges of Big Data Analysis” argued convincingly that the promise of high-dimensional data is overstated. Most data collection was just scaled-up convenience sampling. Convenience samples masked obvious mistakes behind misguided confidence: yes, you had a large sample size, but the sample measured the wrong things, or the analysis overstated effects as generalisable when they reflected only the available patterns. Industry adjusted. We slowly learned that relevant data was better than convenient data.
AI-assisted workflows risk the same trap at a different level of abstraction. Where Big Data offered convenient data, LLMs offer convenient answers: fluent, immediate, and untethered from the messy provenance that would tell you whether to trust them.
AI Assisted Workflows and Convenient Answers
The data scientist plays the forward scout. They work in the untested, unexamined areas of the commercial environment. In areas of low traction, inadequate data and poor oversight data scientists work to drive discoveries in those domains. A good data scientist communicates the real deficits and works to improve them. Much of their value sits in the discovery process, and in the recommendations made under such uncertainty.
AI-assisted workflows help the data scientist discover the limits of the available data faster and build illustrative prototypes sooner. But they will rarely decline to give an answer when asked for one. The good data scientist will give the uncomfortable answer, and then a plan for how to get a better one.
Overpromising and Under delivering
When the promise of an innovation is over-extended it almost necessarily under delivers. This was true of Big-Data, this will likely be true of AI too. It’s not to say there is no value there, but that a tool applied widely without focus is often less useful than hoped. The demand for data science in the early 2000s is similar to demand for adaptive agents today. We’re throwing agentic compute at problems in the same way business threw data-science at their problems 20 years ago. The lesson we learned from that period is that if you throw one solution at every task-domain your success will be unevenly distributed across those domains.
Data Science and Commercial Relevance
The accordion workflow drives an explore/exploit process whose optimisation goal is itself ill-defined. In any one project we optimise for some commercial outcome, but we’re also betting that exposure will aid goal discovery. What to work on and why!
The greatest commercial value of data science is not a series of automations or a model-based solution. It is a cohort of professionals whose understanding of the business is second to none — who see which levers the business has, how responsive its outcomes are to those levers, and where tight analogies hold across domains. Their judgment is gleaned through hard-won experimentation; their breadth lets them set new targets, not just hit old ones.
AI can automate the mechanical parts of this work. What it cannot automate is the frontier: shaping how the company sees success, steering it on a new course or finding relevance in the swamp of frothy AI slop. In particular AI cannot reliably choose which frontier is worth pursuing and what evidence should change the company’s mind.
How to Build Environments for Relevant Data Science
The standard “ticket-taker” model of analytics management kills the accordion motion. When data science operates as a downstream service bureau fetching data for ad-hoc requests or building isolated dashboards while training models on predetermined specs, it compresses their focus and falsely assumes that leadership already knows the right questions to ask, and the right things to build.
This is not to say operational reporting has no value. Reliable KPIs and automated dashboards are the floor, not the ceiling. The problem is when the floor becomes the entire job description, and leaves no room for innovation.
Kenneth Stanley and Joel Lehman in Why Greatness Cannot Be Planned demonstrate why strict objective-seeking fails in complex domains. Breakthroughs rarely come from marching toward a fixed target. They come from “stepping stones” and unexpected intermediate insights whose strategic value cannot be predicted from the starting line. A data team managed like a Jira assembly line is systematically blinded to these stepping stones.
What works instead:
Include data science at problem definition, not problem assignment. Position data scientists upstream, as co-authors of strategy. Before declaring a metric or launching an initiative, involve them to evaluate framing, test assumptions, and inspect data provenance.
Reward stepping-stone discovery over ticket velocity. Assess data scientists on qualitative impact: the doomed initiatives they invalidate early, the hidden levers they uncover, the clarity they bring to ambiguous spaces.
Give autonomy to follow anomalous signals across departmental boundaries. The broken logging script found while inspecting customer-support data becomes the stepping stone to a company-wide retention fix.
Eliminate the translation barrier. A forward scout’s value lies in translating low-level data realities into high-level strategic direction. Every layer of filtering dilutes the signal.
Deliver near-term value without losing sight of strategic goals. Every piece of work should serve the larger conversation between the business and its data.
When leadership treats data science as a strategic peer rather than a tactical support desk, the organization stops building models for problems it doesn’t fully understand. It gains a permanent strategic radar and discovers where commercial leverage actually lies.
How to Cultivate Good Data Scientists
The environment is only one half of the equation. The accordion motion is a demanding workflow and requires robust habits of mind. The repeated stretch-and-compress instills comfort with uncertainty and builds a set of reflexes you want to train. This motion is also exhausting. The constant context-switching between executive strategy and low-level debugging is a primary driver of data scientist burnout. The habits below are not just about competence but sustainability.
Resist premature certainty. The instinct to train a model or a build dashboard before understanding the domain is strong and rewarded by ticket-based management. Resist it. The first job in any new backyard is to map what you don’t know. Encourage engagement and questioning as default.
Read the Business. Incentive structures, org charts, budget flows — these are the features of the domain model. A data scientist who understands why a team measures what it measures can spot misalignment faster than any anomaly detector. The technical work follows from this understanding, not the other way around.
Build the habit of writing things down for people who aren’t you. Not documentation for its own sake, but the discipline of translating a finding into a decision. If you can’t write a one-paragraph recommendation that a non-technical executive would act on, you haven’t finished the analysis. The accordion compresses all the way down to rigorous method and stretches all the way up to plain language. Both endpoints matter.
Invest in statistical foundations, not just tooling. Tools change every few years; the logic of inference does not. A data scientist who understands why a bootstrap works will adapt to any framework. One who only knows how to call sklearn.model_selection.cross_val_score is stuck when the problem doesn’t fit the API. Go deep enough that you can build the tool you need when no one has built it yet. This is even easier today than before, but you need to know why to build.
Seek lateral exposure deliberately. Don’t wait for the org chart to send you somewhere new. Volunteer for the messy cross-functional project. Sit in on the sales review. Read the support tickets. Every domain you touch extends the range of analogies you can draw, and analogies are the primary currency of strategic insight. The accordion stretches wider each time you play in a new backyard.
Own your Expertise/Authority: The idea of influence without authority is often cited as a way for Data Science ICs to contribute to the business, but it concedes too much. You can influence, but you shouldn’t concede your authority. A data scientist can and should make recommendations without equivocating if they have the data to back the claim.
These habits compound. A data scientist who practises them for five years has a qualitatively different understanding of the business than one who spent the same five years optimising models in a single vertical. The first can set targets; the second can only hit them.
Conclusion: Zooming out
The environment and the individual are two halves of the equation. An organisation that treats data science as a strategic peer gives the accordion room to stretch. A practitioner who cultivates these habits builds the range to make that room worth having, and the capacity to hold the sharpest notes.
A general-purpose LLM can be thrown at every backyard just as a data scientist once was. The lesson of the last twenty years is that what makes either valuable is not breadth of deployment but depth of engagement. The required judgement, skill and specificity comes from actually playing in the backyard rather than reading about it.
Citation
@online{forde2026,
author = {Forde, Nathaniel},
title = {Notes on {Good} {Data} {Science}},
date = {2026-08-20},
langid = {en},
abstract = {Good data science is an accordion motion of stretching
wide across domains, then compressing deep into any one of them.
Drawing on the lessons of the Big Data era and Kenneth Stanley’s
work on objective-free search, this post argues that the value of
data science lies not in automation but in frontier discovery:
shaping how a company sees success. It covers both the
organisational conditions and the individual habits of mind that
sustain this kind of work.}
}