Shaping a Data Team
A data engineer told me about some terrible code he’d uncovered in his company’s data platform. Every day, the code moved a massive data file to a server on the opposite side of the country. That server performed the most complex logic, then shipped the file back. What was that “complex logic”? It looked up the sales representative’s name. It was a simple join.
“I feel like a fish out of water, gasping for oxygen. That’s how blown away I am,” he said.
Mindless data processing solutions like this show up in data platforms everywhere. People use automation to solve a problem in isolation, without considering how their solution affects the overall system. They treat data problems like rote work, repeating the one solution they know in every situation. It’s like learning to use a calculator without ever learning arithmetic.
Robotic Data Solutions
Rote, mindless work is so common in data engineering that programmers even coined an acronym for it: WET programming. Write Everything Twice. We Enjoy Typing. Waste Everyone’s Time. Good data engineers, on the other hand, follow the DRY (Don’t Repeat Yourself) methodology.
You can’t simply blame this problem on bad data engineers; managers create the situation by treating data solutions as something to outsource, handing projects to contractors who don’t understand their company, their data, or the business problems they face.
The main problem with rote work isn’t that it makes your code inefficient; instead, it lacks creativity. You can’t outsource originality.
Dave Bourgeois, Associate Dean and head of Biola University’s AI Lab, says that the best way to prepare for automation is to “learn your craft and be very good at it,” and then know how to automate it with tools. His point is not that you need to become a technical expert on AI - quite the opposite. You should understand the purpose of your work and see how automation can improve the results.
The person who understands the purpose of their work is best prepared to use the right tool for the job.
How to Avoid Becoming a Robot
There’s no magic solution to understanding the purposes behind your work; that takes experience and effort. Here’s some advice to help you grow in these areas:
1 - Start your projects with theory.
I like to imagine I’m the CEO and can afford to make any change I want to solve a problem. What would that ideal solution look like if you could do that?
That’s the approach I’ve always used with data teams: start projects with a discussion about the best theoretical solution. Ignore the current limitations you face. Read good books on enterprise data strategy. If you always solve the problem at hand without knowing what the best overall solution should look like, you end up with a massive web of mindless data solutions.
2 - Don’t work in isolation.
When you’re working as a one-person team, all your ideas sound great - to you. It’s when you try explaining them to others that you start to see holes in your own solutions.
If you asked me for just one change I’d make to your data team, I’d probably tell you to diversify the job roles. Change people’s responsibilities so that every project requires work from at least two people. Almost every data team I see ignores this advice, even when they agree with it. It’s much easier to rely on a single technical expert to get things done. If that’s your situation, try handing off some responsibilities to others.
3 - Don’t sweat the technical.
You’ll never outsmart the robots, so don’t try. That’s like buying a garage full of new tools before you ever own a car. Instead, learn about the tools as you go.
My friend and AI architect Yakov Shkolnikov explains this point powerfully: “To everyone discovering AI without needing to understand gradient descent: welcome. Build something we haven’t imagined yet. Ask questions we haven’t thought to ask.”
Finally, a warning for managers. Does your daily job look more like a purchasing agent than a data strategist? Are you just monitoring project schedules to see if you’ll hit your commit dates? That’s a common trap that sidelines many IT leaders. If you lose touch with the work itself - knowing how data works, not just technically, but how to connect it to decisions - then a robot may come for your job too.
Even though robots can turn data into information these days, they can’t convert knowledge into wisdom. That’s where your creativity kicks in, and that’s how you beat the robot coming to steal your data job.
To help remind you of these concepts, I’m now adding a hit 80s song to the Frictionless Data Spotify playlist each week. This week, enjoy Are Friends Electric? by Gary Numan.