CIO Journal
The next time you’re in Silicon Valley, be sure to visit the Winchester House. It’s about five miles from Apple and Nvidia corporate headquarters and right across the street from Santana Row, the trendiest mall in the area. With 160 rooms, 47 fireplaces, and 2,000 doors, it’s hard to miss. Watch out for its many mysterious features, like stairs leading to nowhere and upper-level doors opening to clear air.
The massive Queen Anne Style Victorian mansion was once the personal residence of Sarah Winchester, the widow of firearm magnate William Wirt Winchester. People tour it to see its size, its architectural curiosities, and…its obvious lack of a master plan.
Thanks for reading Frictionless Data!
Her husband’s passing left Ms Winchester with mountains of money, and (according to legend) also left her haunted by the legacy of death that it represented. She spent 38 years of her life expanding her residence, aimlessly adding rooms and floors for what appeared to be therapeutic reasons. The workers just built whatever she asked for. Only the 1906 San Francisco earthquake slowed the room additions.
Today, the Winchester House is a great place for IT engineers to go on a team outing and see what a real haunted house looks and feels like. Admission is $46.06.
Does navigating your company’s data feel like walking through a house with stairways to nowhere? Maybe your IT team built their own Winchester house with your data. I’ve seen that happen at many companies, even right near the real Winchester House in San Jose.
One of them, a high-profile company with huge R&D investments, needed quick, accurate ROI decisions to compete against its AI software rivals. Although they had good intentions, the IT team didn’t know how to link financial data to product data, and the Finance team couldn’t explain how to do it either. Layers and layers of system logic (built by data engineers) and complicated spreadsheets (built by financial analysts) combined to make the data platform fragile, slow, and very expensive.
None of the executives knew how the data worked; they only knew their staff struggled to use it for decisions.
In another case, the leaders of a nearby biotech company gave all their employees access to Tableau Software so they could create whatever they wanted. The data team simply processed tickets to move data to whatever location the Tableau users requested. It started with good intentions - granting everyone control over their own data sounded great at the time - but there was no plan to align data across the company.
In both cases, their data solutions looked like a collection of different Winchester Houses, all built on the same property.
Mortgaging the Architecture
Maybe you’ve heard the industry term for this problem: “technical debt.” It describes the cost of taking shortcuts and failing to align business processes while creating software solutions. A2022 McKinsey study concluded that technical debt is a universal problem in large companies.
You can’t blame outcomes like this entirely on the technicians who built your systems. Bad solutions often start when executives make technical decisions without listening to what their data teams really think. They assume they’ve found a silver bullet solution for their data problems, then pressure their technical teams to conform to their vision.
One of the most popular executive shortcuts in recent memory was “data democratization.”
The wonders of a decentralized world, where everyone handles everything on their own, sounded convincing when vendors sold new analytics software for everyone’s laptop computer. Unfortunately, people grew to depend on the uncoordinated data solutions that resulted from that mindset. Instead of agreeing on common processes and standard definitions for data and business logic, every division - and sometimes every analyst - followed their own path.
Describing a situation like this as “debt” makes a lot of sense; it doesn’t really matter how you got there or who to blame, because you need to pay it off either way. You will, however, need to change your spending habits, and that’s the hardest part.
You and I are in this together. News about your technical debt isn’t easy to accept, and it’s not a great strategy for writing popular newsletters either. But I hope you’re convinced of its importance, and we’ll give it more thought here before you’re forced to declare bankruptcy.
To help remind you of these concepts, I’m now adding a hit 80s song to theFrictionless Data Spotify playlist each week. This week, enjoy this cover of Stairway to Heaven from the 80s rock band Great White.
BONUS: Stairway to Heaven perfectly fits the story of Sarah Winchester, but the first draft of this article featured Our House by Madness. Why not enjoy both?
Thanks for reading Frictionless Data! Subscribe for free to receive new posts and support my work.