Acme Inc., a mid-sized retailer with 50 stores, wanted to use all the customer and sales data they had accumulated over the years. The CEO envisioned using analytics to optimize inventory, target promotions, and increase sales.
They hired a team of software engineers to build a custom data warehouse. The engineers spent months integrating all the fragmented systems and building ETL pipelines to centralize the data.
After six months of work, they presented the data warehouse to the executives. But when the leadership team asked for insights to drive business decisions, the engineers struggled. Their skills were in building software, not analyzing data.
Engineers Can’t Do Everything
One of the biggest misconceptions about tech in general is that every engineer can do everything. It’s like with doctors — if you’re a doctor, the expectation would be that you can advise your aunt when her headaches, help a cousin fix his broken leg, and, of course, tell your friend if he needs to go to a doctor when his tooth hurts a lot, and his cheek is swollen.
There are many stories on the internet about how software developers are asked to set up a new TV (because you’re good with computers) or UX designers are asked to help someone’s kid with linear algebra (you’re smart, so you must be good at math). It’s interesting how these stereotypes transfer to businesses as well. Unfortunately, these stereotypes from a lack of knowledge can hurt a business.
Transitioning to a Tech Company
If I wanted to start a construction company, I would struggle without construction experience. I don’t know who to hire, how to manage people or ensure they do quality work. People rarely start successful companies with no industry experience.
When companies reach a point where they lack technical skills, many hire software engineers or teams of engineers. The goals vary — building custom software, automation, etc. But more than simply hiring engineers is needed to get started. It’s like starting a delivery company by just hiring drivers.
If you’re considering this path, realize you’re now becoming a tech company. Building an engineering team takes knowledge and experience, just like starting a new tech company. You can either hire someone with that experience or gain it over time. If that’s different from your goal, consider low-code tools, existing products, or other options.
If you decide to gain this experience, you must know that having engineers analyze requirements, test their work, set up data pipelines, and take on other strategic roles is quite inefficient.
For example, engineers may build statistical models when their core skill is writing software. But their insights will likely be much less helpful than those created by data scientists focusing on just that type of work. Without oversight, engineers tackle too many responsibilities in small companies.
Strategic hires like a CTO provide technical leadership for building capable teams. They develop long-term data strategies and oversee successful implementations. The right hires turn insights into real business impact. Simply hiring engineers does not.
Leveraging Data
Often, companies transition to “tech companies” because they want to utilize data. This could be existing customer data, purchased third-party data, scraped web data, or legacy data from old servers.
The expectation becomes that engineers will collect, normalize, clean up, and gain insights from the data. However, handling data as an engineer differs from data science, which is closer to strategic analysis.
Software engineers are skilled at building software to view, edit, and manipulate data. Data scientists study large datasets to uncover patterns using statistical languages like R or complex machine learning models. Hiring the right roles is crucial.
A Chief Technology Officer (CTO) could oversee a long-term data strategy and build an effective implementation team. With leadership, engineers focus on software development rather than tackling strategic data tasks beyond their core skills. What they can do is lay the foundation for data teams to do their work.
Hiring the Right Roles
To build an effective data team, first identify your goals. Most companies know their current approach isn’t working but need to know what will.
If the goal is collecting, storing, processing, and analyzing data for insights, you need engineers and data scientists. Data scientists don’t enjoy building pipelines and infrastructure. They want to focus on analysis.
Hire backend engineers experienced with building data pipelines to handle the architecture. Many engineers claim “working with data” experience when building CRUD apps. You need pipeline experts.
A Chief Technology Officer or Chief Data Officer provides critical leadership for the data strategy and team building. If full-time is too expensive, consider consulting or fractional executives.
With strategic leadership setting the vision, data scientists can focus on analysis while experienced backend engineers build the pipelines. The right hires enable impactful use of data vs. just overloaded engineers and analysts.
The key is avoiding a situation where engineers are tasked with analysis beyond their core software skills. Build a balanced team with leadership, engineers, and data scientists playing to their strengths.
Originally published on Medium.com