It’s rare to find a data team that does not feel overworked and under the pressure of a massive backlog. While some companies over hire and then go through layoffs, other teams suffer from chronic under investment. Good resource planning seems to be scarce in the data world. Why do you think that is?
The short answer is too many short-sighted, tactical decisions and too little long-term strategic talent acquisition and planning. I don’t see this as a problem that is particular for data teams, but I can understand why the impact and volatility are more dramatic in the data domain.
Why is this problem so dramatic in data?
When locating data teams in a team topology*, I rarely experienced data competence and capacity as fully integrated into cross-functional stream-aligned teams* (i.e. delivery or discovery product engineering teams). What I often see is that data teams are appendixes.
Data teams often operate outside the core domain of the business. In good times, their task is to identify, boost, and leverage growth potential In hard times, organizations remove or reduce parts of the organization that seem to be non-vital to focus on the core product domain and operations only. Because data teams are not part of the core and were designed as an appendix, it is easy to cut them. When you want to go lean, you cut off all the extras. If we didn’t have a data team, would we still be around? The answer for senior managers is often yes.
This leads to high volatility and workload peaks as the organization follows this vicious cycle.
Why are so many data teams constantly overworked?
We can look at this problem through Convey’s Law, which says that the organizational design determines the design of the output. Convey put it “any organization that designs a system will produce a design whose structure is a copy of the organization’s communication structure.”
Often generation one of the team is let go or quits, but the system they created, i.e. the data pipelines, data warehouse, etc. stays. When better times arrive, generation two gets hired. Generation two finds system design that was created according to the organizational structure of generation one. Generation two needs to reverse engineer and understand the system, and realizes that they need more headcount to compensate for the learning curve and maintenance to even achieve the same level of output as generation one.
On top of that, as the company is growing and innovating, the organization wants the generation two data team to produce high-value and high-visibility activities such as insight and algorithmic products. Teams think that there might be a few days of overwork, but find themselves overworking over a prolonged period of time. This can lead to over hiring, when not done strategically in alignment of system complexity and business capabilities. Massive attrition can also happen due to team members looking for a more orderly work environment with better work-life balance.
How can a data leader accurately estimate their resource needs?
The first thing you need to do is demand a seat at the table.
If you are not part of the business strategy, you will never be able to do your job. When you know the strategic objectives, then you can identify the current complexity of the system and create a trajectory of its future state to estimate an appropriate headcount to a) operate and b) enhance the system.
If you know the growth targets, then you know what the current system is capable of and estimate the capacity to operate at the lowest baseline needed for business continuity.
What could a resilient data organization look like?
In a resilient organization, there will always be a place for additional core data teams for infrastructure and operations, acting as an enabler team and owning complicated subsystems. If a team owns a complicated subsystem such as data infrastructure, it is clear they are here to stay. As you scale up and down, you still need these people to keep the lights on.
Other more domain-focused data competencies should be fully integrated into the core domain of the organization.Companies can focus on organizational resilience through strategic talent acquisition (and retention) for fully integrated cross-functional stream-aligned teams.
What are some best-practices you have seen in your career that data leaders can learn from and implement in their work?
Data teams need to exist in the core product or other business area domain. Cascading shared objectives are quite helpful. The two main elements are:
Cascading
The executive leadership team identifies the strategic objectives for the company and the different departments. Data inherits these objectives and uses them as a basis for the data objectives.
Shared
The team works together towards a common objective.
Tactics can be different, but the strategy and objectives are shared.
Who is Stevan Lazic?
Stevan Lazić is a founder, entrepreneur, and product engineering leader, working for various enterprises, such as Chegg, AutoScout24, and Amboss, in Germany and the USA for more than 15 years.
After founding and bootstrapping the healthy food platform “Tangelo” as CTO & Co-Founder, he signed up for several engineering leadership and executive positions in Berlin, San Francisco, and New York.
His latest mission was to lead the organizational transformation of a newly created engineering chapter toward a business-cluster-driven matrix organization at Babbel.
His current focus is on organizational design, scaling product and engineering teams, and mapping business strategy to an ever-evolving day-to-day operating model and its respective tech strategy.
Related content to check out
Blogs
Operationtal KPIs that will let you know when you data team is adding value
Data strategy is part of corporate strategy
Podcasts
Data Operations on Analytics Anonymous with Valentin Umbach
More background about Convey’s Law.
