The right mix of governance and freedom in architecture is still up for debate, end-to-end solutions are often-heard recommendations, low & no-code is expanding access to data and the customer journey could be seen as a source of revenue are some insights I gleaned from this spring’s Big Data World.
Finally, being able to see people again live, instead of through a video chat was great. There were many inspiring talks, interesting individuals to meet and great vendors to discover. Here are some of the key takeaways that I have from the Big Data World in Frankfurt.
Data strategy is hard to implement, and nobody has found the model data organization
- Many people talk about data strategy, but the talks were more about the implementation of data strategy rather than what it actually is! The talks and conversation point to data strategy meaning everything from part of the actual business strategy to a plan how to manage data, sometimes even data education and professional development.
- There seemed to be haze of disillusioned with the central data team and company data platform, coupled with the frustration with the mess of decentralization and failed self-service attempts. While data teams are expensive, often answering massive queues of tasks and projects and thus perceived as bottlenecks in the organization, non-data teams struggle to understand and work with the data. No consensus seemed to exist on the ideal organization.
Data architectures are trying to answer the conundrum of how to make data accessible AND governable
- Data warehouse or data lake? Neither has lived up to its promise. Data warehouses must be well-defined to be useful and there are too many joins. Furthermore, once you design one, the entire organization has changed, and you have to design it all over again. Data lakes turn into data swamps, as do data warehouses left to run afoul.
- Data meshes are to the data warehouse what microservices are to the monolith. Data meshes break up the data warehouse and bring the data back to the domains. Zalando has made data headlines by implementing a data mesh with Data Bricks.
- Data lakehouse: This is another architecture I look forward to learning about. Thanks to a metadata level that you can query, the data lake has the findability of a data warehouse with the flexibility of a lake.
- Data vault is an architecture that dates back to the 1990’s, but one that I learned about at the conference. Data vaults are an architecture that is supposed to provide the structure and findability of a data warehouse with the versatility provided by other architectures. Thanks to the team at VaultSpeed for taking time to explain the concept to me and giving me this book. Data Vaults provide an organizational ontology with hubs for different business entities and satellites for the different attributes, with links describing the relationship between the hubs.
With great freedom comes great responsibility. Data cataloging and governance (some of my favorite topics, as my close contacts know), are becoming hot items.
- Centralized definitions for KPIs and metrics that are coded into the information management platform was a priority that was often mentioned. Inconsistencies in definitions, dashboards and messy data pipelines were common topics in many talks and discussions.
- Governance across the data lifecycle, as well as quality control are formerly also-ran topics that are now hot. Governance and findability were mentioned as being especially important in decentralized organizations and with accompanying architectures such as the data mesh.
- User profiling and measuring usage of the data is another feature that vendors are increasingly offering so that data teams can truly become customer-centric, also internally.
Is coding a bottle neck?
No code and low code where hot topics as well. See my LinkedIn post on this subject. In essence, requiring quality code is a roadblock to usage, maintenance and governance of a data platform. The less code is required, the more people can build and manipulate data.
Customer centricity begins with data and can end with profit
- Don’t cheap out on a customer data platform (CDP) – or data management platform. There is a difference between the two. Having solid data management and automation processes in marketing is important for the multitude of customer-centric use cases. There was a consensus to focus on tools that are comprehensive in their functionality as well as easy to use – and low code, no code.
- The customer journey can also become a profit center. Customer journey, customer segmentation, stitching user data into profiles are all tasks that data teams do as part of customer centricity. Such tasks are often seen as overhead, supporting, for example, marketing activities or strategy and planning. There were also use cases presented for how contact with the customer throughout the relationship should be human-centric, as well as taking on an important retention and sometimes compassionate role.
After the two-day conference, listening to talks and speaking with vendors, I have a feeling that the data industry is maturing, getting more organized, getting more strategic, but still unsure about how to fulfill the promise of data being the new oil.
