When I took on the challenge of creating Babbel Live, we had already tried and failed building an online live language classes service twice. Although Babbel was a large, established player in online language-learning, we didn’t have the product on the market yet. Other companies had well-established offerings for live language instruction. The lockdown was having a positive effect in people’s willingness to learn a language online, and other players were growing quickly.
While we understood our user needs in depth, we had learned from previous attempts that delivery speed was a missing ingredient for us.This time around, we improved by using data to inform our decisions and gain product delivery speed. The two ways we wanted to create data-informed decisions were:
- Data-informed product development decisions
- Better progress tracking
Data-informed vs. data-driven decisions?
Data-driven decisions take a quantitative point of information and base the reasoning primarily on that, while data-informed complements the thinking with qualitative consumer insights, UX and market research.
When you’re developing new products and businesses, you cannot be only data-driven in your decisions. Your early product development decisions have to be triangulated with qualitative input, i.e. user experience (UX) and consumer insights, and market intelligence. You can also look at prices, portfolio offerings, messaging architecture, product positioning. These are things that you can track a lot more regularly than the typical market research that company does once a year.
Implementing a short data cycle was key to success. Here are some learnings.
This time, we wanted to create a very short data cycle so that we could be fast in taking decisions. Key components of this data cycle were:
Automation
You can automate the collection of qualitative input. We were able to make decisions because we had feedback points into the hundreds. A researcher did not have to sit in 300 interviews to 300 feedback points. He could use 2 hours a week to synthesize the information and another hour to present it with the team. The rest of his time was spent on high value add activities such as iterating and improving the feedback gathering mechanisms, localizing the questions, improving the questions, and managing the
UX researcher
We prioritized hiring a UX researcher, and we added a feedback form at the end of every class. These two things were the key enablers for us on a weekly basis to create a healthy amount of data points and have somebody who can synthesize that data into insight.
Early event tagging is not the answer
People think that you should tag events right away. Sure it’s beneficial, yet the problem is that when you do quantitative analysis at a very early stage, you hope that this will give you valuable data, but the truth is that if you have 50 people using the product, it will never be enough to inform decisions. For that reason, in an early stage of a product, you need to use qualitative inputs.
Prioritization of hypotheses
The product team would discuss and prioritize what hypothesis to test through product improvements. For example we would say our goal is to have people joining more classes per week. Our hypothesis would be for example, if I schedule additional classes during lunch, i.e. when people are free, they will book more and student activity will increase. Imagine literally hundreds of such hypotheses. We would all have things we wanted to try. You have to prioritize what are the most important hypotheses to validate.
Hierarchy of hypothesis
As a lead, I told the team to create a hierarchy of hypotheses, so that we could see the dependencies. We would go for the most critical hypothesis first and help us deprioritize.
Better progress monitoring leads to a more accurate path to success.
Middle posts, meaning midway milestones, are critical elements that teams often forget. Product teams often have long conversations about what the product will look like. It doesn’t matter if we have an accurate view of the destination, if we are flying blind on our journey. The middle points serve as street lights along the road to the destination. They helped me assess sooner whether I was diverging, and they helped me measure if I was going too slow. You need to identify what are the metrics for each goal that you can measure along the way that validate that end goal.
We identified key milestones
- Can we build it without this thing breaking? We needed to reach a level where a couple hundred students were using the platform every month, and it didn’t break.
- Do people want to use it? We had a couple of thousand students using Live for free regularly, thus we knew we were adding value.
- Would they pay for it? We validated the willingness of students to pay with paywall experiments and measured conversions.
Habits that enabled success
Along the path, I encouraged my leads and executive team to spend less time theoretically discussing the ultimate product vision and instead redirected our attention to defining clear milestones, monitoring them regularly and generating insights to refine both the product development and business vision later.
Not because an exciting vision is not necessary, rather because milestones and mastering them are more crucial for early product survival and, later on, successful implementation.
Involve the entire agile team in a weekly data review meeting
We had a weekly data review meeting with the whole team. There we would ask: „have our product changes had a positive effect in 1) usage and 2) intent to use?” The meeting would take place on Fridays so that we could digest the information and use it as input in Monday planning.
We stopped assuming that our subjective preferences were the same as the users’, because on a weekly basis we were faced with the reality of user feedback.
Through these regular syncs we learnt things like.
- Users don’t mind THAT much an ugly booking system, so long the quality of the teachers and the class is great
- User onboarding on how to get to the call was MORE crucial than all the other features (add class to calendar, download class material, etc)
- Teachers needed a LOT more tech support that anticipated
Monthly progress reviews
With product leaders (CTO, CPO, VP Learning) we were meeting monthly, to review progress towards milestones and understand as a group what were the drivers of our performance or underperformance.
In this round, it was also key to leave our subjective preferences aside – what we wished this new product could do for us as a business, from a learning efficacy and revenue growth potential. Instead, we kept each other honest by doubling down on what the user feedback was telling us and using THIS information to refine the product vision.
Conclusion
Successful product leads don’t need to be Elon Musk or Steve jobs. They can look like anybody. I would encourage us to move away from the archetype of a visionary product lead, aka crazy scientist having eureka moments in his lab. Instead, we should welcome product leads who are pragmatic listeners.
Who is Belen Caeiro?
Belen is VP of Product at Babbel, where she leads the product development and operations teams building the new online tutoring services (group classes, 1:1 classes).
Prior to Babbel, Belen led the international roll-out of SoundCloud’s monetization options, and was involved in the early-stage development of other online companies in the entertainment and ticketing industries.
Related content to check out:
From D3M Labs:
Consumer insights and data analytics: The ying and yang of how and what
Belen’s thought leadership
How to create a success in a crowded market with Belen Caeiro (Babbel)
Reforge’s take on early product development
