{"id":1129,"date":"2023-03-13T17:28:57","date_gmt":"2023-03-13T17:28:57","guid":{"rendered":"https:\/\/d3mlabs.de\/?p=1129"},"modified":"2023-04-19T16:39:02","modified_gmt":"2023-04-19T16:39:02","slug":"solving-the-experimentation-dilemma-and-growing-the-new-york-times-an-interview-with-shane-murray","status":"publish","type":"post","link":"https:\/\/d3mlabs.de\/?p=1129","title":{"rendered":"Solving the speed vs. quality experimentation dilemma and growing the New York Times- an interview with Shane Murray"},"content":{"rendered":"\n

What were your initial career interests? I see you studied mathematics and finance.  <\/strong><\/p>\n\n\n\n

For me there was a mix of passion and pragmatism. When I left high school, I was not sure what I wanted to do. Mathematics was an interest area for me, and finance was a pragmatic way to use my skills. The double degree in mathematics and finance at the University of New South Wales in Australia, where I grew up, was useful to learn how to apply mathematics in the real world. There are elements of the degree that I use today. Applied probability theory and decision making under uncertainty, for example, are subjects that I\u2019ve leaned on throughout my career in analytics.<\/p>\n\n\n\n

Why did you decide to go into analytics?<\/strong><\/p>\n\n\n\n

I went through a handful of interview rounds with hedge funds, which were competitive. Finally, I landed a job with a research firm in Sydney, Australia. At this firm, I applied experiment design and discrete choice modeling frameworks using survey-based data to understand the attributes that go into a decision and then simulate how populations make decisions. Then I landed a job at Memetrics, a software startup, where I was running large scale digital experiments for customers across the globe and contributing new algorithms to automatically analyze these experiments in our software.<\/p>\n\n\n\n

You write, among other subjects, about experimentation. What interests you about experimentation?<\/strong><\/p>\n\n\n\n

I\u2019ve always had an interest in behavioral science and seeking to understand the rational and irrational drivers behind how humans make decisions. That is essentially the job of analyst and data scientists in most companies today.<\/p>\n\n\n\n

\n

Experiments allow you to simplify the complexity of how humans make decisions into probabilistic models, then make an informed and occasionally transformative business decision.<\/em><\/p>\n<\/blockquote>\n\n\n\n

What is experimentation for you?<\/strong><\/p>\n\n\n\n

The range of techniques I have used start with discrete experiments, extend into adaptive techniques like multi-armed bandits, and include natural or quasi-experiments where you lack the ability to randomly assign treatments. But discrete A\/B and multivariate experiments tend to be the bread and butter of most analytics teams. <\/p>\n\n\n\n

Why is it so challenging for so many organizations to implement experiments?<\/strong><\/p>\n\n\n\n

In my experience, the best experimentation is multi disciplinary.<\/p>\n\n\n\n

\n

An experiment starts with a hypothesis, which leans on the expertise of different disciplines; editors, product designers, engineers, marketers. <\/em><\/p>\n<\/blockquote>\n\n\n\n

The work led by analytics teams, collaborating across disciplines, includes: The hypothesis definition, the translation of that hypothesis into an experiment design, the definition of outcomes to measure, segments to explore, and the required sample size and duration. <\/p>\n\n\n\n

Building and implementing experiments usually falls on design and engineering teams, before an analyst or data scientist delivers statistical models or tests, and interprets the results to inform a decision.<\/p>\n\n\n\n

Most of the experiments I have run have been based on digital experiences. At the New York Times, this could include the paywall, redesigning the homepage, content recommendations, product marketing and pricing.  Prior to The Times I consulted with organizations like American Express to run similar experiments on marketing landing pages, emails, paid search, social, and even direct mail inserts.<\/p>\n\n\n\n

Speed and quality seems to be the eternal friction point. Marketing and product often want the former while data teams struggle to argue for more thoroughness and time in the name of quality. Why is that?<\/strong><\/p>\n\n\n\n

Without the right foundation in place, you often observe an inverse relationship between speed and quality. Speed is imperative in digital product development for a number of reasons; from building momentum in a team, finding wins that show you\u2019re on the right path, to beating your competition.<\/p>\n\n\n\n

But in experimentation, speed can often amount to short cuts, such as not defining hypotheses, or not collecting sufficient sample. These shortcuts can ultimately be detrimental to the quality of the decision. <\/p>\n\n\n\n

So, invariably you find this dynamic, where product or marketing Stakeholders push for speed and analysts are the bastions of quality.<\/p>\n\n\n\n

What can be done to solve this conflict of speed vs. quality?<\/strong><\/p>\n\n\n\n

The onus is on the data teams to build the capabilities and processes to scale experiments with quality built in. For a start, education about methodology and statistics is important. Rolling out lightweight processes for experimental design review so you can critique and improve before launch is also good.<\/p>\n\n\n\n

Often in the name of speed, experiments are set up and deployed without a clear understanding of what success looks like and how a \u201cwinner\u201d will be decided. Analysts then wind up providing analysis of multiple outcomes and scenarios. This slows down the process of experimentation. and puts onus on the analyst post experiment to come up with compelling insights that lead to a decision. <\/p>\n\n\n\n

\n

If, instead, you can align on a clear success criteria in the planning process, that would be helpful. For more complex decisions, you should also have alignment about the trade off between different outcomes. At the New York Times, that could be a trade off between subscriptions and reader engagement. Aligning on such trade-offs will make the decision process smoother. <\/em><\/p>\n<\/blockquote>\n\n\n\n

How could that look organizationally?<\/strong><\/p>\n\n\n\n

It usually makes sense to start with a central, cross functional team that are owners of process and technology. Then when you decentralize in order to scale experimentation across an organization, these are often the people who can advocate for best practices in the respective functions.<\/p>\n\n\n\n

Are there a few methods, mind-sets or habits that could create a more positive culture of collaboration? <\/strong><\/p>\n\n\n\n

Mindset: Respecting expertise and different perspectives, across product, editorial, marketing, design, engineering and data. <\/p>\n\n\n\n

Habits: Focus on hypotheses over \u201cideas\u201d. It should be more about proving well-defined hypotheses than testing your own ideas.<\/p>\n\n\n\n

Experimentation projects often stall or take significantly longer than expected. Why do people often get caught off-guard by the complexity?<\/strong><\/p>\n\n\n\n

Technical implementation.<\/p>\n\n\n\n

Implementation is often more challenging than people realize, because when deploying an experiment across different systems, there can be a bunch of unknowns about how these systems will work together, especially given you are introducing new code to the system. <\/p>\n\n\n\n

Additionally, there is often a disconnect between analytics teams and technical counterparts about what you need for a successful experiment, such as the specifics of random allocation. <\/p>\n\n\n\n

The choice to build in-house vs. buy\/subscribe with a vendor is often a point of contention in analytics teams. What principles do you suggest?<\/strong><\/p>\n\n\n\n

In my opinion, the decision to build should be a fairly high bar. If you are running simple experiments on marketing landing pages, then there are many solutions to fill that need. However, if you are running more complex product experiments that must integrate with various backend and frontend systems, or need to measure a wide variety of user behaviors and outcomes, then the decision is less clear.  <\/p>\n\n\n\n

For example, at The New York Times, we elected to build a solution in 2014 given we needed an allocation mechanism that could be customized and sit within our tech stack to reduce latency in the page load, and we often ran experiments where you wanted to understand the impact across a wide array of reading habits, subscription outcomes and advertising revenue that were available in our data warehouse.<\/p>\n\n\n\n

So for me, there are two main sources of complexity that you should consider: <\/span><\/p>\n\n\n\n

Measurement<\/em>: How complex are the user behaviors and outcomes that you need to track across experiments? The more complex, the more it makes sense for the solution to be natively integrated with your data warehouse. <\/p>\n\n\n\n

Allocation:<\/em> How much will you need to tailor the way users are allocated to experiments, both in terms of how a user is identified, and how integrated these frameworks are with your frontend and backend systems?  <\/p>\n\n\n\n

These days, I\u2019m seeing more vendor tools emerge that help data teams to scale product experimentation with warehouse-native solutions, so that you avoid creating data silos.<\/p>\n\n\n\n

Are compromises necessary? If yes, what are a couple of compromises that could be acceptable?<\/strong><\/p>\n\n\n\n

In general, data teams should look for ways to systematically increase experimentation speed and autonomy. <\/p>\n\n\n\n

Automated reporting that allows for decentralized autonomy. <\/span><\/p>\n\n\n\n

Data teams can automate analysis and reporting that covers the most common metrics, such that teams have immediate access and can make faster decisions. Provided these decentralized teams are not seeing a negative impact on a critical counter-metric, they can make decisions that maximize their goal.  <\/p>\n\n\n\n

Forecasting future value based on experiments:<\/span><\/p>\n\n\n\n

Often data teams are attempting to optimize Lifetime Value, but must either run experiments for many months or use near-term indicators of lifetime value. <\/p>\n\n\n\n

The Dropbox data team builds user-level expected revenue metrics that takes weeks rather than months to estimate. They are using the behavior they observe in the experiment and immediately after to project or forecast the expected revenue of a user over the span of the next few years, allowing them to make a recommendation on the experiment that approximates the long term impact of the business. <\/p>\n\n\n\n

\n

Evaluating return on experimentation and course correcting where speed is not having the desired impact. You can do a meta analysis of your experiments to understand where you are getting return and where the practices might need to be improved. <\/em><\/p>\n<\/blockquote>\n\n\n\n

What surprised you about being a data leader at the New York Times? <\/strong><\/p>\n\n\n\n

Despite being a 170 year old company steeped in tradition, The New York Times is undergoing continual transformation. Arriving in 2013 and spending nearly a decade there, we shifted from print to digital, desktop to mobile, and advertising to subscription-first. It was a pivotal time to be building out a data team that played a central role in this transformation.<\/p>\n\n\n\n

How was it being a data leader at the New York Times while Donald Trump was President of the USA?<\/strong><\/p>\n\n\n\n

While we were measuring the impact of each news cycle, including those related to the Trump administration, on the subscription business, it was critical as a data team to understand how we could drive sustainable subscription growth beyond any news cycle.  <\/p>\n\n\n\n

\n

The role of the data team, in building the business from less than a million to more than 8 million digital subscribers, included modeling the reader behaviors that lead to subscription, experimenting with different ways to balance the free and paid experiences, and informing decisions on where and how to participate in news experiences on large platforms owned by Google, Facebook, Apple and others.  <\/em><\/p>\n<\/blockquote>\n\n\n\n

How was it being an immigrant working at such a New York institution during such a critical moment in American history?<\/strong><\/p>\n\n\n\n

It was thrilling. I always had an interest in journalism and have a brother who was a radio journalist in Australia. There is simply no better place than The Times to observe and be part of journalism in America. <\/p>\n\n\n\n

The mission of The Times, to seek truth and help people understand the world, is important and meaningful to me. And a culture that rewards integrity and curiosity was incredibly well aligned with how I wanted to run a data team. <\/p>\n\n\n\n

What would surprise other people inside and outside the USA?<\/strong><\/p>\n\n\n\n

The New York Times has become a tech company, albeit one with a journalistic core, which makes it unique and impressive. When I left, we had built up a data team with more than 150 people, including a data platform team, analysts providing insights and running experiments across the business and newsroom, and applied machine learning teams that were having a direct and significant impact on the business. <\/p>\n\n\n\n

Does it still have a print version?<\/strong><\/p>\n\n\n\n

Yes, I still get print delivered. The print plant in College Point Queens is a great place to see print robotics on display. <\/p>\n\n\n\n

You are now an executive at Monte Carlo. Why are you so passionate about data reliability? <\/strong><\/p>\n\n\n\n

Throughout my career in data, I felt like we had one hand tied behind our back because of data quality and reliability issues.<\/p>\n\n\n\n

You are providing data to the business, in the form of dashboards, insights and machine learning, that are increasingly relied upon to make decisions and drive operations. Ultimately, data teams experience loss of credibility if you are making decisions based on unreliable data, and more often today this is tied to a direct loss of revenue. My team was writing manual tests to check the quality of the data, but these methods are tricky to scale. <\/p>\n\n\n\n

Once I was a customer of Monte Carlo, I saw the shift from going from manual to automated monitoring, and being able to see the upstream issues and downstream impact that allow you to effectively troubleshoot data incidents. Ultimately, this leads to a real impact on reliability and trust in data. <\/p>\n\n\n\n

My role at Monte Carlo gives me the opportunity to partner with our customers on the operational changes and transformation that comes with Monte Carlo, to build trust in their data and drive success in their initiatives. <\/p>\n\n\n\n

What have been a few secrets to your success?<\/strong><\/p>\n\n\n\n

The three things that have helped me. <\/p>\n\n\n\n

    \n
  1. Building expertise around a specific problem space, which for me was experimentation. Going deep in methodology, research and being an active participant in communities driving how experimentation is done.<\/li>\n\n\n\n
  2. As I have grown into more senior roles, curiosity about different disciplines. I relied as much on partners in design and tech as I did on peers in data.<\/li>\n\n\n\n
  3. Having managed large organizations, I learned that management is frequently about caring about peoples\u2019 careers, problems and frustrations. Occasionally you might impart some wisdom, but mostly you are creating the conditions for people to succeed. <\/li>\n<\/ol>\n\n\n\n

    <\/p>\n\n\n\n

    Who is Shane Murray<\/strong>?<\/p>\n\n\n

    \n
    \"\"<\/figure>\n<\/div>\n\n\n

    Shane is the field chief technology officer at Monte Carlo, partnering with data leaders on their data strategy and operations, to realize the maximal value from their data observability and data quality initiatives. Prior to Monte Carlo, Shane was the senior vice president of data & insights at The New York Times, leading 150+ employees across data science, analytics, governance and data platforms. <\/em><\/p>\n\n\n\n

    <\/p>\n\n\n\n

    Other articles from the contextualizing the world with data series.<\/strong><\/p>\n\n\n\n

    Part 1, Advertising: Why creatives in advertising should embrace data science and data mining \u2013 an interview with Les Guessing<\/a>.<\/strong><\/p>\n\n\n\n

    Part 2, Customer Relationship Management: Nurturing the customer relationship with data \u2013 an interview with Sarah Carr.<\/a><\/strong><\/p>\n\n\n\n

    Part 3, Public Relations: On the communication front with the Ukrainian PR army \u2013 an interview with Liuka Lobarieva. <\/a><\/strong><\/p>\n\n\n\n

    <\/p>\n\n\n\n

    Related content<\/strong><\/p>\n\n\n\n

    <\/p>\n\n\n\n

    The prevelance of AI and the importance of engaging in dialogue<\/strong><\/a> \u2013 A blog by Varsh Anilkumar<\/p>\n\n\n\n

    Beyond AI, the realities of operationalizing AI \u2013 A podcast interview with Elizabeth Press<\/strong><\/a><\/p>\n\n\n\n

    Why the public needs to know more about AI \u2013 An interview with Varsh Anilkumar<\/strong><\/a><\/p>\n\n\n\n

    What is the future of AI adoption?<\/strong><\/a><\/p>\n\n\n\n

    Check out other D3M Labs Series- In depth look at specific topics that spans multiples interviews and posts<\/strong><\/p>\n\n\n\n

    How to manage a data science product<\/a>    Deutsch<\/a><\/h3>\n\n\n\n

    Data science is moving from R&D into products \u2013 both online and off. Managing data products requires evolution from both traditional software and hardware development.<\/p>\n\n\n\n

    The future of the analyst<\/a><\/h3>\n\n\n\n

     Is the role of the analyst endangered? What is the future of the most visible role in analytics, and the one responsible for delivering the insight? What is the future of the analyst?<\/p>\n","protected":false},"excerpt":{"rendered":"

    Contextualizing our world with data, part 4: Journalism. Solving the speed vs. quality dilemma and growing the New York Times, also during the Trump years. Shane Murray, Field Chief Technology Officer at Monte Carlo and former Senior Vice President of data & insights at The New York Times, talks with about experimentation and growing a digital subscriber business, the New York Times. Shane talks about how to solve the experimentation speed vs. quality dilemma \u2013 and often outright conflict \u2013 between business stakeholders and data teams. Shane also talks about how the New York Times transformed itself into a digital subscription product and tech company. <\/p>\n","protected":false},"author":1,"featured_media":1131,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[41,38,21,24,17],"tags":[],"class_list":["post-1129","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-data-careers","category-data-education","category-data-quality","category-product-analytics","wpcat-41-id","wpcat-38-id","wpcat-21-id","wpcat-24-id","wpcat-17-id"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t