Tutorial 10: Causal Analytics to Improve User Sign Up Experiences

Learning Goals

By the end of this tutorial, students should be able to:

• Describe how experiments help evaluate product or design changes.
  
• Join and prepare experimental and event-level datasets to compute key outcomes such as funnel completion rates.
  
• Compare and interpret results across variations, expressing differences in both percentage points and relative terms, and evaluating whether they are meaningful in a business context.
  
• Communicate analytic findings in clear, decision-oriented language suitable for a managerial audience, using one concise table or figure.

The Business Challenge

A fast-growing fintech company has just completed an A/B test to evaluate a new sign-up process that existing users must complete before accessing a new feature release in their account dashboard. The goal of the experiment was to introduce this new process without creating excessive friction or drop-off during sign-up.

The company rolled out a Test variation of the sign-up flow, updating the layout and copy of key steps, while a Control group kept the original version. You’ve received detailed event-level data showing when each user started, moved through the funnel, and confirmed. The product team wants to know not only whether the Test increased completions, but why — and whether it created any new friction for users.

Your Task

1. Quantify the effect of the new variation on completion rates.
   
2. Interpret the magnitude of the improvement — is it large, small, or somewhere in between?
3. Explore additional metrics to understand user behaviour inside the sign up process.
   
4. Prepare an executive summary explaining whether the new sin should be rolled out to all customers.

Loading R Packages

Here’s the R packages we will need to complete the exercises:

library(tidyverse)   # data wrangling
library(lubridate)   # dates and times
library(janitor)     # cleaning up and duplicate checks
library(vtable)      # balance table

The data comes in multiple files that we will load as we need them.

Prepare these Exercises before Class

Prepare these exercises before coming to class. Plan to spend 30 minutes on these exercises.

Exercise 1: Understanding the Business Context

Before we start analysing the data, take a step back and think about the business motivation behind this experiment.

The company has introduced a new personalised insights dashboard — but to use it, existing customers need to go through an additional sign-up process.
From a business perspective, this is a risky move: any new step in the customer journey could add friction and cause users to drop off before completion.

In marketing and product analytics, we often visualise this process as a funnel — a series of steps users move through on their way to a goal (for example, from start to confirm).
Each step represents a decision point where users can continue or abandon the process.

1. Why might the company have decided to test a new sign-up flow rather than simply launching it for everyone?
   
2. What kinds of outcomes or trade-offs should the company be paying attention to?
   
3. What can a funnel tell us about customer behaviour that a simple completion rate cannot?
   
4. In this experiment, what would “success” look like from both the company’s and the customer’s perspective?

Exercise 2: Why Do We Test?

Before analysing the data, take a moment to think about why companies design controlled experiments in the first place.

The fintech company could have simply launched the new sign-up process for everyone and watched how overall sign-ups changed. Instead, they decided to run an A/B test, randomly assigning users to the Control (old process) or Test (new process) version.

1. Why is it risky to compare completion rates before and after the new sign-up process without a test?

2. How does random assignment in an A/B test help us make a fair comparison?

Exercise 3: Exploring the Experimental Data

It’s time to look at the data from the experiment.

You have two datasets:
- ab_test_demographics: information about each client.
- ab_test_experiment_clients: information on which clients were assigned to the Control or Test variation.

Our goal is to merge these datasets and check whether allocation to treatment and control groups is balanced across deomgraphic characteristics.

(a) Load the two datasets into R by completing the code below:

demogs <- YOURCODE 
treatment <- 
    YOURCODE |> 
    clean_names()

(b) Merge the treatment indicator variable into the demogs data by completing the code below. We recommend you use an inner_join() to do this.

demogs <- 
    demogs |>
    YOURCODE

(c) What are the unique values in variation column? If needed, drop any rows of data that you won’t be able to use in the Treatment vs Control comparison.

# how values in variation?
demogs |> 
    YOURCODE

# Perform any required operations to remove rows
# from the data
demogs <-
    demogs |> 
    YOURCODE

(d) In a well-designed experiment, each variation should include comparable types of users. How does connect back to the idea of random assignment?

(e) Complete the code below to compare demographic compositions of users across treatments. Explain your findings.

demogs |> 
    select(-client_id) |> 
    st(group = 'YOUR_TREATMENT_INDICATOR')

(f) Why is checking balance between treatments the first thing we do before comparing performance?

In-Class Exercises

You will discuss these exercises in class with your peers in small groups and with your tutor. These exercises build from the exercises you have prepared above, you will get the most value from the class if you have completed those above before coming to class.

Exercise 4: Working with Web Log Data

Now that you’ve checked that users were randomly assigned to the Test and Control groups, it’s time to examine what they actually did inside the experiment.
In this step, you’ll work with event-level data — information about each time a user interacted with the sign-up process.

(a) Load and inspect the web log data. The data are split across two files. Run the code below to combine them into one dataset.

weblogs_files <- 
    c('data/ab_test_web_data_pt_1.txt',
      'data/ab_test_web_data_pt_2.txt')

weblog <- 
    weblogs_files |> 
    map_df(read_csv)

glimpse(YOURCODE)

(b) Explain the structure of the data. What kind of information does each row represent?

(c) A common problem when working with event log data is duplicate rows, where the same information is recorded multiple times. Clean the data to remove duplicate rows by completing the code below:

weblog_clean <- 
    weblog |> 
    YOURCODE

(d) Merge the cleaned web log data with the dataset that contains information on which clients are assigned to each treatment. Drop any observations that are not assigned to either “Test” or “Control”.

weblog_clean <- 
    weblog_clean |> 
    YOUR_CODE |> 
    YOUR_CODE

Exercise 5: Measuring Funnel Completion

Now that we’ve linked event-level data with the experimental information, let’s create our first outcome measure — funnel completion.

Each client goes through a series of process steps (for example, start, step_1, step_2, confirm).
A funnel completion occurs if the client reaches the final step, "confirm".

(a) Compute a client-level indicator for whether each client completed the funnel by completing the code below.

completion_by_visitor <- 
    YOUR_CODE |>
    group_by(YOUR_CODE) |>
    YOUR_CODE(
        completed = any(YOUR_CODE == "confirm"),
        .groups = "drop"
    )

(b) Summarise completion rates by treatment.

completion_summary <- 
    YOUR_CODE |>
    YOUR_CODE |>
    summarise(
        completes = YOUR_CODE,
        total     = YOUR_CODE,
        pct       = completes / total * 100,
        .groups   = "drop"
    )

(c) Create a simple visual to compare completion rates across the two variations. Use a bar chart to display the percentage of users who completed the funnel.

completion_summary |> 
    YOUR_CODE +
    YOUR_CODE +
    geom_text(aes(label = round(pct, 1)), vjust = -0.5) +
    labs(
        title = "Funnel Completion Rate by Variation",
        x = "Variation",
        y = "Completion Rate (%)"
    ) +
    theme_minimal() +
    theme(legend.position = "none")

(d) Interpret your findings. By how many percentage points did the change in sign up process influence completions?

Exercise 6: Comparing Outcomes Across Variations

We’ve now calculated completion rates for each variation.
In this exercise, you’ll quantify how much higher (or lower) the completion rate was in the Test variation compared to the Control group — and reflect on whether the difference is meaningful.

(a) Compute the difference in percentage points between the Test and Control variations.

completion_summary |> 
    summarise(
        diff_pct_points = YOUR_CODE
    )

(b) Compute the percent change in completion relative to the Control group.

completion_summary |> 
    mutate(
        YOUR_CODE
    )

(c) You already have the difference in completion rates from part (a).
Let’s use that value directly and divide it by the standard deviation of completion in the Control group to see how large the effect is in practical terms.

# pull the numeric difference from part (a)
diff_rate <- completion_diff |> pull(diff_pct_points) / 100

# compute Control group’s SD
control_sd <- 
    completion_by_visitor |> 
    YOUR_CODE|> 
    YOUR_CODE(sd_rate = YOUR_CODE) |> 
    pull(sd_rate)

# difference in standard-deviation units of the control group
diff_sd_units <- diff_rate / control_sd
print(diff_sd_units)

(d) Would you consider this a small, moderate, or large effect in a typical online experiment? Explain Your answer.

Exercise 7: Communicating Experimental Insights

Now we need to translate your analytical results into a short executive brief written for a business audience.

Your goal is to communicate what was tested, what happened, and what the company should do next — clearly and persuasively, without technical detail.

Instructions

Length: 400 words.
Exhibit: 1 table or figure.

Write this as an executive brief to be read and understood by managers.

• Bullets must be written as full sentences.
  
• Use the writing principles from Write Like an Amazonian — short, declarative sentences; clarity before style.
  
• No visible code in the brief.
  
• Do not produce new analysis, tables, or figures in this section — use the results already created in Exercises 5–6.
  

INSERT YOUR FIGURE OR TABLE HERE
(For example: your completion-rate comparison chart or summary table from Exercise 5.)

Executive Summary

3–4 sentences summarising the research question, key findings, and why they matter for the company.
Hint: State the business problem, the intervention, and the observed difference in completion rates.

Key Insights

• Up to 3 bullets describing what the data and analysis show.
  
• Focus on evidence — e.g., “Completion rates were 3.7 percentage points higher in the Test group.”
  
• Avoid statistical jargon; use plain, business‐relevant language.

Business Implications

• Up to 3 bullets on why these findings matter for the organisation.
  
• Connect the analytics result back to strategic goals.

Recommended Actions

• Up to 3 bullets outlining clear, actionable next steps.
  
• Recommendations should be specific, realistic, and aligned with business priorities.

Exercise 8: Understanding Mid-Funnel Outcomes

After reviewing your executive brief, one manager comments:

“This tells me how many users finished the sign-up flow — but I want to understand what happened in the middle.
Where exactly are we gaining or losing customers?”

(a) What does the manager mean by “mid-funnel outcomes”?
How are these different from overall completion rates?

(b) Why might focusing only on completions hide important insights about customer experience or friction points?

(c) Intuitively, how could you use the same event-level data to explore mid-funnel performance?
Describe two or three possible ways (e.g., visual summaries or simple metrics) without writing any code.

(d) What kinds of business questions could mid-funnel analysis help answer that top-line completion rates cannot?