Background¶

Our company XYZ has an initiative to improve email signup rates when users visit our website. The current design is clunky and we want to test a more efficient flow. Marketing works with data science to design an experiment where 50% of users that reach the website are randomly assigned to the control group, while 50% of users in the treatment group observe the new flow.

In this example, we simulate an experiment where the control users are 30% likely to sign up for email marketing and the treatment drives a relative increases in the signup rate by 10%.

Simulate data¶

In [5]:
import numpy as np
import pandas as pd 

np.random.seed(12345)

# experiment sample size
num_users = 100000

# randomly assign treatment and control groups
groups = np.random.choice([0,1],size=num_users,p=[0.5,0.5])

# probability of signup for treatment and control
control_p = 0.3
treatment_effect = 0.1
treatment_p = control_p * (1+treatment_effect)

# generate signups based on group membership and probabilities 
signup = np.where(groups==0,np.random.choice([0,1],size=num_users,p=[1-control_p,control_p]), 
                  np.random.choice([0,1],size=num_users,p=[1-treatment_p,treatment_p]))

# combine into dataframe
data = pd.DataFrame({
    'treated':groups,
    'signup': signup
})
In [7]:
print(data.groupby(['treated'])['signup'].agg(['count','mean']).round(2))

         count  mean
treated             
0        49952  0.30
1        50048  0.33

Evaluate experiment using linear regression¶

In [26]:
import statsmodels.formula.api as smf 

formula = 'signup ~ treated'
model = smf.ols(formula,data).fit()

print(model.summary())

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                 signup   R-squared:                       0.001
Model:                            OLS   Adj. R-squared:                  0.001
Method:                 Least Squares   F-statistic:                     131.4
Date:                Sat, 30 Dec 2023   Prob (F-statistic):           2.06e-30
Time:                        10:48:54   Log-Likelihood:                -65247.
No. Observations:              100000   AIC:                         1.305e+05
Df Residuals:                   99998   BIC:                         1.305e+05
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
Intercept      0.2992      0.002    143.938      0.000       0.295       0.303
treated        0.0337      0.003     11.465      0.000       0.028       0.039
==============================================================================
Omnibus:                  1242876.903   Durbin-Watson:                   2.001
Prob(Omnibus):                  0.000   Jarque-Bera (JB):            18215.398
Skew:                           0.789   Prob(JB):                         0.00
Kurtosis:                       1.629   Cond. No.                         2.62
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Extract key values from the regression summary¶

In [33]:
print("The estimated absolute impact is: {0:.2f}% \
      \nThe estimated relative impact is {1:.2f}% \
      \nThe t-statistic is {2:.1f} \
      \nThe p-value is {3:.2f}%".format(
        100*model.params['treated'],
        100*model.params['treated']/model.params['Intercept'],
        model.tvalues['treated'],
        100*model.pvalues['treated']
))

The estimated absolute impact is: 3.37%       
The estimated relative impact is 11.26%       
The t-statistic is 11.5       
The p-value is 0.00%

Replicate using a two-sample t-test comparing means¶

In [38]:
from scipy.stats import ttest_ind 

treated_users = data[data['treated']==1]['signup']
control_users = data[data['treated']==0]['signup']

t_stat, p_value = ttest_ind(treated_users, control_users)

print(f"t-statistic: {round(t_stat,1)}")
print(f"p-value: {100*round(p_value,3)}%")

t-statistic: 11.5
p-value: 0.0%

Conclusion¶

We simulated data for an experiment where the true effect of the treatment increased signup rates by a relative 10% from a baseline signup rate of 30%. Naturally there will be sampling error as we only observe users in the experiment. In this case, our estimated treatment effect was a relative increase of 11.3% and the result was highly statistically significant (p <<< 0.05).

In [ ]: