Now, we now have the options for our mannequin. We’ll break up our information into 3 units:
1- Coaching dataset : It’s the set of information the place we’ll prepare our mannequin
2 – Take a look at dataset : Knowledge used to judge the efficiency of our mannequin.
3- After modification dataset: Knowledge used to compute the uplift utilizing our mannequin.
from sklearn.model_selection import train_test_splitstart_modification_date = dt.datetime(2024, 2,1)
X_before_modification = X[X.index < start_modification_date]
y_before_modification = y[y.index < start_modification_date].kpi
X_after_modification = X[X.index >= start_modification_date]
y_after_modification = y[y.index >= start_modification_date].kpi
X_train, X_test , y_train , y_test = train_test_split(X_before_modification, y_before_modification, test_size= 0.25, shuffle = False)
Word : You need to use a fourth subset of information to carry out some mannequin choice. Right here we gained’t do loads of mannequin choice, so it doesn’t matter lots. However it can should you begin to choose your mannequin amongst tenths of others.
Word 2: Cross-validation can be very doable and beneficial.
Word 3 : I do suggest splitting information with out shuffling (shuffling = False). It’ll enable you to pay attention to the eventual temporal drift of your mannequin.
from sklearn.ensemble import RandomForestRegressormannequin = RandomForestRegressor(min_samples_split=4)
mannequin.match(X_train, y_train)
y_pred = mannequin.predict(X_test)
And right here you prepare your predictor. We use a random forest regressor for its comfort as a result of it permits us to deal with non-linearity, lacking information, and outliers. Gradients Boosting Bushes algorithms are additionally excellent for this use.
Many papers about Artificial Management will use linear regression right here, however we expect it’s not helpful right here as a result of we’re not actually within the mannequin’s interpretability. Furthermore, deciphering such regression will be difficult.
Counterfactual Analysis
Our prediction might be on the testing set. The primary speculation we’ll make is that the efficiency of the mannequin will keep the identical after we compute the uplift. That’s the reason we have a tendency to make use of loads of information in our We contemplate 3 totally different key indicators to judge the standard of the counterfactual prediction :
1-Bias : Bias controls the presence of a spot between your counterfactual and the true information. It’s a robust restrict in your capacity to compute as a result of it gained’t be diminished by ready extra time after the modification.
bias = float((y_pred - y_test).imply()/(y_before_modification.imply()))
bias
> 0.0030433481322823257
We usually categorical the bias as a proportion of the typical worth of the KPI. It’s smaller than 1%, so we must always not count on to measure results larger than that. In case your bias is simply too huge, you need to test for a temporal drift (and add a pattern to your prediction). You too can appropriate your prediction and deduce the bias from the prediction, supplied you management the impact of this correction of recent information.
2-Customary Deviation σ: We additionally need to management how dispersed are the predictions across the true values. We subsequently use the usual deviation, once more expressed as a proportion of the typical worth of the kpi.
sigma = float((y_pred - y_test).std()/(y_before_modification.imply()))
sigma
> 0.0780972738325956
The excellent news is that the uncertainty created by the deviation is diminished when the variety of information factors enhance. We desire a predictor with out bias, so it could possibly be mandatory to simply accept a rise within the deviation if allowed to restrict the bias.
It can be attention-grabbing to take a look at bias and variance by trying on the distribution of the forecasting errors. It may be helpful to see if our calculation of bias and deviation is legitimate, or whether it is affected by outliers and excessive values.
import seaborn as sns
import matplotlib.pyplot as pltf, ax = plt.subplots(figsize=(8, 6))
sns.histplot(pd.DataFrame((y_pred - y_test)/y_past.imply()), x = 'kpi', bins = 35, kde = True, stat = 'chance')
f.suptitle('Relative Error Distribution')
ax.set_xlabel('Relative Error')
plt.present()
3- Auto-correlation α: Normally, errors are auto-correlated. It implies that in case your prediction is above the true worth on a given day, it has extra likelihood of being above the following day. It’s a downside as a result of most classical statistical instruments require independence between observations. What occurred on a given day ought to have an effect on the following one. We use auto-correlation as a measure of dependence between sooner or later and the following.
df_test = pd.DataFrame(zip(y_pred, y_test), columns = ['Prevision','Real'], index = y_test.index)
df_test = df_test.assign(
ecart = df_test.Prevision - df_test.Actual)
alpha = df_test.ecart.corr(df_test.ecart.shift(1))
alpha
> 0.24554635095548982
A excessive auto-correlation is problematic however will be managed. A doable causes for it are unobserved covariates. If as an illustration, the shop you need to measure organized a particular occasion, it may enhance its gross sales for a number of days. It will result in an sudden sequence of days above the prevision.
df_test = pd.DataFrame(zip(y_pred, y_test), columns = ['Prevision','Reel'], index = y_test.index)f, ax = plt.subplots(figsize=(15, 6))
sns.lineplot(information = df_test, x = 'date', y= 'Reel', label = 'True Worth')
sns.lineplot(information = df_test, x = 'date', y= 'Prevision', label = 'Forecasted Worth')
ax.axvline(start_modification_date, ls = '--', colour = 'black', label = 'Begin of the modification')
ax.legend()
f.suptitle('KPI TX_1')
plt.present()
Within the determine above, you possibly can see an illustration of the auto-correlation phenomenon. In late April 2023, for a number of days, forecasted values are above the true worth. Errors are usually not unbiased of each other.
Affect Calculation
Now we will compute the influence of the modification. We evaluate the prediction after the modification with the precise worth. As all the time, it’s expressed as a proportion of the imply worth of the KPI.
y_pred_after_modification = mannequin.predict(X_after_modification)
uplift =float((y_after_modification - y_pred_after_modification).imply()/y_before_modification.imply())
uplift
> 0.04961773643584396
We get a relative enhance of 4.9% The “true” worth (the information used have been artificially modified) was 3.0%, so we’re not removed from it. And certainly, the true worth is usually above the prediction :
We will compute a confidence interval for this worth. If our predictor has no bias, the scale of its confidence interval will be expressed with:
The place σ is the usual deviation of the prediction, α its auto-correlation, and N the variety of days after the modification.
N = y_after_modification.form[0]
ec = sigma/(sqrt(N) *(1-alpha))print('68%% IC : [%.2f %% , %.2f %%]' % (100*(uplift - ec),100 * (uplift + ec) ))
print('95%% IC : [%.2f %% , %.2f %%]' % (100*(uplift -2 *ec),100 * (uplift +2*ec) ))
68% IC : [3.83 % , 6.09 %]
95% IC : [2.70 % , 7.22 %]
The vary of the 95% CI is round 4.5% for 84 days. It’s cheap for a lot of purposes, as a result of it’s doable to run an experiment or a proof of idea for 3 months.
Word: the arrogance interval may be very delicate to the deviation of the preliminary predictor. That’s the reason it’s a good suggestion to take a while to carry out mannequin choice (on the coaching set solely) earlier than choosing a great mannequin.
Mathematical formulation of the mannequin
To date we now have tried to keep away from maths, to permit for a better comprehension. On this part, we’ll current the mathematical mannequin beneath the mannequin.
