Ride Time Estimation
Overview: In this example, we consider a regression task where we want to predict the trip time given the trip distance, number of passengers, booking date, vendor ID, etc.
Dataset: We use the New York City Taxi Trip Duration dataset from Kaggle, where the data contains features such as vendor_id, pickup_datetime, passenger_count, pickup_location, drop_location, etc., and the trip durations (in seconds). We want to train and ML model that takes the input features and predicts the trip duration.
Monitoring: In this notebook, we will see how we can use UpTrain package to monitor model accuracy, run data integrity checks, and add SHAP in UpTrain dashboard to explain model predictions.
# Divide data into train-test splits
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.1)
# Get ids for data to locate them with their ids in the UpTrain dashboard
test_ids = list(X_test['id'])
Let’s see how the input looks like
# Let's see how the data looks
X_train.head(3)
| vendor_id | passenger_count | store_and_fwd_flag | dist | pickup_year | pickup_month | pickup_day | pickup_dow | pickup_hour | pickup_min | pickup_sec | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 306046 | 1 | 1 | 0 | 0.006330 | 2016 | 5 | 6 | 4 | 12 | 22 | 21 |
| 253777 | 1 | 1 | 0 | 0.020341 | 2016 | 4 | 15 | 4 | 14 | 2 | 12 |
| 882330 | 1 | 1 | 0 | 0.116831 | 2016 | 3 | 18 | 4 | 9 | 26 | 43 |
Train our LightGBM Regression model
model = LGBMRegressor(n_estimators=500)
model.fit(X_train, y_train)
Check the training MAPE (Mean Absolute Percentage Error) of our model
preds = abs(model.predict(X_train))
mae_train = metrics.mean_absolute_error(y_train, preds)
print(f"Training Mean Percentage Error: {mae_train:.2f} seconds")
mape_train = metrics.mean_absolute_percentage_error(y_train, preds)
print(f"Training Mean Absolute Percentage Error: {mape_train*100:.2f} %")
Training Mean Percentage Error: 421.72 seconds
Training Mean Absolute Percentage Error: 74.38 %
Defining Monitors over test data using the UpTrain Framework
Next, we define monitors over our ML model in test/production with UpTrain.
Accuracy monitors
We define Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) accuracy monitors for our regression task
# To monitor MAE
mae_monitor = {
"type": uptrain.Monitor.ACCURACY,
"measurable_args": {
'type': uptrain.MeasurableType.MAE,
},
}
# To monitor (MAPE)
mape_monitor = {
"type": uptrain.Monitor.ACCURACY,
"measurable_args": {
'type': uptrain.MeasurableType.MAPE,
},
}
SHAP explanability
SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the ML model predictions and it available as a python package. Through UpTrain dashboard, we will use SHAP to explain our model’s preditions.
# To visualize the SHAP explanability in dashboard
shap_visual = {
"type": uptrain.Visual.SHAP,
"model": model,
# Limit the number of points for which SHAP values are calculated to reduce runtime.
"shap_num_points": 10000,
# Tell the framework the column name that corresponds to data-point id
"id_col": "id"
}
Data integrity monitor
We can also define data integrity checks over our test data. One obvious check is that the number of passengers should >= 1.
# Data integrity monitor to check if feature 'pasenger_count' is >=1
data_integrity_monitor = {
"type": uptrain.Monitor.DATA_INTEGRITY,
"measurable_args": {
'type': uptrain.MeasurableType.INPUT_FEATURE,
'feature_name': 'passenger_count'
},
"integrity_type": "greater_than",
"threshold": 1
}
Define the checks in config and pass it to the UpTrain framework
cfg = {
"checks": [mae_monitor, mape_monitor, data_integrity_monitor, shap_visual],
"logging_args": {"st_logging": True},
}
framework = uptrain.Framework(cfg_dict=cfg)
Launch the model in Production
batch_size = 2000
cols = list(X_test.columns)
for idx in range(int(len(X_test)/batch_size)):
indices = range(idx*batch_size, (idx+1)*batch_size)
this_elems = X_test.iloc[idx*batch_size: (idx+1)*batch_size, :]
this_preds = abs(model.predict(this_elems))
# Add ids to corresponding data points to preserve them in dashboard
this_elems = this_elems.assign(id=list(test_ids[idx*batch_size: (idx+1)*batch_size]))
# Log input and outputs to the UpTrain framework
ids = framework.log(inputs=this_elems, outputs=this_preds)
# Attach ground truth
ground_truth = list(y_test[idx*batch_size: (idx+1)*batch_size])
framework.log(identifiers=ids, gts=ground_truth)
SHAP Explanability at the UpTrain Dashboard
As we can notice from the dashboard, we get two plots for SHAP explainability. The plot on the left is the average weight of each feature. As expected, the feature “dist” (that represents the distance of the trip) has the biggest impact on the trip duration prediction. Moreover, pickup hour and taxi vendor id also somewhat affect the trip duration.
On the right, we can see how the model arrives at the prediction for any data-point (in this case ID id1787379). Due to the low distance of the trip, the feature “dist” contributed -256.06 to the overall trip duration.
Demo Video: SHAP explainability
We have added a small illustration on how it looks at the UpTrain dashboard below.
MAE and MAPE Accuracy
The following is how the MAE and MAPE accuracies are evolving with time on the UpTrain dashboard.
Data Integrity
We added a data integrity check for feature passenger_count >=1. We observe that data integrity is maintained through the test dataset (as data integrity is close to one)
