Overview: In this example, we consider a regression task where we want to predict the trip time given the trip distance, number of passengers, booking date, vendor ID, etc.

Dataset: We use the New York City Taxi Trip Duration dataset from Kaggle, where the data contains features such as vendor_id, pickup_datetime, passenger_count, pickup_location, drop_location, etc., and the trip durations (in seconds). We want to train and ML model that takes the input features and predicts the trip duration.

Monitoring: In this notebook, we will see how we can use UpTrain package to monitor model accuracy, run data integrity checks, and add SHAP in UpTrain dashboard to explain model predictions.

# Divide data into train-test splits
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.1)

# Get ids for data to locate them with their ids in the UpTrain dashboard
test_ids = list(X_test['id'])

​
Let’s see how the input looks like

# Let's see how the data looks
X_train.head(3)
vendor_idpassenger_countstore_and_fwd_flagdistpickup_yearpickup_monthpickup_daypickup_dowpickup_hourpickup_minpickup_sec
3060461100.0063302016564122221
2537771100.0203412016415414212
8823301100.1168312016318492643

​
Train our LightGBM Regression model

model = LGBMRegressor(n_estimators=500) 
model.fit(X_train, y_train)

​
Check the training MAPE (Mean Absolute Percentage Error) of our model

preds = abs(model.predict(X_train))
mae_train = metrics.mean_absolute_error(y_train, preds)
print(f"Training Mean Percentage Error: {mae_train:.2f} seconds")
mape_train = metrics.mean_absolute_percentage_error(y_train, preds)
print(f"Training Mean Absolute Percentage Error: {mape_train*100:.2f} %")

Training Mean Percentage Error: 421.72 seconds
Training Mean Absolute Percentage Error: 74.38 %

​
Defining Monitors over test data using the UpTrain Framework

Next, we define monitors over our ML model in test/production with UpTrain.

​
Accuracy monitors

We define Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) accuracy monitors for our regression task

# To monitor MAE
mae_monitor = {       
    "type": uptrain.Monitor.ACCURACY,
    "measurable_args": {
        'type': uptrain.MeasurableType.MAE,
    },
}

# To monitor  (MAPE)
mape_monitor = {
    "type": uptrain.Monitor.ACCURACY,
    "measurable_args": {
        'type': uptrain.MeasurableType.MAPE,
    },
}

​
SHAP explanability

SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the ML model predictions and it available as a python package. Through UpTrain dashboard, we will use SHAP to explain our model’s preditions.

# To visualize the SHAP explanability in dashboard
shap_visual = {
    "type": uptrain.Visual.SHAP,
    "model": model,
    # Limit the number of points for which SHAP values are calculated to reduce runtime.
    "shap_num_points": 10000,
    # Tell the framework the column name that corresponds to data-point id
    "id_col": "id"
}

​
Data integrity monitor

We can also define data integrity checks over our test data. One obvious check is that the number of passengers should >= 1.

# Data integrity monitor to check if feature 'pasenger_count' is >=1
data_integrity_monitor =  {
    "type": uptrain.Monitor.DATA_INTEGRITY,
    "measurable_args": {
        'type': uptrain.MeasurableType.INPUT_FEATURE,
        'feature_name': 'passenger_count'
    },
    "integrity_type": "greater_than",
    "threshold": 1
}

​
Define the checks in config and pass it to the UpTrain framework

cfg = {
    "checks": [mae_monitor, mape_monitor, data_integrity_monitor, shap_visual],
    "logging_args": {"st_logging": True},
}
framework = uptrain.Framework(cfg_dict=cfg)

​
Launch the model in Production

batch_size = 2000
cols = list(X_test.columns)
for idx in range(int(len(X_test)/batch_size)):
    
    indices = range(idx*batch_size, (idx+1)*batch_size)
    this_elems = X_test.iloc[idx*batch_size: (idx+1)*batch_size, :]
    this_preds = abs(model.predict(this_elems))
    
    # Add ids to corresponding data points to preserve them in dashboard
    this_elems = this_elems.assign(id=list(test_ids[idx*batch_size: (idx+1)*batch_size]))
    
    # Log input and outputs to the UpTrain framework
    ids = framework.log(inputs=this_elems, outputs=this_preds)
    
    # Attach ground truth
    ground_truth = list(y_test[idx*batch_size: (idx+1)*batch_size])
    framework.log(identifiers=ids, gts=ground_truth)

​
SHAP Explanability at the UpTrain Dashboard

As we can notice from the dashboard, we get two plots for SHAP explainability. The plot on the left is the average weight of each feature. As expected, the feature “dist” (that represents the distance of the trip) has the biggest impact on the trip duration prediction. Moreover, pickup hour and taxi vendor id also somewhat affect the trip duration.

On the right, we can see how the model arrives at the prediction for any data-point (in this case ID id1787379). Due to the low distance of the trip, the feature “dist” contributed -256.06 to the overall trip duration.

​
Demo Video: SHAP explainability

We have added a small illustration on how it looks at the UpTrain dashboard below.

gif

​
MAE and MAPE Accuracy

The following is how the MAE and MAPE accuracies are evolving with time on the UpTrain dashboard.

​
Data Integrity

We added a data integrity check for feature passenger_count >=1. We observe that data integrity is maintained through the test dataset (as data integrity is close to one)