Perform Experiments with UpTrain
Experiments help you perform A/B testing, so you can compare and choose the options most suitable for you. This notebook shows you how to perform experiments with UpTrain. The experiment we will be demonstrating is to compare the responses given by a model when passed contexts of different lengths. This is done by using achunk_size
parameter that limits the number of tokens in the context
passed to the model.
We will only look at the code that is specific to performing experiments. We will not be looking at the entire process of extracting the context and generating the response. To learn more about that, please refer to the Data Driven Experimentation Demo.
Install UpTrain
Run the following commands in your terminal to install UpTrain:Import Libraries
Create an UpTrain API Client
Before we can start using UpTrain, we need to create an API client. You can do this by passing your API key to theAPIClient
constructor.
Create your data
You can define your data as a simple dictionary with the following keys:question
: The question you want to askcontext
: The context relevant to the questionresponse
: The response to the question
chunk_size
: The limit on the number of tokens in the context
Experiments
Now that we have our data, we can perform experiments on it using UpTrain. We use theevaluate_experiments
method to do this. This method takes the following arguments:
project_name
: The name of your projectdata
: The data you want to log and evaluateevals
: The evaluations you want to perform on your dataexp_columns
- A list of all the columns that act as identifiers to indicate which experiment the row belongs to. You can enter multiple column names here.
Get your results
