OpenAI Evals
Performance OpenAI Evals using UpTrain
Overview: In this example, we will see how to use UpTrain to run openai evals. You can run any of the standard evals defined in the evals registry or create your custom one with custom prompts. We will use a Q&A task as an example to highlight both of them.
Problem: The workflow of our hypothetical Q&A application goes like this,
- User enters a question.
- The query converts to an embedding, and relevant sections from the documentation are retrieved using nearest neighbour search.
- The original query and the retrieved sections are passed to a language model (LM), along with a custom prompt to generate a response.
Our goal is the evaluate the quality of the answer generated by the model using openai evals.
Solution: We illustrate how to use the Uptrain Evals framework to assess the chatbot’s performance. We will use a dataset built from logs generated by a chatbot made to answer questions from the Streamlit user documentation. For model grading, we will use GPT-3.5-turbo with grading type as ‘coqa-closedqa-correct’ as well as define our custom grading prompt for the same.
Install required packages
# !pip install uptrain
import os
import polars as pl
Let’s now load our dataset and see how that looks
url = "https://uptrain-assets.s3.ap-south-1.amazonaws.com/data/qna-streamlit-docs.jsonl"
dataset_path = os.path.join(os.getcwd(), "qna-notebook-data.jsonl")
if not os.path.exists(dataset_path):
import httpx
r = httpx.get(url)
with open(dataset_path, "wb") as f:
f.write(r.content)
dataset = pl.read_ndjson(dataset_path).select(
pl.col(["question", "document_text", "answer"])
)
print("Number of test cases: ", len(dataset))
print("Couple of samples: ", dataset[0:2])
So, we have a bunch of questions, retrieved documents and final answers (i.e. LLM responses) for them. Let’s evaluate the correctness of these answers using OpenAI evals.
Using UpTrain Framework to run evals
UpTrain provides integrations with openai evals to run any check defined in the registry.
We wrap these evals in an Operator class for ease of use.
-
OpenAIGradeScore
: Calls openai evals with the given eval_name. Provide a column corresponding to the input and completion. Documentation -
ModelGradeScore
: Define your model grading eval. Define your custom prompt, the weightage given to each option and the mapping to link dataset columns to the variables required in the prompt. Documentation
from uptrain.framework import Check
from uptrain.operators import ModelGradeScore, OpenAIGradeScore, Histogram
check = Check(
name="Eval scores",
operators=[
## Add your checks here
# First is the openai grade eval
OpenAIGradeScore(
col_in_input="document_text",
col_in_completion="answer",
eval_name="coqa-closedqa-correct",
col_out="openai_eval_score",
),
# Second is our custom grading eval
ModelGradeScore(
grading_prompt_template="""
You are an evidence-driven LLM that places high importance on supporting facts and references. You diligently verify claims and check for evidence within the document to ensure answers rely on reliable information and align with the documented evidence.",
You are comparing an answer pulled from the document for a given question
Here is the data:
[BEGIN DATA]
************
[Document]: {document}
************
[Question]: {user_query}
************
[Submitted answer]: {generated_answer}
************
[END DATA]
Compare the factual content of the submitted answer with the document as well as evaluate how well it answers the given question. Ignore any differences in style, grammar, or punctuation.
Answer the question by selecting one of the following options:
(A) The submitted answer is a subset of the document and answers the question correctly.
(B) The submitted answer is a subset of the document but is not an appropriate answer for the question.
(C) The submitted quote is a superset of the document but is consistent with the document as well as answers the question correctly.
(D) The submitted quote is a superset of the document but is consistent with the document although it does not answer the question correctly.
(E) The submitted quote is a superset of the document and is not consistent with the document.
""",
eval_type="cot_classify",
choice_strings=["A", "B", "C", "D", "E"],
choice_scores={"A": 1.0, "B": 0.2, "C": 0.5, "D": 0.1, "E": 0.0},
context_vars={
"document": "document_text",
"user_query": "question",
"generated_answer": "answer",
},
col_out="custom_eval_score",
),
],
plots = [Histogram(x="custom_eval_score", nbins=5)]
)
Now that we have defined our evals, we will wrap them in a CheckSet
object. CheckSet
takes the source (i.e. test dataset file), the above-defined Check
and the directory where we wish to save the results.
from uptrain.framework import CheckSet, Settings
from uptrain.operators import JsonReader
# Our checkset has source as our dataset and checks as the above-defined check
check_set = CheckSet(
source=JsonReader(fpath=dataset_path),
checks=[check],
)
Running the checks
# We first define a settings object
settings = Settings(
logs_folder = "uptrain_logs", # Results are saved in this folder
openai_api_key = "sk-**********************************"
)
# Run the checkset
check_set.setup(settings).run()
[Optional] Visualize results on streamlit
from uptrain.dashboard import StreamlitRunner
runner = StreamlitRunner(settings.logs_folder)
runner.start()
Using UpTrain’s Managed Service
Create an API Key
To get started, you will first need to get your API key from the Uptrain Website.
- Login with Google
- Click on “Create API Key”
- Copy the API key and save it somewhere safe
from uptrain import APIClient, Settings
UPTRAIN_API_KEY = "up-*******************************" # Insert your UpTrain API key here
settings = Settings(
uptrain_access_token=UPTRAIN_API_KEY,
)
client = APIClient(settings)
Step 2: Add dataset
Unlike the previous method where you had to create a dataset in Python, this method requires you to upload a file containing your dataset. The supported file formats are:
- .csv
- .json
- .jsonl
You can add the dataset file to the UpTrain platform using the add_dataset
method.
To upload your dataset file, you will need to specify the following parameters:
name
: The name of your datasetfpath
: The path to your dataset file
Let’s say you have a dataset file called qna-notebook-data.jsonl
in your current directory. You can upload it using the code below.
client.add_dataset(name="qna-dataset", fpath=dataset_path)
Step 3: Add Checkset
A checkset contains the operators you wish to evaluate your model on.
You can add a checkset using the add_checkset
method.
To add a checkset, you will need to specify the following parameters:
name
: The name of your checksetcheckset
: The checkset you wish to addsettings
: The settings you defined while creating the API client
client.add_checkset(
name="qna-checkset",
checkset=check_set,
settings=settings
)
Step 4: Add run
A run is a combination of a dataset and a checkset.
You can add a run using the add_run
method.
To add a run, you will need to specify the following parameters:
dataset
: The name of the dataset you wish to addcheckset
: The name of the checkset you wish to add
response = client.add_run(
dataset="qna-dataset",
checkset="qna-checkset"
)
Step 5: View the results
You can view the results of your evaluation by using the get_run
method.
client.get_run(response["run_id"])
You can also view the results on the UpTrain Dashboard by entering your API key as a password.
Was this page helpful?