Performance OpenAI Evals using UpTrain
OpenAIGradeScore
: Calls openai evals with the given eval_name. Provide a column corresponding to the input and completion. Documentation
ModelGradeScore
: Define your model grading eval. Define your custom prompt, the weightage given to each option and the mapping to link dataset columns to the variables required in the prompt. Documentation
CheckSet
object. CheckSet
takes the source (i.e. test dataset file), the above-defined Check
and the directory where we wish to save the results.
add_dataset
method.
To upload your dataset file, you will need to specify the following parameters:
name
: The name of your datasetfpath
: The path to your dataset fileqna-notebook-data.jsonl
in your current directory. You can upload it using the code below.
add_checkset
method.
To add a checkset, you will need to specify the following parameters:
name
: The name of your checksetcheckset
: The checkset you wish to addsettings
: The settings you defined while creating the API clientadd_run
method.
To add a run, you will need to specify the following parameters:
dataset
: The name of the dataset you wish to addcheckset
: The name of the checkset you wish to addget_run
method.