Custom Python Evals

UpTrain offers a multitude of pre-built evaluations that use custom prompt templates to evaluate your model’s performance. These checks include multiple use cases covering (respose quality, tonality, context awareness, code related evaluations and a lot more…) You can also create your own custom prompt templates for evaluations, you can check out the Custom Prompt Evals Tutorial. All of these evaluations involve making LLM calls. This is not always necessary. Some evaluations can be done with simple Python code, for example:

Check for the total number of distinct words
Check for the average number of unique words
Check for the presence of “numbers”

You can of course make an LLM call for that, but why to spend money when you can directly code the checks for that. In this tutorial, we will show you how to create such custom evaluations using Python code.

Install UpTrain

Install UpTrain by running the following command:

pip install uptrain

Define the custom evaluation

We will use UpTrain to check for these custom evaluations over the following cases:

Check for the average number of unique words
Check for average length of words

First, let’s import the required dependencies

from uptrain import EvalLLM, Settings
from uptrain.operators.base import ColumnOp, register_custom_op, TYPE_TABLE_OUTPUT
import polars as pl

Example 1: Check for the average number of unique words

Note: Please ensure to add the prefix “score_” to the value in col_out_score if you wish to log these results on uptrain’s locally hosted dashboard

@register_custom_op
class DiverseVocabularyEval(ColumnOp):
    
    col_in_text: str = "response"
    col_out_score: str = "score_diverse_vocabulary"

    def setup(self, settings: Settings):
        return self

    def run(self, data: pl.DataFrame) -> TYPE_TABLE_OUTPUT:
        scores = data.get_column(self.col_in_text).map_elements(lambda s : round(len(set(s.split())) / len(s.split()), 2))
        return {"output": data.with_columns([scores.alias(self.col_out_score)])}

Example 2: Check for average length of words

@register_custom_op
class AverageWordLengthEval(ColumnOp):
    col_in_text: str = "response"
    col_out_score: str = "score_average_word_length"

    def setup(self, settings: Settings):
        return self

    def run(self, data: pl.DataFrame) -> TYPE_TABLE_OUTPUT:
        scores = data.get_column(self.col_in_text).map_elements(lambda s : round(sum(map(lambda word: len(word), s.split())) / len(s.split()), 2))
        return {"output": data.with_columns([scores.alias(self.col_out_score)])}

Run the evaluations

Let’s define a dataset

data = [
    {
        "question": "What are the primary components of a cell?",
        "response": "A cell comprises a cell membrane, cytoplasm, and nucleus. The cell membrane regulates substance passage, the cytoplasm contains organelles, and the nucleus houses genetic material."
    },
    {
        "question": "How does photosynthesis work?",
        "response": "Photosynthesis converts light energy into chemical energy in plants, algae, and some bacteria. Chlorophyll absorbs sunlight, synthesizing glucose from carbon dioxide and water, with oxygen released as a byproduct."
    },
    {
        "question": "What are the key features of the Python programming language?",
        "response": "Python is a high-level, interpreted language known for readability. It supports object-oriented, imperative, and functional programming with a large standard library, dynamic typing, and automatic memory management."
    }
]

All done! Now let’s run these evaluations

eval_llm = EvalLLM(Settings())

results = eval_llm.evaluate(
    project_name = 'UpTrain',
    data=data,
    checks=[
        DiverseVocabularyEval(col_in_text="response"),
        AverageWordLengthEval(col_in_text="response"), 
    ],
)

Note: By default UpTrain runs locally on your system. You can also ensure this by passing Settings(evaluate_locally=True) to EvalLLM

Visualize these results

Now that you have generated these evaluations, you can also visualize the results on UpTrain’s Dashboard.This Dashboard is a part of UpTrain’s open-source offering and runs locally on your device.Check out this documentation to get started with UpTrain Dashboard

Bonus

We have already defined some prebuilt evaluations that you can use without the hassle of writing the code for them

Operator	Description	Input	Output
`DocsLinkVersion()`	Extracts version numbers from URLs in response	`response`	`docs_link_version`
`WordCount()`	Calculate the number of words in response	`response`	`word_count`
`TextLength()`	Calculate the length of text in response	`response`	`text_length`
`KeywordDetector()`	Detects the presence of a keyword in response	`response`, `keyword`	`keyword_detector`

from uptrain.operators.language.text import WordCount, KeywordDetector

eval_llm = EvalLLM(Settings())

results = eval_llm.evaluate(
    project_name = 'UpTrain',
    data=data,
    checks=[
        WordCount(col_in_text = "response"),
        KeywordDetector(col_in_text = "response", keyword = 'Python'), 
    ],
)

Note: If you face any difficulties, need some help with using UpTrain or want to brainstorm on custom evaluations for your use-case, speak to the maintainers of UpTrain here.

Tutorial

Open this tutorial in GitHub

Have Questions?

Join our community for any questions or requests

Getting Started

Pre-configured Evaluations

Supported LLMs

Integrations

Tutorials

FAQ

Custom Python Evals

Bonus

Tutorial

Have Questions?

​Bonus

Tutorial

Have Questions?

Bonus