​
Get your UpTrain API key

Sign up for an UpTrain account and get your API key. You will be provided with 100 free credits to get started. You can also contact us for a demo or to get more credits.

If you do not wish to create an UpTrain account, you can use your OpenAI API key instead. These evaluations are also run on UpTrain’s infrastructure. Check out the Open Source API Client tutorial to learn how to do this.

​
Install UpTrain

Run the following commands in your terminal to install UpTrain:

pip install uptrain

​
Create an UpTrain API Client

Before we can start using UpTrain, we need to create an API client. You can do this by passing your API key to the APIClient constructor.

UPTRAIN_API_KEY = "up-***************"
client = APIClient(uptrain_api_key=UPTRAIN_API_KEY)

​
Create your data

You can define your data as simple dictionary with the following keys:

  • question: The question you want to ask
  • context: The context relevant to the question
  • response: The response to the question
data = [{
    "question": [
        "Which is the most popular global sport?",
        "Who created the Python language?",
    ]
    "context": [
        "The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people.",
        "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
    ]
    "response":[
        "Football is the most popular sport with around 4 billion followers worldwide",
        "Python language was created by Guido van Rossum.",
    ]
}]

​
Log and evaluate your data

Now that we have our data, we can log it and evaluate it using UpTrain. We use the log_and_evaluate method to do this. This method takes the following arguments:

  • project_name: The name of your project
  • data: The data you want to log and evaluate
  • evals: The evaluations you want to perform on your data

You can find the list of all available evaluations here.

from uptrain.framework import APIClient, Evals, CritiqueTone
import json

results = client.log_and_evaluate(
    project_name="Sample-Project",
    data=data,
    evals=[Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_RELEVANCE, CritiqueTone(persona="teacher")]
)

​
Get your results

print(json.dumps(results, indent=3))

​
Experiments

You can also run experiments on your data using the evaluate_experiments method. Experiments help you perform A/B testing with prompts, so you can compare and choose the options most suitable for you.

The evaluate_experiments method takes all the same arguments as the log_and_evaluate with the addition of the following:

  • exp_columns - A list of all the columns that act as identifiers to indicate which experiment the row belongs to. You can enter multiple column names here.

Let’s look at an example with prompt_variation as a column that indicates the experiment.

data = pd.DataFrame([{
    'question': 'Which is the most popular global sport?',
    'context': "The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.",
    'response': "1. The most popular global sport is determined by factors such as TV viewership, social media presence, number of participants, and economic impact. 2. Football is considered the most popular sport in the world, with events like the FIFA World Cup and star players like Ronaldo and Messi attracting over 4 billion followers. 3. Cricket is particularly popular in countries like India, Pakistan, Australia, and England, with events like the ICC Cricket World Cup and the Indian Premier League (IPL) having substantial viewership. 4. Basketball has gained global popularity, especially in the USA, Canada, China, and the Philippines, largely thanks to the NBA. 5. Tennis also has a significant global audience, with major tournaments like Wimbledon, the US Open, French Open, and Australian Open, and notable players like Roger Federer, Serena Williams, and Rafael Nadal. 6. Field Hockey enjoys popularity in countries like India, Netherlands, and Australia and has followers in many other parts of the world. So, the most popular global sport is football, followed by cricket, basketball, tennis, and field hockey, depending on the region and various factors.",
    'prompt_variation': 'chain-of-thought'
},
{
    'question': 'Which is the most popular global sport?',
    'context': "The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.",
    'response': "- Which is the most popular global sport?  - Popularity of sports can be measured in different ways:    - TV viewership    - Social media presence    - Number of participants    - Economic impact  - Football:    - FIFA World Cup    - Ronaldo and Messi draw over 4 billion followers  - Cricket:    - Popular in India, Pakistan, Australia, and England    - ICC Cricket World Cup    - Indian Premier League (IPL)  - Basketball:    - NBA    - Popularity in the USA, Canada, China, Philippines  - Tennis:    - Major tournaments: Wimbledon, US Open, French Open, Australian Open    - Players: Roger Federer, Serena Williams, Rafael Nadal  - Field Hockey:    - Popular in India, Netherlands, Australia    - Followers in many parts of the world  In summary, football is the most popular global sport, followed by cricket, basketball, tennis, and field hockey, with variations in popularity depending on region and measurement criteria.",
    'prompt_variation': 'tree-of-thought'
}])

results = client.evaluate_experiments(
    project_name="Sample-Experiment",
    data=data,
    evals=[Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_RELEVANCE, CritiqueTone()],
    exp_columns=['prompt_variation']
)

print(json.dumps(results, indent=3))

If you wish to create your own custom evaluations with your own custom graphs, you can follow our UpTrain API Client tutorial to learn how to do so.

IntroductionCore Idea