Re-ranking involves reordering nodes based on relevance to the query and choosing top n nodes. Different evaluations are performed based on the number of nodes returned after re-ranking.

  1. Same Number of Nodes

    Context Reranking: Checks if the order of re-ranked nodes is more relevant to the query than the original order.

  2. Different Number of Nodes:

    Context Conciseness: Examines whether the reduced number of nodes still provides all the required information.

These evaluations collectively ensure the robustness and effectiveness of the RAG query engine, SubQuestionQueryGeneration operator, and the re-ranking process in the LlamaIndex pipeline.

​
How to do it?

1

Install UpTrain and LlamaIndex

pip install -q html2text llama-index pandas tqdm uptrain cohere
2

Import required libraries

from llama_index import (
    ServiceContext,
    VectorStoreIndex,
)
from llama_index.node_parser import SentenceSplitter
from llama_index.readers import SimpleWebPageReader
from llama_index.callbacks import CallbackManager, UpTrainCallbackHandler
from llama_index.postprocessor.cohere_rerank import CohereRerank
from llama_index.service_context import set_global_service_context
from llama_index.query_engine.sub_question_query_engine import (
    SubQuestionQueryEngine,
)
from llama_index.tools.query_engine import QueryEngineTool
from llama_index.tools.types import ToolMetadata
3

Setup UpTrain Open-Source Software (OSS)

You can use the open-source evaluation service to evaluate your model. In this case, you will need to provie an OpenAI API key. You can get yours here.

Parameters:

  • key_type=“openai”
  • api_key=“OPENAI_API_KEY”
  • project_name_prefix=“PROJECT_NAME_PREFIX”
callback_handler = UpTrainCallbackHandler(
    key_type="openai",
    api_key="sk-...",  # Replace with your OpenAI API key
    project_name_prefix="llama",
)
Settings.callback_manager = CallbackManager([callback_handler])
4

Load and Parse Documents

Load documents from Paul Graham’s essay “What I Worked On”.

documents = SimpleWebPageReader().load_data(
  [
      "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt"
  ]
)

Parse the document into nodes.

parser = SentenceSplitter()
nodes = parser.get_nodes_from_documents(documents)
5

Re-ranking

Re-ranking is the process of reordering the nodes based on their relevance to the query. There are multiple classes of re-ranking algorithms offered by Llamaindex. We have used CohereRerank for this example.

The re-ranker allows you to enter the number of top n nodes that will be returned after re-ranking. If this value remains the same as the original number of nodes, the re-ranker will only re-rank the nodes and not change the number of nodes. Otherwise, it will re-rank the nodes and return the top n nodes.

We will perform different evaluations based on the number of nodes returned after re-ranking.

5.1

Re-ranking (With same number of nodes)

If the number of nodes returned after re-ranking is the same as the original number of nodes, the following evaluation will be performed:

  • Context Reranking: Check if the order of the re-ranked nodes is more relevant to the query than the original order.
api_key = "**********************************"  # Insert cohere API key here
cohere_rerank = CohereRerank(
    api_key=api_key, top_n=5
)  # In this example, the number of nodes before re-ranking is 5 and after re-ranking is also 5.

index = VectorStoreIndex.from_documents(
    documents=documents, service_context=service_context
)

query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[cohere_rerank],
    service_context=service_context,
)

response = query_engine.query(
    "What did Sam Altman do in this essay?",
)

Question: What did Sam Altman do in this essay?
Context Reranking Score: 0.0
5.2

Re-ranking (With different number of nodes)

If the number of nodes returned after re-ranking is the lesser as the original number of nodes, the following evaluation will be performed:

  • Context Conciseness: If the re-ranked nodes are able to provide all the information required by the query.
api_key = "**********************************"  # insert cohere API key here
cohere_rerank = CohereRerank(
    api_key=api_key, top_n=2
)  # In this example, the number of nodes before re-ranking is 5 and after re-ranking is 2.

index = VectorStoreIndex.from_documents(
    documents=documents, service_context=service_context
)
query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[cohere_rerank],
    service_context=service_context,
)

# Use your advanced RAG
response = query_engine.query(
    "What did Sam Altman do in this essay?",
)

Question: What did Sam Altman do in this essay?
Context Conciseness Score: 1.0

Tutorial

Open this tutorial in Colab

Have Questions?

Join our community for any questions or requests

Was this page helpful?

Sub-Question Query Generation EvaluationLangfuse