- Same Number of Nodes Context Reranking: Checks if the order of re-ranked nodes is more relevant to the query than the original order.
- Different Number of Nodes: Context Conciseness: Examines whether the reduced number of nodes still provides all the required information.
How to do it?
1
Install UpTrain and LlamaIndex
2
Import required libraries
3
Setup UpTrain Open-Source Software (OSS)
You can use the open-source evaluation service to evaluate your model. In this case, you will need to provie an OpenAI API key. You can get yours here.Parameters:
key_type
=“openai”api_key
=“OPENAI_API_KEY”project_name_prefix
=“PROJECT_NAME_PREFIX”
4
Load and Parse Documents
Load documents from Paul Graham’s essay “What I Worked On”.Parse the document into nodes.
5
Re-ranking
Re-ranking is the process of reordering the nodes based on their relevance to the query. There are multiple classes of re-ranking algorithms offered by Llamaindex. We have used CohereRerank for this example.The re-ranker allows you to enter the number of top n nodes that will be returned after re-ranking. If this value remains the same as the original number of nodes, the re-ranker will only re-rank the nodes and not change the number of nodes. Otherwise, it will re-rank the nodes and return the top n nodes.We will perform different evaluations based on the number of nodes returned after re-ranking.
5.1
Re-ranking (With same number of nodes)
If the number of nodes returned after re-ranking is the same as the original number of nodes, the following evaluation will be performed:- Context Reranking: Check if the order of the re-ranked nodes is more relevant to the query than the original order.
5.2
Re-ranking (With different number of nodes)
If the number of nodes returned after re-ranking is the lesser as the original number of nodes, the following evaluation will be performed:- Context Conciseness: If the re-ranked nodes are able to provide all the information required by the query.