The RAG query engine plays a crucial role in retrieving context and generating responses. To ensure its performance and response quality, we conduct the following evaluations:
Context Relevance: Determines if the context extracted from the query is relevant to the response.
Factual Accuracy: Assesses if the LLM is hallcuinating or providing incorrect information.
Response Completeness: Checks if the response contains all the information requested by the query.
You can check out the complete list of evaluations UpTrain supports here
You can use the open-source evaluation service to evaluate your model. In this case, you will need to provie an OpenAI API key. You can get yours here.Parameters:
key_type=“openai”
api_key=“OPENAI_API_KEY”
project_name_prefix=“PROJECT_NAME_PREFIX”
Copy
callback_handler = UpTrainCallbackHandler( key_type="openai", api_key="sk-...", # Replace with your OpenAI API key project_name_prefix="llama",)Settings.callback_manager = CallbackManager([callback_handler])
4
Load and Parse Documents
Load documents from Paul Graham’s essay “What I Worked On”.
UpTrain callback handler will automatically capture the query, context and response once generated and will run the following three evaluations (Graded from 0 to 1) on the response:
Context Relevance: Determines if the context extracted from the query is relevant to the response.
Factual Accuracy: Assesses if the LLM is hallcuinating or providing incorrect information.
Response Completeness: Checks if the response contains all the information requested by the query.
Copy
index = VectorStoreIndex.from_documents( documents,)query_engine = index.as_query_engine()max_characters_per_line = 80queries = [ "What did Paul Graham do growing up?", "When and how did Paul Graham's mother die?", "What, in Paul Graham's opinion, is the most distinctive thing about YC?", "When and how did Paul Graham meet Jessica Livingston?", "What is Bel, and when and where was it written?",]for query in queries: response = query_engine.query(query)
Copy
Question: What did Paul Graham do growing up?Context Relevance Score: 0.0Factual Accuracy Score: 1.0Response Completeness Score: 0.0Question: When and how did Paul Graham's mother die?Context Relevance Score: 0.0Factual Accuracy Score: 1.0Response Completeness Score: 0.0Question: What, in Paul Graham's opinion, is the most distinctive thing about YC?Context Relevance Score: 1.0Factual Accuracy Score: 1.0Response Completeness Score: 1.0Question: When and how did Paul Graham meet Jessica Livingston?Context Relevance Score: 1.0Factual Accuracy Score: 1.0Response Completeness Score: 0.5Question: What is Bel, and when and where was it written?Context Relevance Score: 1.0Factual Accuracy Score: 1.0Response Completeness Score: 0.0