Helps analyse failure causes in a RAG pipeline
Failure Case | Explanation | Example |
---|---|---|
Poor Context Utilization | The citations from the context are irrelevant to a user’s query | The LLM cites information on offers rather than refunds from a context containing information on both refunds and offers, for a question “Can I get a refund?” |
Poor Retrieval | The context given to an LLM does not have information relevant to the question | The user asks “Do you deliver to Bangalore?” but the context does not have any information to deliveries in Bangalore |
Hallucinations | The generated response is not supported by information present in the context | The LLM generates a response “We deliver to Bangalore” when the information present in the context is: “We are going to start delivers in Bangalore soon” |
Poor Citation | The response generated can not be verified with the citation | The LLM cites “We deliver to Delhi” from the context for a response saying “We deliver to Bangalore” |
Incomplete Question | The user’s question itself does not make sense | The user asks something like: “When delivery?”, “What location?” |
Install UpTrain
Let's define a sample dataset to run evaluations
Parameter | Explanation |
---|---|
question | This is the query asked by your user. |
context | This is the context that you pass to an LLM (retrieved context) |
response | The response generated by the LLM |
cited_context | The relevant portion of the retrieved context that the LLM cites to generate response. |
Perform failure analysis using UpTrain
EvalLLM
to perform RCA on your RAG pipeline.
You need an OpenAI key to generate evaluations using UpTrain.Let's look at the results
Parameter | Explanation |
---|---|
error_mode | The specific failure reason identified in your data |
error_resolution_suggestion | Actionable insights to improve your RAG pipeline |