Response Consistency
Assesses how consistent the response is with the question asked as well as with the context provided.
Response Consistency is the measure of how well the generated response aligns with both the question asked and the context provided.
In evaluating response consistency, it is important to assess whether the information provided in the response directly addresses the query posed by the user and is coherent with any additional context given.
Columns required:
question
: The question asked by the usercontext
: Information retrieved to answer the questionresponse
: The response given by the model
How to use it?
Sample Response:
The response states the use of painkillers for the treatment of pneumonia.
This contradicts the context which states that pneumonia is typically treated with antibiotics, rest, and supportive care
Resulting in a low response consistency score.
How it works?
We evaluate response consistency through the following steps:
- Generating an argument as to why the given response is appropriate for the question asked.
- Rating the generated argument on a score of 0 to 1, as per how logical the argument seems to be.
Was this page helpful?