> ## Documentation Index
> Fetch the complete documentation index at: https://docs.uptrain.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG Query Engine Evaluations

The RAG query engine plays a crucial role in retrieving context and generating responses. To ensure its performance and response quality, we conduct the following evaluations:

* [Context Relevance](/predefined-evaluations/context-awareness/context-relevance): Determines if the context extracted from the query is relevant to the response.
* [Factual Accuracy](/predefined-evaluations/context-awareness/factual-accuracy): Assesses if the LLM is hallcuinating or providing incorrect information.
* [Response Completeness](/predefined-evaluations/response-quality/response-completeness): Checks if the response contains all the information requested by the query.

<Info>You can check out the complete list of evaluations UpTrain supports [here](/predefined-evaluations/overview) </Info>

### How to do it?

<Steps>
  <Step title="Install UpTrain and LlamaIndex">
    ```python theme={null}
    pip install -q html2text llama-index pandas tqdm uptrain cohere
    ```
  </Step>

  <Step title="Import required libraries">
    ```python theme={null}
    from llama_index import (
        ServiceContext,
        VectorStoreIndex,
    )
    from llama_index.node_parser import SentenceSplitter
    from llama_index.readers import SimpleWebPageReader
    from llama_index.callbacks import CallbackManager, UpTrainCallbackHandler
    from llama_index.postprocessor.cohere_rerank import CohereRerank
    from llama_index.service_context import set_global_service_context
    from llama_index.query_engine.sub_question_query_engine import (
        SubQuestionQueryEngine,
    )
    from llama_index.tools.query_engine import QueryEngineTool
    from llama_index.tools.types import ToolMetadata
    ```
  </Step>

  <Step title="Setup UpTrain Open-Source Software (OSS)">
    You can use the open-source evaluation service to evaluate your model. In this case, you will need to provie an OpenAI API key. You can get yours [here](https://platform.openai.com/account/api-keys).

    Parameters:

    * `key_type`="openai"
    * `api_key`="OPENAI\_API\_KEY"
    * `project_name_prefix`="PROJECT\_NAME\_PREFIX"

    ```python theme={null}
    callback_handler = UpTrainCallbackHandler(
        key_type="openai",
        api_key="sk-...",  # Replace with your OpenAI API key
        project_name_prefix="llama",
    )
    Settings.callback_manager = CallbackManager([callback_handler])
    ```
  </Step>

  <Step title="Load and Parse Documents">
    Load documents from Paul Graham's essay "What I Worked On".

    ```python theme={null}
    documents = SimpleWebPageReader().load_data(
      [
          "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt"
      ]
    )
    ```

    Parse the document into nodes.

    ```python theme={null}
    parser = SentenceSplitter()
    nodes = parser.get_nodes_from_documents(documents)
    ```
  </Step>

  <Step title="RAG Query Engine Evaluation">
    UpTrain callback handler will automatically capture the query, context and response once generated and will run the following three evaluations (Graded from 0 to 1) on the response:

    * [Context Relevance](/predefined-evaluations/context-awareness/context-relevance): Determines if the context extracted from the query is relevant to the response.
    * [Factual Accuracy](/predefined-evaluations/context-awareness/factual-accuracy): Assesses if the LLM is hallcuinating or providing incorrect information.
    * [Response Completeness](/predefined-evaluations/response-quality/response-completeness): Checks if the response contains all the information requested by the query.

    ```python theme={null}
    index = VectorStoreIndex.from_documents(
        documents,
    )
    query_engine = index.as_query_engine()

    max_characters_per_line = 80
    queries = [
        "What did Paul Graham do growing up?",
        "When and how did Paul Graham's mother die?",
        "What, in Paul Graham's opinion, is the most distinctive thing about YC?",
        "When and how did Paul Graham meet Jessica Livingston?",
        "What is Bel, and when and where was it written?",
    ]
    for query in queries:
        response = query_engine.query(query)
    ```

    ```bash theme={null}
    Question: What did Paul Graham do growing up?
    Context Relevance Score: 0.0
    Factual Accuracy Score: 1.0
    Response Completeness Score: 0.0


    Question: When and how did Paul Graham's mother die?
    Context Relevance Score: 0.0
    Factual Accuracy Score: 1.0
    Response Completeness Score: 0.0


    Question: What, in Paul Graham's opinion, is the most distinctive thing about YC?
    Context Relevance Score: 1.0
    Factual Accuracy Score: 1.0
    Response Completeness Score: 1.0


    Question: When and how did Paul Graham meet Jessica Livingston?
    Context Relevance Score: 1.0
    Factual Accuracy Score: 1.0
    Response Completeness Score: 0.5


    Question: What is Bel, and when and where was it written?
    Context Relevance Score: 1.0
    Factual Accuracy Score: 1.0
    Response Completeness Score: 0.0
    ```
  </Step>
</Steps>

<CardGroup cols={2}>
  <Card title="Tutorial" href="https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/docs/examples/callbacks/UpTrainCallback.ipynb" icon="infinity" color="#808080">
    Open this tutorial in Colab
  </Card>

  <Card title="Have Questions?" href="https://join.slack.com/t/uptraincommunity/shared_invite/zt-1yih3aojn-CEoR_gAh6PDSknhFmuaJeg" icon="slack" color="#808080">
    Join our community for any questions or requests
  </Card>
</CardGroup>