A language model can generate a plausible response regardless of your company policies, products or process. Retrieval augmented generation, or RAG, is a technique that closes this gap by retrieving the relevant information before responding.
How a RAG pipeline is built
The majority of RAG pipelines have 2 primary stages: ingestion and retrieval. Ingestion processes documents, splitting them into useful passages, enriching with any relevant metadata and transforming into searchable embeddings. The retrieval step uses the question to find the relevant passages and provides them as context for the model to use when answering the question.
The end result is a response based on the information specific to your company.
The quality of the retrieval determines the quality of the answer
The quality of the answer depends on the quality of information retrieved. The answer will only be as good as the data passed to it. Documents need to be split into passages based on semantics, not just characters. Tables, headers, permissions, source links and other information should be retained where possible.
Documents should also have metadata added indicating the type, department, effective date, customer, permissions, or other relevant information. Metadata allows for additional filtering during retrieval, increasing relevance and improving privacy and compliance.
Measuring effectiveness
The effectiveness of the retrieval process should be measured based on what is most important to your use-case. Did the retrieval return the right documents? Is the answer actually using the correct information? Is it possible for the model to respond honestly when it doesn’t know the answer?
Visibility into the process is vital and can be achieved through citations in the answer and a feedback button. RAG pipelines aren’t magical but with carefully prepared data, smart filtering, and visibility into the process, they turn a generic language model into something truly useful.