How to Build a RAG System Using Company Data

Retrieval-Augmented Generation retrieves relevant context from enterprise data sources before a generative model produces an answer.

RAG involves more than loading documents into a vector database. Design data preparation, access control, retrieval quality, source citations and evaluation together.

Source Inventory

First inventory sources such as policies, procedures, technical documents, product information, support content and contracts.

Non-current or conflicting documents directly affect the quality of the retrieval.

Document parsing

When extracting text from PDF, Word, HTML or other formats, preserve headings, tables, lists and section structure as far as possible.

Chunking

Chunks that are too small can lose context; oversized chunks can reduce retrieval precision. Consider sections, headings and semantic structure.

Embeddings and indexing

Convert chunks into embedding vectors and add them to a search index. Metadata can retain source, section, date, department and access level.

Hybrid retrieval

Combine semantic search with lexical search, such as keyword matching or BM25, rather than relying only on semantic vector search.

Reranking

Candidates in the first retrieval result can be reordered with a second ranking layer and only the most relevant context can be sent to the model.

Citation and Provenance

The user should be able to see which document the answer is based on. The source name, section, and, if necessary, direct document link can be shown.

RAG Evaluation

Retrieval recall
Context Precision
Answer correctness
Groundedness in sources
Unsupported claim
Citation accuracy
Latency

How to evaluate an AI system in production

Before production, prepare separate evaluation sets for the source inventory, document parsing and chunking. Do not decide from a few successful demo questions alone: use real user tasks to create successful, failed and boundary examples. Define expected behaviour, acceptable error limits and whether human intervention is needed for each example, so model, retrieval and workflow changes can be compared on the same test set.

Production monitoring must combine conventional application metrics and AI quality metrics. Track latency, token or model cost, tool errors and API timeouts technically; track incorrect sources, unsupported claims, user corrections, human overrides and task success for product quality. When model or prompt versions change, use regression evaluations to check whether the previous quality level is retained.

Data security and governance

Treat data sent to the model, retrieval sources, conversation logs and tool outputs as separate data flows in enterprise AI. Do not unnecessarily include personal or commercially sensitive data in prompts; restrict access by user role. Assess model and provider selection, processing locations and log-retention periods alongside the organisation’s data-protection and security processes. The AI layer must not create a parallel data-access channel that bypasses existing authorisation.

AI production checklist

Are the use case and success metric defined?
Is an evaluation dataset available?
Are prompt, model and retrieval versions tracked?
Are user permissions enforced in the AI layer?
Do tool calls use schema validation?
Is human approval required for risky actions?
Are latency and model costs measured?
Are wrong answers and user corrections classified?

Failure taxonomy and controlled rollout

Grouping every failure under “AI gave the wrong answer” makes root-cause analysis difficult. Classify retrieval problems, missing context, wrong tool selection, authorisation blocks, hallucinations, timeouts, schema-validation errors and ambiguous requests separately. Identifying which area is responsible — the source inventory, document parsing and chunking — helps improve the right component instead of simply replacing the model. This also enriches the evaluation dataset with real production failures over time.

A new model, prompt, retrieval setting or agent workflow need not reach every user at once. Start with a limited group or low-risk tasks and compare quality, latency, cost and escalation to humans with the previous version. Stop rollout if critical metrics worsen and allow a return to the previous version. This turns uncontrolled prompt edits into a versioned, measurable release process.

Cost and usage guardrails

Production AI requires resource management as well as quality management. Track tokens, model calls, tool use, retrieval volume and processing time per user or task. Unexpected loops, unnecessarily long context and repeated tool calls can increase cost and latency. Guardrails such as maximum steps, timeouts, context limits, model fallback policies and per-user usage limits provide operational control. Interpret cost alongside task success; optimise flows that are cheap but unsuccessful or expensive and unnecessarily complex.

Log failure types in separate categories
Record prompt, model and workflow versions
Test releases with a controlled user group
Define rollback criteria in advance
Limit tool steps and processing time
Measure token and model costs per task
Feed production failures back into the evaluation set

B10 Digital Agency’s AI & automation approach

B10 Digital Agency treats enterprise AI not merely as model integration, but as production systems combining data access, RAG, tools, agent workflows, security, human-in-the-loop and measurable evaluation.

Frequently Asked Questions

Is RAG the same as fine-tuning?

No. RAG retrieves external information when answering; fine-tuning changes model behaviour or knowledge through training.

Should each document be added to the vector database?

No. Select documents by freshness, access permissions and usefulness.

Does RAG solve the problem of hallucination entirely?

No. It can reduce the risk, but evaluation and response boundaries remain necessary.

Why is metadata filter important?

It filters suitable sources during retrieval by date, department or user permissions.

Make enterprise AI measurable

Explore B10 Digital Agency’s AI & Automation approach to integrate AI assistants, RAG, agents and workflow automation into existing systems securely and measurably.

Explore AI & Automation Solutions →