AI Engineering: Complete Guide to LLMs, RAG and Agents

Start by specifying what the assistant may answer, which sources count as authoritative and what it must never do. In this example, employees ask questions about approved internal procedures. The assistant should retrieve relevant passages, produce a grounded answer and cite the source documents. If evidence is missing or contradictory, it should explain the limitation rather than invent a policy. Ticket creation is a separate capability requiring explicit confirmation. Define success through representative tasks and measurable constraints, such as answer correctness, citation support, response time and cost per successful request. These requirements should drive model selection, not the other way round.

The LLM system architecture has two connected paths. An ingestion pipeline reads documents, extracts text, attaches metadata, creates chunks, computes embeddings and updates a searchable index. A request pipeline authenticates the user, interprets the question, retrieves authorised evidence, constructs a prompt and calls the model. It then checks the output and returns an answer with source references. A controlled tool layer handles any approved actions. Around both paths sit evaluation, tracing, configuration management and operational monitoring. Keep these boundaries explicit so that replacing an embedding model, changing a prompt or switching inference providers does not require rebuilding the entire application.

Retrieval quality begins with document quality. Extract useful text while preserving headings, tables, document identifiers and meaningful relationships between sections. Remove navigation noise and duplicated material, but retain details that change an instruction’s meaning. Chunk documents along semantic boundaries where possible, then test chunk sizes against actual questions. Small chunks can omit necessary context; large chunks can bury the answer and increase prompt costs. Attach metadata such as ownership, source location, revision date and access policy before indexing. For the support assistant, a procedure should remain traceable to its current approved version, rather than appearing as an anonymous block of text.

Dense retrieval represents queries and document chunks as numerical vectors produced by an embedding model. A similarity search finds passages whose representations are close to the query, enabling matches even when wording differs. Document and query embeddings must use compatible models and any required input formats. Dense search can still miss exact product codes, unusual names or negation. Hybrid retrieval combines it with lexical search, often followed by a reranker that scores candidate passages against the question. Retrieve enough candidates to preserve recall, then select a smaller evidence set. Validate this pipeline using labelled queries rather than assuming semantic similarity guarantees relevance.

A vector database stores embeddings and supports similarity search, usually alongside metadata filtering and index management. Some teams use a dedicated service; others add vector capabilities to an existing database or search platform. Choose according to filtering requirements, update frequency, operational experience and observed workload. Approximate nearest-neighbour indexes trade some retrieval accuracy for faster search, so configuration deserves testing with realistic data. Distance functions, vector dimensions and embedding normalisation must match the embedding model’s requirements. The database is not the source of truth: keep original documents and provenance separately, with stable identifiers connecting each retrieved chunk to its authoritative source.

Index maintenance is part of application correctness. When a policy changes, the ingestion pipeline should replace or retire affected chunks without leaving stale versions silently discoverable. Use repeatable processing, explicit document versions and deletion propagation. A change of embedding model generally requires re-embedding the corpus and rebuilding or migrating the relevant index; mixing incompatible embeddings undermines search. Access filtering must also remain correct during updates. Enforce permissions before unauthorised content can reach the model, and verify that the chosen search configuration handles selective filters reliably. Test incremental updates, full rebuilds and recovery procedures before the assistant becomes an operational dependency.

Retrieval augmented generation, or RAG, supplies retrieved evidence to a model at request time. For the support assistant, the prompt should distinguish system instructions, the user’s question and quoted source material. Ask the model to ground policy claims in that evidence and associate citations with specific claims. Keep source identifiers attached throughout prompt construction and rendering. RAG makes changing knowledge easier to maintain because documents can be updated without retraining the generator. However, it does not guarantee factual answers: retrieval may fail, sources may conflict, and the model may misinterpret evidence. Citation presence alone is not proof that a claim is supported.

The practical RAG vs fine-tuning decision is about the problem being solved. Use retrieval when answers depend on current, private or attributable knowledge. Consider fine-tuning when a model repeatedly struggles with a stable behaviour, specialised output pattern or task despite good prompting and sufficient context. Fine-tuning adjusts model weights using training examples; it is not a dependable replacement for an updatable factual store. The approaches can be combined, with a tuned model answering from retrieved evidence. Before training, establish a baseline, inspect failures and protect a separate evaluation set. Otherwise, apparent improvement may reflect memorisation or changes elsewhere in the pipeline.

AI agents use model outputs to choose actions, call tools or adapt a sequence of steps towards a goal. This flexibility is useful when a request cannot be handled by one retrieval-and-answer pass, but it adds uncertainty and operational complexity. For the support assistant, a deterministic workflow may be enough: retrieve the procedure, collect required ticket fields, show a preview and request approval. Use agentic planning only where its flexibility produces a measurable benefit. A bounded workflow is usually easier to test than an open-ended loop that can repeatedly search, revise its plan and invoke external services without clear stopping conditions.

Treat every tool as a typed interface with a documented purpose, validated arguments and narrowly scoped credentials. The model may propose a ticket, but application code should verify permissions and require confirmation before submission. Enforce limits on tool calls, execution time and spending outside the model. Make write operations idempotent where possible so retries do not create duplicate tickets. Persist workflow state explicitly, including completed actions and outstanding approvals, rather than relying on conversation text alone. Tool responses are also untrusted inputs: a retrieved page or service error must not become an instruction that overrides the assistant’s rules or authorises further actions.

LLM evaluation should measure the complete application while also isolating failing components. Build a dataset of representative questions, expected evidence and acceptable outcomes, including ambiguous requests and cases where the assistant should abstain. For retrieval, measure whether relevant passages appear within the candidate set using metrics such as recall at a chosen cutoff. Where relevance labels support it, ranking metrics help assess ordering. For answers, examine correctness, evidence support, citation accuracy and completeness. For tools, test argument validity, permission enforcement and action success. Track latency and cost alongside quality, because an improvement that breaches operational constraints may not be usable.

Automated checks are useful for structured outputs, citation identifiers and known policy violations. Model-based judges can help assess open-ended answers, but they require clear rubrics and calibration against human review. Avoid treating a judge model’s preference as objective truth. Keep evaluation examples separate from prompt development and fine-tuning data, and maintain slices for difficult documents, languages, permissions and failure scenarios. Run regression tests whenever models, prompts, chunking or retrieval settings change. In production, sample traces responsibly and investigate negative feedback. User satisfaction alone cannot establish factual reliability, especially when an incorrect answer sounds confident or an unauthorised action appears convenient.

The main security boundary is the application, not the prompt. Authenticate users, enforce document-level authorisation and apply least privilege to every service account. Prompt injection occurs when untrusted content tries to redirect model behaviour, including through retrieved documents or tool outputs. Clear instruction boundaries and model-side safeguards can help, but they do not provide complete protection. Prevent retrieved text from granting permissions, selecting arbitrary credentials or bypassing approval gates. Restrict network destinations and available tools according to the task. A support assistant should not acquire broad administrative access merely because a user asks it to troubleshoot a problem comprehensively and quickly.

Data protection also covers embeddings, prompts, conversation history, caches and telemetry. Embeddings should not be treated as anonymised data merely because they are numerical. Check provider policies and contractual controls for retention, training use, processing location and deletion against organisational requirements. Redact sensitive values where appropriate, encrypt data in transit and at rest, and avoid logging credentials or unnecessary personal information. Test cross-user and cross-tenant leakage, including cache collisions and stale access rules. For higher-impact actions, provide clear previews, approval records and audit trails. Plan how to revoke access, disable a tool and investigate a suspected disclosure without depending on model cooperation.

Deploying LLM applications requires managing variable inference latency, token consumption and external service limits. Choose managed inference or self-hosting according to privacy needs, workload, model availability and operational capacity. Separate ingestion workers from interactive requests so a large document update cannot exhaust the answering service. Apply timeouts, bounded retries, concurrency limits and backpressure. Streaming can improve perceived responsiveness, but partial output may need buffering or checks before display in sensitive workflows. Cache only where freshness and authorisation rules permit it. For this assistant, cache keys must distinguish relevant permissions and document versions rather than sharing answers solely because questions look similar.

LLMOps extends deployment discipline to models, prompts, retrieval configuration, datasets and evaluation results. Version these artefacts together so each release is reproducible and reversible. Trace requests across retrieval, reranking, generation and tools, recording useful timings and identifiers without exposing sensitive content. Monitor failure rates, retrieval misses, token use and latency distributions, then compare them with the evaluation baseline. Release changes gradually, with rollback criteria and safe fallbacks when dependencies fail. If evidence retrieval is unavailable, the assistant should disclose that limitation rather than confidently improvise internal policy. Scale from measured bottlenecks: adding inference capacity will not fix slow filtering or a broken ingestion pipeline.