What is RAG

Retrieval-Augmented Generation. The model reads relevant documents from your database, then answers based on them.

Flow: user asks question → vector search for relevant docs → LLM answers using those docs as context.

What is Fine-tuning

Train an existing model on your specific data. Model “learns” your domain knowledge.

Requires: examples of question-answer pairs, GPU time, ML expertise.

When to Use RAG

Data changes frequently. Adding new documents = no retraining.

You need citations. RAG naturally shows sources.

Small budget. No GPU costs, no ML team.

Most business use cases.

When to Use Fine-tuning

Specific writing style or format needed consistently.

Domain-specific jargon the base model doesn’t know.

Very stable knowledge base. Not changing often.

Better cost economics at scale (millions of queries).

RAG Architecture

Documents → chunker (split into pieces) → embedder (convert to vectors) → vector DB (Pinecone, Weaviate, pgvector).

Query time: embed question → search vector DB → get top 5 chunks → send to LLM with question.

Chunking Strategy

Fixed size: simple but breaks context.

Sentence-based: better readability, variable size.

Semantic: chunks by topic. Best quality, most complex.

Overlap: 100-200 tokens between chunks preserves context.

Embedding Models

OpenAI text-embedding-3-small: cheap, works well for English.

Cohere Embed v3: great for multilingual.

Open source: sentence-transformers – if you want to self-host.

Common RAG Pitfalls

Poor retrieval = bad answers. Test retrieval separately from generation.

Long context: LLM ignores middle of context. Put most important info at start/end.

No feedback loop: track which answers users mark helpful. Improve retrieval based on that.

Fine-tuning in Practice

OpenAI fine-tuning: gpt-3.5 fine-tuning is affordable. gpt-4 fine-tuning available but expensive.

Open source: Llama 3, Mistral. More control but more work.

Data requirements: minimum 100 examples. 1000+ for good results.

Hybrid Approach

Most production systems: RAG + light fine-tuning.

Fine-tune for tone/format. RAG for facts/knowledge.

Best of both worlds.

Based on Real Projects

This guide is based on our work with:

• RepuAI
• SmartSearch

Further Reading

If this guide helped you, you might also want to read our comprehensive guide on AI Solutions.