Building RAG Applications — A Practical Guide

Retrieval-Augmented Generation (RAG) has become the standard pattern for building AI applications that need to reason over your own data. Instead of fine-tuning a model, you retrieve relevant context and feed it to the LLM at inference time.

This post covers building a RAG system from scratch: chunking strategies, embedding generation, vector storage, retrieval techniques, and response generation.

The RAG Architecture

At its core, a RAG application has three stages:

┌─────────────┐    ┌──────────────┐    ┌──────────────┐
│  Ingestion  │───▶│  Retrieval   │───▶│  Generation  │
│             │    │              │    │              │
│ Documents → │    │ Query →      │    │ Context +    │
│ Chunks →    │    │ Embedding →  │    │ Query →      │
│ Embeddings  │    │ Vector Search│    │ LLM Response │
└─────────────┘    └──────────────┘    └──────────────┘

Each stage has its own challenges. Let us work through them.

Document Chunking Strategies

The quality of your RAG system depends heavily on how you chunk your documents. Too small, and you lose context. Too large, and you retrieve irrelevant information.

Fixed-Size Chunking

The simplest approach is fixed-size chunks with overlap:

interface ChunkConfig {
  chunkSize: number;
  chunkOverlap: number;
}

function chunkDocument(
  text: string,
  config: ChunkConfig
): string[] {
  const chunks: string[] = [];
  let start = 0;

  while (start < text.length) {
    const end = Math.min(start + config.chunkSize, text.length);
    chunks.push(text.slice(start, end));

    // Move forward, accounting for overlap
    start = end - config.chunkOverlap;
  }

  return chunks;
}

// Usage for ~500 character chunks with 100 character overlap
const chunks = chunkDocument(documentText, {
  chunkSize: 500,
  chunkOverlap: 100,
});

This works but has a major flaw: it can split sentences in half, losing semantic coherence.

Semantic Chunking

A better approach respects sentence and paragraph boundaries:

interface SemanticChunk {
  text: string;
  metadata: {
    startIndex: number;
    endIndex: number;
    sentenceCount: number;
  };
}

function chunkSemantically(
  text: string,
  maxChunkSize: number
): SemanticChunk[] {
  // Split into sentences
  const sentences = text.match(/[^.!?]+[.!?]+/g) || [text];

  const chunks: SemanticChunk[] = [];
  let currentChunk = '';
  let startIndex = 0;
  let sentenceCount = 0;

  for (const sentence of sentences) {
    const potentialChunk = currentChunk + sentence;

    if (potentialChunk.length <= maxChunkSize) {
      currentChunk = potentialChunk;
      sentenceCount++;
    } else {
      // Finalize current chunk
      if (currentChunk) {
        chunks.push({
          text: currentChunk.trim(),
          metadata: {
            startIndex,
            endIndex: startIndex + currentChunk.length,
            sentenceCount,
          },
        });
      }

      // Start new chunk
      startIndex += currentChunk.length;
      currentChunk = sentence;
      sentenceCount = 1;
    }
  }

  // Don't forget the last chunk
  if (currentChunk) {
    chunks.push({
      text: currentChunk.trim(),
      metadata: {
        startIndex,
        endIndex: startIndex + currentChunk.length,
        sentenceCount,
      },
    });
  }

  return chunks;
}

For code or structured documents, you might want to chunk based on markdown headers or code blocks instead of sentences. The key principle: respect natural boundaries in the content.

Metadata Preservation

Your chunks are only useful if you know where they came from:

interface DocumentChunk {
  id: string;
  text: string;
  metadata: {
    documentId: string;
    documentTitle: string;
    chunkIndex: number;
    startIndex: number;
    endIndex: number;
    createdAt: Date;
    tags?: string[];
    category?: string;
  };
  embedding?: number[];
}

function createDocumentChunk(
  documentId: string,
  documentTitle: string,
  text: string,
  chunkIndex: number,
  startIndex: number,
  endIndex: number
): DocumentChunk {
  return {
    id: `${documentId}-${chunkIndex}`,
    text,
    metadata: {
      documentId,
      documentTitle,
      chunkIndex,
      startIndex,
      endIndex,
      createdAt: new Date(),
    },
  };
}

Metadata enables filtering (e.g., “only search documents from 2024”) and provides attribution in the generated response.

Embedding Generation

Once you have chunks, you need to convert them into embeddings. I use OpenAI’s text-embedding-3-small for most applications — it is fast, accurate, and cost-effective.

interface EmbeddingService {
  generateEmbeddings(texts: string[]): Promise<number[][]>;
}

class OpenAIEmbeddingService implements EmbeddingService {
  private apiKey: string;
  private model = 'text-embedding-3-small';
  private batchSize = 100; // OpenAI's limit

  constructor(apiKey: string) {
    this.apiKey = apiKey;
  }

  async generateEmbeddings(texts: string[]): Promise<number[][]> {
    const allEmbeddings: number[][] = [];

    // Process in batches
    for (let i = 0; i < texts.length; i += this.batchSize) {
      const batch = texts.slice(i, i + this.batchSize);

      const response = await fetch('https://api.openai.com/v1/embeddings', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer ${this.apiKey}`,
        },
        body: JSON.stringify({
          model: this.model,
          input: batch,
        }),
      });

      if (!response.ok) {
        throw new Error(`Embedding API error: ${response.statusText}`);
      }

      const data = await response.json();
      const embeddings = data.data.map((item: any) => item.embedding);

      allEmbeddings.push(...embeddings);
    }

    return allEmbeddings;
  }
}

Batching is important for throughput. Processing 100 embeddings at a time is much faster than one at a time.

Vector Storage

For production applications, you need a vector database. Options include Pinecone, Weaviate, Qdrant, and pgvector for PostgreSQL. Here is a simple implementation using pgvector:

import { Pool } from 'pg';

class VectorStore {
  private pool: Pool;

  constructor(connectionString: string) {
    this.pool = new Pool({ connectionString });
  }

  async initialize(): Promise<void> {
    await this.pool.query(`
      CREATE TABLE IF NOT EXISTS document_chunks (
        id TEXT PRIMARY KEY,
        text TEXT NOT NULL,
        embedding vector(1536),
        metadata JSONB,
        created_at TIMESTAMPTZ DEFAULT NOW()
      );

      CREATE INDEX IF NOT EXISTS chunks_embedding_idx
      ON document_chunks
      USING ivfflat (embedding vector_cosine_ops)
      WITH (lists = 100);
    `);
  }

  async insertChunks(chunks: DocumentChunk[]): Promise<void> {
    const client = await this.pool.connect();

    try {
      await client.query('BEGIN');

      for (const chunk of chunks) {
        if (!chunk.embedding) continue;

        await client.query(
          `INSERT INTO document_chunks (id, text, embedding, metadata)
           VALUES ($1, $2, $3, $4)
           ON CONFLICT (id) DO UPDATE
           SET text = EXCLUDED.text,
               embedding = EXCLUDED.embedding,
               metadata = EXCLUDED.metadata`,
          [
            chunk.id,
            chunk.text,
            `[${chunk.embedding.join(',')}]`,
            JSON.stringify(chunk.metadata),
          ]
        );
      }

      await client.query('COMMIT');
    } catch (error) {
      await client.query('ROLLBACK');
      throw error;
    } finally {
      client.release();
    }
  }

  async search(
    queryEmbedding: number[],
    limit: number = 5,
    filters?: Record<string, any>
  ): Promise<Array<{ chunk: DocumentChunk; similarity: number }>> {
    let whereClause = '';
    const params: any[] = [`[${queryEmbedding.join(',')}]`, limit];
    let paramIndex = 3;

    if (filters) {
      const conditions: string[] = [];
      for (const [key, value] of Object.entries(filters)) {
        conditions.push(`metadata->>$${paramIndex} = $${paramIndex + 1}`);
        params.push(key, value);
        paramIndex += 2;
      }
      if (conditions.length > 0) {
        whereClause = 'AND ' + conditions.join(' AND ');
      }
    }

    const result = await this.pool.query(
      `SELECT id, text, metadata, 1 - (embedding <=> $1) as similarity
       FROM document_chunks
       WHERE embedding IS NOT NULL
       ${whereClause}
       ORDER BY embedding <=> $1
       LIMIT $2`,
      params
    );

    return result.rows.map(row => ({
      chunk: {
        id: row.id,
        text: row.text,
        metadata: row.metadata,
      },
      similarity: row.similarity,
    }));
  }
}

The vector cosine similarity operator (<=>) in pgvector makes semantic search efficient. The IVFFlat index accelerates queries significantly as your dataset grows.

Retrieval Techniques

Basic semantic search is a good start, but production RAG systems need more sophisticated retrieval.

Hybrid Search

Combine semantic search with keyword search for the best results:

interface SearchResult {
  chunk: DocumentChunk;
  semanticScore: number;
  keywordScore: number;
  combinedScore: number;
}

async function hybridSearch(
  query: string,
  vectorStore: VectorStore,
  keywordIndex: KeywordIndex,
  limit: number = 5
): Promise<SearchResult[]> {
  // Get semantic results
  const queryEmbedding = await embeddingService.generateEmbeddings([query]);
  const semanticResults = await vectorStore.search(queryEmbedding[0], limit * 2);

  // Get keyword results
  const keywordResults = await keywordIndex.search(query, limit * 2);

  // Combine and re-rank
  const combined = new Map<string, SearchResult>();

  // Add semantic results
  for (const { chunk, similarity } of semanticResults) {
    combined.set(chunk.id, {
      chunk,
      semanticScore: similarity,
      keywordScore: 0,
      combinedScore: similarity * 0.7, // Weight semantic higher
    });
  }

  // Add keyword results and update scores
  for (const { chunk, score } of keywordResults) {
    const existing = combined.get(chunk.id);
    if (existing) {
      existing.keywordScore = score;
      existing.combinedScore = existing.semanticScore * 0.7 + score * 0.3;
    } else {
      combined.set(chunk.id, {
        chunk,
        semanticScore: 0,
        keywordScore: score,
        combinedScore: score * 0.3,
      });
    }
  }

  // Sort by combined score
  return Array.from(combined.values())
    .sort((a, b) => b.combinedScore - a.combinedScore)
    .slice(0, limit);
}

Hybrid search catches cases where semantic similarity misses but keyword matching works (e.g., exact product codes, names).

Re-Ranking

After retrieval, re-rank results based on the specific query:

interface ReRanker {
  rerank(query: string, results: DocumentChunk[]): Promise<DocumentChunk[]>;
}

class CrossEncoderReRanker implements ReRanker {
  async rerank(query: string, results: DocumentChunk[]): Promise<DocumentChunk[]> {
    // Use a cross-encoder model for more accurate scoring
    const scores = await Promise.all(
      results.map(async chunk => {
        const response = await fetch('https://api.openai.com/v1/chat/completions', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
          },
          body: JSON.stringify({
            model: 'gpt-4o-mini',
            messages: [
              {
                role: 'system',
                content: 'Rate the relevance of this document to the query on a scale of 0-10.',
              },
              {
                role: 'user',
                content: `Query: ${query}\n\nDocument: ${chunk.text.slice(0, 500)}...`,
              },
            ],
          }),
        });

        const data = await response.json();
        const scoreText = data.choices[0].message.content;
        const score = parseInt(scoreText.match(/\d+/)?.[0] || '5');
        return { chunk, score };
      })
    );

    return scores
      .sort((a, b) => b.score - a.score)
      .map(item => item.chunk);
  }
}

Re-ranking is computationally expensive, so use it on a smaller set of pre-filtered results (e.g., re-rank the top 20 candidates).

Response Generation

With retrieved context in hand, generate a response using an LLM:

interface RAGConfig {
  maxContextLength: number;
  maxResponseLength: number;
  temperature: number;
}

class RAGGenerator {
  async generate(
    query: string,
    contextChunks: DocumentChunk[],
    config: RAGConfig
  ): Promise<string> {
    // Build context from chunks
    const context = contextChunks
      .map((chunk, i) => `[Source ${i + 1}]: ${chunk.text}`)
      .join('\n\n');

    const systemPrompt = `You are a helpful assistant. Answer the user's question using the provided context.
If the answer is not in the context, say so. Cite your sources using [Source N] notation.`;

    const userPrompt = `Context:\n${context}\n\nQuestion: ${query}`;

    const response = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
      },
      body: JSON.stringify({
        model: 'gpt-4-turbo',
        messages: [
          { role: 'system', content: systemPrompt },
          { role: 'user', content: userPrompt },
        ],
        max_tokens: config.maxResponseLength,
        temperature: config.temperature,
      }),
    });

    const data = await response.json();
    return data.choices[0].message.content;
  }
}

Citing sources builds trust and lets users verify the information.

Putting It All Together

class RAGApplication {
  private vectorStore: VectorStore;
  private embeddingService: EmbeddingService;
  private generator: RAGGenerator;

  async ingestDocument(document: {
    id: string;
    title: string;
    text: string;
  }): Promise<void> {
    // Chunk the document
    const semanticChunks = chunkSemantically(document.text, 1000);
    const chunks = semanticChunks.map((chunk, i) =>
      createDocumentChunk(
        document.id,
        document.title,
        chunk.text,
        i,
        chunk.metadata.startIndex,
        chunk.metadata.endIndex
      )
    );

    // Generate embeddings
    const texts = chunks.map(c => c.text);
    const embeddings = await this.embeddingService.generateEmbeddings(texts);

    // Attach embeddings to chunks
    chunks.forEach((chunk, i) => {
      chunk.embedding = embeddings[i];
    });

    // Store in vector database
    await this.vectorStore.insertChunks(chunks);
  }

  async query(queryText: string): Promise<{
    answer: string;
    sources: DocumentChunk[];
  }> {
    // Generate query embedding
    const [queryEmbedding] = await this.embeddingService.generateEmbeddings([
      queryText,
    ]);

    // Retrieve relevant chunks
    const results = await this.vectorStore.search(queryEmbedding, 5);
    const sources = results.map(r => r.chunk);

    // Generate response
    const answer = await this.generator.generate(queryText, sources, {
      maxContextLength: 4000,
      maxResponseLength: 1000,
      temperature: 0.7,
    });

    return { answer, sources };
  }
}

Lessons Learned

Building RAG applications has taught me several important lessons:

1. Chunking matters more than you think. Poor chunking is the number one cause of bad RAG performance. Invest time in understanding your document structure and chunk accordingly.

2. Metadata is your friend. Rich metadata enables filtering and provides context that improves retrieval accuracy.

3. Evaluate continuously. Use a test set of questions and expected answers to measure retrieval quality. Monitor precision and recall over time.

4. Consider the user interface. Show sources, let users filter by date or category, and provide feedback mechanisms when answers are not helpful.

5. Start simple, then optimize. Basic semantic search with good chunking often beats complex retrieval algorithms with poor chunking. Add complexity only when you have specific problems to solve.

Conclusion

RAG systems combine the strengths of vector search and LLMs to create applications that can reason over your own data. The key components — thoughtful chunking, quality embeddings, efficient vector storage, and careful retrieval — work together to deliver accurate, contextual responses.

Start with a simple implementation, measure its performance, and iterate. The patterns in this post provide a solid foundation for building production-ready RAG applications.