Vector embeddings and RAG, explained from scratch

I was interviewing at a Melbourne AI startup a while back, and the conversation got to vector embeddings and RAG (Retrieval-Augmented Generation) pretty quickly. The question was: "How would you build something that can search 10,000 technical documents and actually understand what's being asked?"

Plain keyword search doesn't get you there. Someone searches for one phrase, the document uses a synonym, and you return nothing. That gap is the whole reason embeddings and semantic search exist.

So here's the path from text to numbers, why two pieces of text can be "close" mathematically, and what it takes to run a retrieval system that holds up under real traffic.

Part 1: Vector Embeddings Explained

What Are Vector Embeddings?

Say you want a computer to know that "king" and "monarch" mean roughly the same thing. You can't just tell it. A computer deals in numbers, not meaning.

Embeddings get around that by turning text into a long list of numbers, arranged so that things with similar meaning end up near each other in that numeric space.

How Text Becomes Numbers

Embedding models like OpenAI's text-embedding-ada-002, or open-source ones like Sentence Transformers, are neural networks trained on a lot of text. The training is what teaches them which numbers to assign.

import openai
from sentence_transformers import SentenceTransformer
import numpy as np

# Method 1: OpenAI Embeddings (Paid, High Quality)
openai.api_key = "your-api-key"

def get_openai_embedding(text):
    response = openai.Embedding.create(
        input=text,
        model="text-embedding-ada-002"
    )
    return response['data'][0]['embedding']

# Method 2: Open Source Alternative (Free)
model = SentenceTransformer('all-MiniLM-L6-v2')

def get_sentence_embedding(text):
    return model.encode(text)

# Example usage
texts = [
    "Machine learning algorithms",
    "AI and deep learning",
    "Pizza recipe ingredients", 
    "Neural networks training"
]

embeddings = [get_sentence_embedding(text) for text in texts]
print(f"Embedding dimension: {len(embeddings[0])}")  # Usually 384-1536

Measuring Similarity

Once you have vectors you can ask how close two of them are. Cosine similarity is the one you'll see most often. It measures the angle between two vectors and ignores their length, which is what you want here, because you care whether two things point the same direction, not how long the arrows are.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def calculate_similarity(text1, text2, model):
    # Get embeddings
    emb1 = model.encode([text1])
    emb2 = model.encode([text2])
    
    # Calculate cosine similarity
    similarity = cosine_similarity(emb1, emb2)[0][0]
    return similarity

# Example comparisons
model = SentenceTransformer('all-MiniLM-L6-v2')

print("ML vs AI:", calculate_similarity(
    "Machine learning algorithms", 
    "Artificial intelligence systems", 
    model
))  # ~0.85 (very similar)

print("ML vs Pizza:", calculate_similarity(
    "Machine learning algorithms", 
    "Pizza recipe ingredients", 
    model
))  # ~0.15 (not similar)

Part 2: RAG Systems & Vector Databases

The Problem with Traditional Search

Keyword search matches strings, not meaning. Someone searches "machine learning performance optimization," your document says "improving AI model efficiency," and you return nothing. Same idea, different words, zero results. Anyone who's used a corporate wiki search box knows the feeling.

What is RAG?

RAG is less clever than it sounds. You don't retrain the model on your data. You search your documents for the bits that look relevant to the question, paste them into the prompt, and ask the LLM to answer using those. That's it. The model never "learned" your docs; it just got handed the right pages before answering.

Vector Database Options

A vector database is just a store that's good at "find me the closest vectors to this one" over millions of rows. You can do it in Postgres with pgvector for a while, and honestly that's where I'd start. Once you outgrow that, the usual suspects:

Building a Simple RAG System

Here's a small one end to end, using Chroma locally and OpenAI for the answer. It's enough to answer questions over a pile of company docs. It's also enough to show you where the real work hides, which is the chunking, not the clever part.

import chromadb
import openai
from sentence_transformers import SentenceTransformer
import PyPDF2
import textwrap

class SimpleRAG:
    def __init__(self):
        # Initialize components
        self.client = chromadb.Client()
        self.collection = self.client.create_collection("documents")
        self.embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
        openai.api_key = "your-openai-key"
    
    def add_document(self, text, doc_id):
        """Add a document to the vector store"""
        # Split into chunks
        chunks = self.chunk_text(text, chunk_size=500)
        
        for i, chunk in enumerate(chunks):
            # Create embedding
            embedding = self.embedding_model.encode(chunk).tolist()
            
            # Store in vector database
            self.collection.add(
                embeddings=[embedding],
                documents=[chunk],
                ids=[f"{doc_id}_chunk_{i}"]
            )
    
    def chunk_text(self, text, chunk_size=500):
        """Split text into overlapping chunks"""
        words = text.split()
        chunks = []
        
        for i in range(0, len(words), chunk_size - 50):  # 50 word overlap
            chunk = " ".join(words[i:i + chunk_size])
            chunks.append(chunk)
        
        return chunks
    
    def search_relevant_docs(self, query, top_k=3):
        """Find most relevant document chunks"""
        query_embedding = self.embedding_model.encode(query).tolist()
        
        results = self.collection.query(
            query_embeddings=[query_embedding],
            n_results=top_k
        )
        
        return results['documents'][0]
    
    def generate_answer(self, query, context_docs):
        """Generate answer using OpenAI with retrieved context"""
        context = "\n\n".join(context_docs)
        
        prompt = f"""
        Based on the following context documents, answer the user's question.
        If the answer isn't in the context, say "I don't have enough information."
        
        Context:
        {context}
        
        Question: {query}
        
        Answer:
        """
        
        response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=500,
            temperature=0.1
        )
        
        return response.choices[0].message.content
    
    def ask_question(self, query):
        """Complete RAG pipeline"""
        # 1. Retrieve relevant documents
        relevant_docs = self.search_relevant_docs(query)
        
        # 2. Generate answer with context
        answer = self.generate_answer(query, relevant_docs)
        
        return answer, relevant_docs

# Usage example
rag = SimpleRAG()

# Add some company documents
rag.add_document("""
Our company policy states that employees can work remotely 
up to 3 days per week. Remote work requests must be approved 
by direct managers and HR department.
""", "policy_001")

rag.add_document("""
The refund policy allows customers to return products within 
30 days of purchase. Digital products are non-refundable 
unless there are technical issues.
""", "policy_002")

# Ask questions
answer, sources = rag.ask_question(
    "Can employees work from home?"
)

print("Answer:", answer)
print("Sources:", sources)

What changes when this hits production

Where it actually gets used

Production Architecture

The demo above is one process. In production it's a few moving parts, and most of the headaches are the boring ones: keeping the index in sync with the source docs, and what to do when the LLM call times out. Rough shape:

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Load Balancer │    │  FastAPI Server │    │  Vector Database│
│                 │────▶│                 │────▶│  (Pinecone/     │
│                 │    │  - Embedding    │    │   Weaviate)     │
└─────────────────┘    │  - Retrieval    │    │                 │
                       │  - Generation   │    └─────────────────┘
                       └─────────────────┘
                              │
                              ▼
                       ┌─────────────────┐    ┌─────────────────┐
                       │  LLM Service    │    │  Document Store │
                       │  (OpenAI/Local) │    │  (S3/MinIO)     │
                       └─────────────────┘    └─────────────────┘

Performance Optimization

Monitoring and Evaluation

Production RAG systems need continuous monitoring to ensure quality and performance.

# Example monitoring code
import time
from dataclasses import dataclass
from typing import List

@dataclass
class RAGMetrics:
    query_latency: float
    retrieval_accuracy: float
    answer_relevance: float
    context_precision: float

class RAGMonitor:
    def __init__(self):
        self.metrics = []
    
    def track_query(self, query: str, answer: str, 
                   retrieved_docs: List[str], expected_answer: str = None):
        start_time = time.time()
        
        # Track latency
        latency = time.time() - start_time
        
        # Calculate retrieval accuracy (if ground truth available)
        accuracy = self.calculate_retrieval_accuracy(
            retrieved_docs, expected_docs
        ) if expected_answer else None
        
        # Store metrics
        metrics = RAGMetrics(
            query_latency=latency,
            retrieval_accuracy=accuracy,
            answer_relevance=self.score_relevance(answer, query),
            context_precision=self.score_context_precision(retrieved_docs)
        )
        
        self.metrics.append(metrics)
        
        # Alert if performance drops
        if latency > 2.0:  # 2 second threshold
            self.send_alert(f"High latency detected: {latency:.2f}s")
    
    def get_performance_summary(self):
        if not self.metrics:
            return None
            
        avg_latency = sum(m.query_latency for m in self.metrics) / len(self.metrics)
        avg_accuracy = sum(m.retrieval_accuracy for m in self.metrics if m.retrieval_accuracy) / len([m for m in self.metrics if m.retrieval_accuracy])
        
        return {
            "avg_latency": avg_latency,
            "avg_accuracy": avg_accuracy,
            "total_queries": len(self.metrics)
        }

Key Takeaways

Vector embeddings and RAG systems represent a fundamental shift in how we build intelligent applications. They enable semantic understanding that goes far beyond traditional keyword matching, opening up possibilities for truly intelligent document search, customer support, and knowledge management systems.

As we've seen, the technology is mature enough for production use, with robust tools and clear patterns emerging. The key to success lies in understanding both the fundamentals and the practical engineering challenges of scaling these systems in enterprise environments.