How to Build a Fully Offline Local RAG Pipeline with Ollama, AnythingLLM, and LlamaIndex

The fix is running a zero-cloud, fully offline local RAG pipeline. By pairing Ollama for localized inference, AnythingLLM for desktop document chat, and LlamaIndex for programmatic orchestration, you create an air-gapped knowledge assistant. No API subscriptions, zero bandwidth costs, and zero chance of your data leaking to a third-party server.

1. System Architecture: How Local RAG Keeps Data Air-Gapped

A typical cloud LLM processes your query by shipping your documents to remote storage, converting them to numerical representations (embeddings) on public clusters, and returning responses via a metered web service.

Local RAG shifts every component onto your hardware:

[ Local File: PDF / Doc / Code ]
               │
               ▼
[ Text Chunking Layer (LlamaIndex) ]
               │
               ▼
[ Local Embedding Model (Ollama: nomic-embed-text) ]
               │
               ▼
[ Embedded Vector Storage (AnythingLLM / ChromaDB) ] ◄── [ User Query ]
               │                                            │
               └───────────────┬────────────────────────────┘
                               ▼
            [ Relevant Text Snippets Context ]
                               │
                               ▼
             [ Local LLM (Ollama: Llama-3.1-8B) ]
                               │
                               ▼
                 [ Final Grounded Answer ]

When you query an offline pipeline, your hardware performs two distinct model passes:

1. The Ingestion & Embedding Phase: An embedding model converts text chunks into mathematical vectors that represent semantic meaning.
2. The Generation Phase: A localized chat LLM ingests the top matching chunks along with your question, producing answers grounded entirely in your private records.

Hardware Footprint & Requirements

Running these workloads locally requires balancing system memory and GPU capacity.

2. Infrastructure Setup: Serving Models Offline with Ollama

Ollama acts as your local model engine. It runs low-footprint quantized models natively on your GPU or CPU without exposing open network ports to the public internet.

Step 1: Install and Initialize Ollama

Download the binary for your OS (macOS, Linux, or Windows) from Ollama’s site. Open your terminal and confirm the service is listening:

ollama --version

Step 2: Fetch Your Local Model Pair

A functional RAG system requires two different models—one to calculate vector embeddings and one to synthesize text.

Execute the following pulls in your terminal:

# 1. Pull the embedding model (137M parameters, low footprint)
ollama pull nomic-embed-text

# 2. Pull the reasoning/chat model (8B parameters, balanced performance)
ollama pull llama3.1:8b

Why two models? Chat models generate natural language, but they suck at building dense math matrices for search. Dedicated embedding models like nomic-embed-text compress text blocks into 768-dimensional vectors, enabling sub-millisecond semantic lookups across thousands of pages.

3. Desktop Document Workspace: GUI Setup with AnythingLLM

If you prefer a clean workspace UI without writing custom frontend code, AnythingLLM acts as an all-in-one desktop client. It handles text extraction, vector storage, and workspace organization automatically.

┌────────────────────────────────────────────────────────────────────────┐
│ AnythingLLM Workspace: [ Confidential Financials ]                    │
├────────────────────────────────┬───────────────────────────────────────┤
│ Workspace Settings             │ Chat Interface                        │
│                                │                                       │
│ LLM Provider:  [ Ollama     ▼ ]│ User: What was Q3 net revenue?        │
│ Model:         [ llama3.1:8b▼ ]│                                       │
│                                │ System (Thinking...):                 │
│ Embedder:      [ Ollama     ▼ ]│ Searching local vector store...       │
│ Model:         [ nomic-e... ▼ ]│ Context matched 3 chunks from Q3.pdf  │
│                                │                                       │
│ Vector DB:     [ LanceDB    ▼ ]│ Assistant: Based on page 14 of the    │
│ (Built-in, fully offline)      │ Q3 report, net revenue was $4.2M...   │
└────────────────────────────────┴───────────────────────────────────────┘

Step-by-Step Configuration Guide:

1. Launch AnythingLLM and create a new workspace (e.g., “Legal Vault”).
2. Navigate to Settings → LLM Provider and select Ollama. Set the Base URL to http://127.0.0.1:11434 and select llama3.1:8b.
3. Navigate to Vector Database settings and choose LanceDB or ChromaDB. Both run embedded inside your local workspace with zero setup.
4. Navigate to Embedding Provider, pick Ollama, and select nomic-embed-text.
5. Drag and Drop Files: Upload your target PDFs, TXT, or Markdown documents into the workspace panel. Click Move to Workspace, then click Save and Embed.

Now you can chat with your desktop files completely offline. Turn off your Wi-Fi card and test it—it won’t drop a single query.

4. Custom Developer Pipeline: Python Orchestration with LlamaIndex

For programmatic control—such as automated data pipelines, custom web apps, or CLI tools—use LlamaIndex. It lets you fine-tune chunking strategies, set similarity metrics, and filter retrieval context.

Step 1: Set Up Your Python Virtual Environment

mkdir local-rag-pipeline && cd local-rag-pipeline
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install llama-index llama-index-llms-ollama llama-index-embeddings-ollama

Step 2: Build the RAG Script

Create a file named offline_rag.py. Place your private text or PDF files inside a folder named ./private_data.

import sys
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding

def main():
    print("Initializing offline LLM and Embedding models...")
    
    # Configure Ollama Local Chat Model
    Settings.llm = Ollama(
        model="llama3.1:8b", 
        request_timeout=120.0,
        base_url="http://localhost:11434"
    )
    
    # Configure Ollama Local Embedding Model
    Settings.embed_model = OllamaEmbedding(
        model_name="nomic-embed-text",
        base_url="http://localhost:11434"
    )
    
    # Set explicit chunking parameters
    Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

    print("Loading documents from ./private_data...")
    try:
        documents = SimpleDirectoryReader("./private_data").load_data()
    except Exception as e:
        print(f"Error loading files: {e}")
        sys.exit(1)

    print("Indexing documents into local vector store...")
    index = VectorStoreIndex.from_documents(documents)

    # Instantiate query engine fetching top 3 matched chunks
    query_engine = index.as_query_engine(similarity_top_k=3)

    print("\nLocal RAG Ready. Ask your question below (Type 'exit' to quit).\n")
    while True:
        prompt = input("Query > ")
        if prompt.strip().lower() in ["exit", "quit"]:
            break
        
        response = query_engine.query(prompt)
        print(f"\n[Answer]:\n{response}\n")

if __name__ == "__main__":
    main()

Run your script:

python offline_rag.py

Pro-Tips & Troubleshooting

1. Fix Hallucinations: Adjust Chunk Overlap First

If your model misses facts spanning across page boundaries, don’t rush to swap your chat LLM. Adjust your chunk overlap settings first. When chunk size is 512 tokens, setting chunk_overlap=50 ensures that sentence context isn’t severed arbitrarily during ingestion.

2. The Golden Rule of Embedding Mismatches

If you change your embedding model (e.g., switching from nomic-embed-text to mxbai-embed-large), you must purge and rebuild your vector database. Vectors produced by different algorithms inhabit completely different coordinate spaces. Querying an index built with nomic using mxbai produces total garbage output.

3. Mitigate Slow Generation Speeds

If your queries stall or lag on desktop hardware, check your system offload settings. Ensure Ollama utilizes your system GPU layers instead of running on CPU threads:

# Check if GPU offloading is active during a query session
ollama ps

If VRAM is restricted, switch your inference model to a smaller quantized build like llama3.2:3b while keeping nomic-embed-text for retrieval accuracy.

Next Steps for Your Offline AI Stack

Building a local RAG stack gives you absolute privacy and control over your data pipelines. You no longer need to compromise between modern semantic search and strict compliance standards.

To expand your setup further:

• Integrate Hybrid Search: Combine semantic vector lookup with keyword search (BM25) inside LlamaIndex for precise code symbol matches.
• Automate Document Ingestion: Set up a cron job or file watcher script that re-indexes your ./private_data directory whenever new files land in your local directory.

Now your critical documents stay where they belong—under your control, on your machine, completely off the grid.