Run a Local LLM with Ollama and Hook It Up to a Simple RAG Pipeline

Run a local LLM with Ollama and add RAG in three steps.
1️⃣ Install Ollama and pull a model → curl -fsSL https://ollama.com/install.sh | sh && ollama pull llama3.1. The server starts on localhost:11434.
2️⃣ Create a vector store (e.g., using FAISS) and index your docs → `python - <<'PY'
from langchain.document_loaders import TextLoader
from langchain.embeddings import OllamaEmbeddings
from langchain.vectorstores import FAISS
loader = TextLoader('data.txt')
docs = loader.load_and_split()
emb = OllamaEmbeddings(base_url='http://localhost:11434')
index = FAISS.from_documents(docs, emb)
index.save_local('faiss_index')
PY`
3️⃣ Query with the local model → `python - <<'PY'
from langchain.chains import RetrievalQA
from langchain.llms import Ollama
from langchain.vectorstores import FAISS
index = FAISS.load_local('faiss_index', OllamaEmbeddings(base_url='http://localhost:11434'))
qa = RetrievalQA.from_chain_type(llm=Ollama(base_url='http://localhost:11434'), retriever=index.as_retriever())
print(qa.run('What is the main benefit of RAG?'))
PY`
🔧 Gotcha: Ollama’s default context window may be smaller than some models; chunk your documents to ≤ 500 tokens to avoid truncation.