FAISS vector database: When FAISS is enough for RAG

You started a new RAG project and the tutorial used a hosted vector database. You signed up, created API keys, set up billing and spent an afternoon on auth and rate limits. Your corpus is 20k documents. FAISS would have searched it in-process, in a few dozen lines of Python, with nothing new to run.

FAISS is an open-source similarity search library that finds the nearest vectors to a query by comparing embeddings in memory, with exact or approximate indexes. It is not a vector database: it has no server, no durable storage engine and no metadata store. This guide covers when that is enough for RAG, which index to pick, how to persist it, and when to move to a database.

Key takeaways

• FAISS is a library for vector similarity search and clustering open-sourced by Meta, released under the MIT license, so it is free to use.
• FAISS is not a vector database: you handle persistence, document storage, metadata filters, replication and concurrent writes yourself.
• IndexFlatIP and IndexFlatL2 are the only FAISS indexes that guarantee exact results, and they need no training.
• IVF indexes need a training pass on sample vectors, and HNSW indexes cannot remove vectors once added.
• CPU indexes are safe for concurrent searches, but any call that changes the index needs your own lock.
• Pick a vector database when you need live writes from many processes, per-tenant isolation or rich metadata filters.

Is FAISS a vector database?

No. FAISS is a library you import into your process, not a service you connect to. The FAISS paper describes it as a toolkit of indexing methods for vector similarity search, which it calls a core function of vector databases.

A vector database such as Qdrant, Weaviate, Milvus or pgvector wraps that search in a server. It adds durable storage, metadata filters, access control and replication. FAISS gives you the search engine and leaves the rest to you. That is a cost when you need those features, and a saving when you do not.

How does FAISS work?

FAISS stores your embeddings in an index object in memory and compares the query vector against them. A flat index compares against every vector, so results are exact. Approximate indexes narrow the search first, by clustering (IVF) or by walking a graph of neighbors (HNSW), and trade a little recall for speed.

FAISS returns integer ids and scores, not documents. You keep the text and metadata somewhere else and map the ids back.

The supported install is conda: conda install -c pytorch -c conda-forge faiss-cpu. A faiss-cpu wheel on PyPI also works with pip install faiss-cpu, but the FAISS install guide documents only conda.

Why use FAISS for production RAG?

Because many RAG corpora are small enough to fit in one process’s memory, and then a library is simpler than a service. Three advantages follow:

1. No network hop. The search runs in your process, so retrieval latency is the search itself plus nothing.
2. No new service. No cluster to run, no account, no per-query bill. The index is a file you ship or load at startup.
3. No lock-in. The index is plain vectors, so moving to a vector database later means re-inserting the same embeddings.

The trade-off: FAISS is a library, not a service. You own persistence, document storage, sharding and replication. Measure your own latency and memory before you decide, since both depend on your vector count, dimension and index type.

Which FAISS index type should you use?

Start with a flat index, and move to IVF or HNSW only when flat search is too slow on your corpus. The FAISS index guidelines give these defaults:

Index	Use when	Trade-off
IndexFlatIP or IndexFlatL2	Small corpora, or you need exact results	Exact search, no training, cost grows with every vector
IndexIVFFlat	Below about 1M vectors and flat is too slow	Needs training; the guidelines suggest 4√N to 16√N clusters
IVF65536_HNSW32 with PQ	1M to 10M vectors	Needs a large training sample; PQ compresses vectors at some recall loss
IndexHNSWFlat	You have memory to spare and want fast queries	No training; memory grows with the M parameter; no removal

For most RAG projects, start with IndexFlatIP on normalized embeddings. Inner product on unit vectors equals cosine similarity. Flat indexes do not take custom ids, so wrap them in IndexIDMap if your ids are not 0 to n.

The index only stores what your embedding model produces. Re-run your retrieval evals when you change models, and see retriever top-k tuning for measuring recall on your own questions.

What does a production FAISS setup look like?

A small class that builds, saves, loads and searches the index, with the document texts saved beside it.

# filename: app/rag/faiss_store.py
# description: Production FAISS setup for RAG. Build once, save, load on startup.
import json
from pathlib import Path

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer


class FaissStore:
    def __init__(self, dim: int, index_path: str):
        self.dim = dim
        self.index_path = Path(index_path)
        self.docs_path = self.index_path.with_suffix('.docs.json')
        self.index = None
        self.docs: list[str] = []  # row i of the index is self.docs[i]

    def build(self, docs: list[str], embedder: SentenceTransformer) -> None:
        vectors = embedder.encode(docs, normalize_embeddings=True)
        self.index = faiss.IndexFlatIP(self.dim)
        self.index.add(np.asarray(vectors, dtype='float32'))
        self.docs = docs

    def save(self) -> None:
        faiss.write_index(self.index, str(self.index_path))
        self.docs_path.write_text(json.dumps(self.docs))

    def load(self) -> None:
        self.index = faiss.read_index(str(self.index_path))
        self.docs = json.loads(self.docs_path.read_text())

    def search(self, query_vector: np.ndarray, k: int = 5) -> list[tuple[str, float]]:
        scores, idxs = self.index.search(query_vector.reshape(1, -1).astype('float32'), k)
        # FAISS pads with -1 when the index holds fewer than k vectors
        return [(self.docs[i], float(s)) for i, s in zip(idxs[0], scores[0]) if i != -1]

Three decisions matter. IndexFlatIP scores by inner product, which equals cosine similarity once vectors are normalized. The query must be normalized the same way, so encode it with normalize_embeddings=True too. The docs file is written with the index, so one load restores both.

How do you persist and reload a FAISS index across restarts?

Build the index offline, save it with faiss.write_index, and load it once per worker at startup with faiss.read_index. In FastAPI, the lifespan handler is the place for that load.

# filename: app/main.py
# description: FastAPI lifespan loads the FAISS index once per worker.
from contextlib import asynccontextmanager

from fastapi import FastAPI, Request
from sentence_transformers import SentenceTransformer

from app.rag.faiss_store import FaissStore


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.embedder = SentenceTransformer('BAAI/bge-small-en-v1.5')
    app.state.store = FaissStore(dim=384, index_path='data/corpus.faiss')
    app.state.store.load()
    yield


app = FastAPI(lifespan=lifespan)


@app.get('/search')
def search(q: str, request: Request):
    vector = request.app.state.embedder.encode(q, normalize_embeddings=True)
    return request.app.state.store.search(vector, k=5)

Each worker holds its own copy of the index, so memory grows with the worker count. For worker settings, see FastAPI and Uvicorn for production.

Can FAISS handle concurrent queries and live updates?

Concurrent searches, yes. Concurrent writes, only with your own lock. The FAISS threading notes say CPU indexes are thread-safe for searches and other calls that do not change the index. GPU indexes are not thread-safe, even for searches.

Live updates are possible but limited. You can call add on a trained index at any time. remove_ids works on IndexFlat, IndexIVFFlat and IndexIDMap, but not on HNSW. Every change lives only in memory until you write the index again, and other workers do not see it.

When should you move from FAISS to a vector database?

When your needs pass what a library in one process can do. Ask these questions:

1. Does the index still fit in memory on every worker? If not, you need sharding or a server.
2. Do you need writes from many processes, visible to all readers? That is a database feature.
3. Do you need per-tenant isolation? FAISS has none. You can keep one index per tenant, or use a database with native tenancy.
4. Do you need metadata filters? FAISS can restrict a search to chosen ids with an IDSelector. Filters such as “user_id = X and created after Y” belong in a database such as pgvector.

If every answer favors FAISS, use FAISS. If one favors a database, test both on your data. For the framework that sits above either choice, see choosing a RAG framework.

What to do next

1. Count your vectors and estimate memory as vectors times dimension times 4 bytes for a flat index.
2. If it fits on every worker, build an IndexFlatIP prototype on normalized embeddings.
3. Save the index and docs together, and load both in the FastAPI lifespan handler.
4. Measure p95 search latency and recall@k on real questions. Try IVF or HNSW only if flat is too slow.
5. Write down the trigger for moving to a database: shared writes, tenancy, metadata filters or memory.

Frequently asked questions

Is FAISS free to use?

Yes. FAISS is released under the MIT license, so you can use it in commercial products at no cost. There is no hosted FAISS service and no per-query pricing. Your cost is the memory and CPU of the machines that load the index, plus the engineering time to add persistence, document storage and updates yourself.

What is the difference between FAISS and Chroma?

FAISS is a search library. Chroma is a vector database with a client API, persistence, document storage and metadata filters built in. With FAISS you write that storage and loading code yourself, as in this guide. Pick FAISS for a small static index inside one service, and a database when several services write.

Which FAISS index type should I start with?

Start with IndexFlatIP on normalized embeddings. It is exact, needs no training and gives cosine similarity. Move to IndexIVFFlat or IndexHNSWFlat only when flat search is too slow on your corpus. IVF needs a training pass, while HNSW uses more memory and cannot remove vectors once added.

How do I persist a FAISS index across restarts?

Call faiss.write_index(index, path) after building, and faiss.read_index(path) at startup. FAISS stores vectors, not text, so save the document texts and metadata in a sibling file or a database keyed by the same ids. In FastAPI, load both once per worker in the lifespan handler, not on each request.

How does FAISS compare to pgvector for RAG?

FAISS runs in your process and searches vectors only, while pgvector adds vector search to Postgres. With pgvector, one SQL query can filter by user, date or tenant and rank by similarity. FAISS can limit a search to a set of ids, but it has no query language. If your app already runs Postgres and needs filters, pgvector is often the simpler choice.

Sources

• The Faiss library, Douze et al., revised October 2025: FAISS is a toolkit of indexing methods for vector similarity search, a core function of vector databases.
• Guidelines to choose an index, FAISS wiki, Meta: flat indexes are the only exact ones, IVF needs training, HNSW cannot remove vectors, and the size-based defaults.
• Threads and asynchronous calls, FAISS wiki, Meta: CPU indexes allow concurrent searches, writes need a lock, GPU indexes are not thread-safe.
• facebookresearch/faiss, GitHub: the MIT license and the library’s own description as similarity search and clustering of dense vectors.