Entity and relationship extraction

The transformer does not read your entire document at once. It works on chunks, because the LLM has a context window and extraction quality drops as the chunk grows. We split first with RecursiveCharacterTextSplitter, run the transformer on each chunk, and merge the results into one graph.

ingestion.py
python
class IngestionPipeline:
    def __init__(self, graph_store, chunk_size=1500, chunk_overlap=150):
        self.graph_store = graph_store
        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap,
            separators=["\n\n", "\n", ". ", " ", ""],
        )

    def _to_documents(self, text, source):
        metadata = {"source": source} if source else {}
        return [
            Document(page_content=chunk, metadata=metadata)
            for chunk in self.splitter.split_text(text)
        ]

    def ingest_text(self, text, source=None):
        """Extract entities + relationships and persist them to Neo4j."""
        docs = self._to_documents(text, source)
        if not docs:
            return {"nodes_created": 0, "relationships_created": 0}

        transformer = _build_transformer()
        graph_documents = transformer.convert_to_graph_documents(docs)
        return self.graph_store.add_graph_documents(graph_documents, include_source=True)

Chunk size of 1500 characters with 150 overlap is a sane default for extraction. Bigger chunks lose triples at the edges, smaller chunks lose cross-sentence relationships. The overlap helps the transformer catch facts that straddle a split point.

AI prompt: Try it: mimic the extractor prompt

Loading practice…

Ordering exercise: Order the ingestion pipeline

Loading practice…

Quiz: Quiz

Loading practice…