Data processing & anonymization

Two related patterns work together in data pipelines: Data Processing cleans, validates, and chunks text for RAG ingestion. Anonymization detects and masks PII (emails, SSNs, names) while preserving data utility, which is essential for GDPR compliance and privacy.

Data pipeline: processing + anonymization

If your knowledge base contains names, emails, or SSNs, the LLM might include them in responses to other users. Anonymizing PII before ingestion protects privacy and helps with GDPR compliance.

Anonymization pattern

patterns/31_data_processing.py + 33_anonymization.py
python
# Data Processing (Pattern 31)
class DataProcessor:
    def clean_text(self, text):
        """Remove extra whitespace, special chars, normalize."""
        text = re.sub(r'\s+', ' ', text).strip()
        return text

    def chunk_content(self, content, chunk_size=500, overlap=50):
        """Split into overlapping chunks for RAG."""
        words = content.split()
        chunks = []
        for i in range(0, len(words), chunk_size - overlap):
            chunk = ' '.join(words[i:i + chunk_size])
            chunks.append(chunk)
        return chunks

# Anonymization (Pattern 33)
class DataAnonymizer:
    def detect_pii(self, text):
        patterns = {
            "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
            "phone": r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
            "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
        }
        detected = {}
        for pii_type, pattern in patterns.items():
            matches = re.findall(pattern, text)
            if matches:
                detected[pii_type] = matches
        return detected

    def anonymize_text(self, text):
        pii = self.detect_pii(text)
        anonymized = text
        for email in pii.get("email", []):
            anonymized = anonymized.replace(email, "[EMAIL]")
        for phone in pii.get("phone", []):
            anonymized = anonymized.replace(phone, "[PHONE]")
        return anonymized

DataProcessor for cleaning/chunking and DataAnonymizer for PII detection/masking.

Quiz: Quiz

Loading practice…

Matching exercise: Match data pipeline concepts

Loading practice…

Flashcards: Flashcards

Loading practice…

You can now clean, chunk, and anonymize data for production RAG pipelines. These data processing patterns are essential for any real-world deployment.

Checkpoint: RAG pipeline checkpoint

Loading practice…