Confidence scoring

Here is the problem with raw extraction: the model always returns a value, even when it is guessing. A blurry invoice number might be "INV-1234" or "INV-1284". Without confidence scores, your downstream system treats both with equal certainty. Confidence scoring lets the model tell you how sure it is about each field.

invoice_analyzer.py
python
class FieldConfidence(BaseModel):
    """Confidence score for an extracted field"""
    value: float = Field(
        description="Confidence score between 0.0 and 1.0"
    )
    reason: Optional[str] = Field(
        None,
        description="Why the confidence is low, if applicable"
    )

class InvoiceDataWithConfidence(BaseModel):
    """Invoice data with per-field confidence scores"""
    is_invoice: bool
    vendor_name: Optional[str] = None
    invoice_number: Optional[str] = None
    invoice_date: Optional[str] = None
    due_date: Optional[str] = None
    items: List[InvoiceItem] = Field(default_factory=list)
    subtotal: Optional[float] = None
    tax_amount: Optional[float] = None
    total_amount: Optional[float] = None
    currency: Optional[str] = "USD"
    confidence: dict[str, FieldConfidence] = Field(
        default_factory=dict,
        description="Per-field confidence scores"
    )

The confidence dictionary maps field names to scores between 0.0 and 1.0. A score of 0.95 means the model is very sure. A score of 0.4 with a reason like "text partially obscured" tells your system to flag this for human review.

invoice_analyzer.py
python
# Extended prompt with confidence scoring instructions
confidence_prompt = """
You are an expert invoice processing agent.

Extract all fields from this invoice AND rate your confidence
for each field on a scale of 0.0 to 1.0.

Confidence guidelines:
- 0.9-1.0: Clearly visible, unambiguous text
- 0.7-0.8: Readable but slightly unclear
- 0.5-0.6: Partially obscured or ambiguous
- Below 0.5: Guessing based on context

If confidence is below 0.7, include a reason explaining why.

Return JSON with all invoice fields plus a "confidence" object
mapping field names to {value, reason}.
"""

The confidence guidelines give the model a calibrated scale. Without them, models tend to report 0.9+ confidence on everything. Explicit ranges with descriptions produce more honest scores.

Confidence-based routing

How confidence scores drive downstream decisions.

Not perfectly, but it is useful as a signal. Vision LLMs are reasonably well-calibrated for document extraction: they know when text is blurry, when a field is missing, and when they are inferring from context. The confidence score is not a ground truth probability. Think of it as a triage signal that separates "definitely right" from "probably needs a human look." In production, you calibrate the thresholds by comparing against a labeled test set.

Quiz: Quiz

Loading practice…