The vision call

A vision LLM call is similar to a text call, with one key difference: instead of sending just a string prompt, you send a multimodal content array that includes both text and an image. The image is base64-encoded and sent inline with your extraction instructions.

The vision extraction flow

How an invoice image becomes structured data.

invoice_utils.py
python
import base64
import mimetypes

class InvoiceUtils:
    """Utilities for processing invoice files (images and PDFs)"""

    @staticmethod
    def encode_image_to_base64(file_path: str) -> str:
        """Convert an image or PDF file to base64 string."""
        with open(file_path, "rb") as image_file:
            return base64.b64encode(image_file.read()).decode('utf-8')

    @staticmethod
    def get_mime_type(file_path: str) -> str:
        """Get the mime type of a file."""
        mime_type, _ = mimetypes.guess_type(file_path)
        return mime_type or "application/octet-stream"

Before you can send an image to a vision LLM, you need to convert it to a base64 string and determine its MIME type. The MIME type tells the model whether it is looking at a PNG, JPEG, or PDF.

invoice_analyzer.py
python
# Build multimodal content for the vision LLM
content = [
    {"type": "text", "text": prompt},
    {
        "type": "image_url",
        "image_url": {
            "url": f"data:{mime_type};base64,{base64_image}"
        }
    }
]

# Send to the vision provider
response_text = await self.llm_provider.generate_text(content)

The content array holds both the text prompt and the image. The image is embedded as a data URL with the base64 content. This is the OpenAI-compatible format that most providers support.

Yes, base64 increases the payload size by about 33%. Most vision APIs accept images up to 20MB after encoding. For invoices, this is rarely a problem since scanned documents are typically under 5MB. If you hit size limits, resize the image before encoding. The model does not need 4K resolution to read text on an invoice.

Quiz: Quiz

Loading practice…