Teaching AI to see

Multimodal means AI that can process multiple input types: text, images, audio, and sometimes video. A "vision model" is a multimodal model that understands images alongside text.

Imagine you're building an app for restaurant owners. They take a photo of a dish and the AI writes an appetizing menu description. Or they photograph a receipt and the AI extracts all the line items. This is multimodal AI in action.

How multimodal input flows through a vision model

Images and text are combined in the messages array and processed together by a vision-enabled model.

Not all models support vision. You need a vision-enabled model like: - Google Gemini (gemini-2.0-flash), works great with a free tier - OpenAI GPT-4o, excellent vision capabilities - Anthropic Claude, strong at document analysis The API is the same completion() call, you just include images in the messages alongside text.

Matching exercise: Match input types to API capabilities

Loading practice…

How vision models process images

Images must be base64-encoded and embedded in the messages array.

The API uses JSON for communication, and JSON is text-only. Base64 encodes binary image data as a text string so it can travel inside a JSON message. It makes the data about 33% larger, but it is the standard way to embed images in API calls. Some providers also support sending a URL instead, but base64 works everywhere.

Base64 is a way to encode binary data (like images) as plain text. Since APIs communicate using JSON (which is text-only), images must be converted to base64 strings before they can be sent. The PIL library (Python Imaging Library) handles loading and resizing images, while the base64 module does the encoding.

02-multimodal.ipynb
python
import base64, io
from PIL import Image

def image_to_base64(image: Image.Image) -> str:
    """Convert PIL Image to base64 string."""
    buffered = io.BytesIO()
    image.save(buffered, format="PNG")
    img_bytes = buffered.getvalue()
    return base64.b64encode(img_bytes).decode('utf-8')

First, we need a helper to convert images to base64 strings. This is the format that vision APIs expect.

With the base64 helper ready, we can build our main analyze_image() function. It combines text prompts with one or more images into a single API call.

02-multimodal.ipynb
python
def analyze_image(
    prompt: str,
    images: list,
    system_message: str = None,
    temperature: float = 0.7,
    max_tokens: int = 300
) -> str:
    """Analyze images with a text prompt."""
    # Build content array with text + images
    content = [{"type": "text", "text": prompt}]

    for img in images:
        base64_image = image_to_base64(img)
        content.append({
            "type": "image_url",
            "image_url": {
                "url": f"data:image/png;base64,{base64_image}"
            }
        })

    messages = []
    if system_message:
        messages.append({"role": "system", "content": system_message})
    messages.append({"role": "user", "content": content})

    response = completion(
        model=DEFAULT_MODEL,
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens
    )
    return response.choices[0].message.content

The key difference from text generation: the content field is now an array containing both text and image_url objects.

Fill in the blanks: Complete the vision API call

Loading practice…

02-multimodal.ipynb
python
# Analyze a food photo
food_image = Image.open("sample_food.png")

result = analyze_image(
    "Describe this dish. What ingredients can you see?",
    [food_image]
)
print(result)
# Output: "This appears to be a Buddha bowl with quinoa,
# roasted sweet potatoes, avocado, chickpeas, and a
# tahini dressing. The greens look like kale..."

Pass any PIL Image to analyze_image() and ask questions about it. The AI describes what it sees in natural language.

AI prompt: Try it with AI

Loading practice…

Quiz: Quiz

Loading practice…

Flashcards: Flashcards

Loading practice…

You can now send images to AI and get intelligent descriptions back. Next, we will use this same capability for practical tasks like reading text from images and analyzing charts.