Prompt optimization and a/b testing

You can score prompts now. The next step is optimization: automatically generating prompt variants and using your evaluation pipeline to pick the best one. This combines meta-prompting (AI writes prompts) with A/B testing (data picks the winner).

AI prompt: Try it with AI

Loading practice…

14_prompt_optimization.py
python
# Full optimization loop: generate prompt variants and evaluate
def optimize_prompt(base_task, test_cases, n_variants=3):
    # Step 1: Use meta-prompting to generate variants
    meta_system = "You are an expert Prompt Engineer."
    variants = []
    for i in range(n_variants):
        variant = get_completion(
            f"Generate an optimized prompt for: {base_task}\n"
            f"Variant {i+1}: try a different approach.",
            system_prompt=meta_system,
            temperature=0.8,
        )
        variants.append(variant)

Meta-prompting asks an LLM to write prompts for you. Higher temperature (0.8) produces diverse variants. Each variant tries a different approach to the same task.

Now that we have generated diverse prompt variants, the next step is to evaluate each one against the same test cases and pick the winner. This is the selection phase, essentially automated A/B testing for prompts.

14_prompt_optimization.py
python
    # Step 2: Evaluate each variant against test cases
    results = []
    for v in variants:
        score = evaluate_prompt(v, test_cases)
        results.append({"prompt": v, "score": score})

    # Step 3: Return the best-performing variant
    best = max(results, key=lambda r: r["score"])
    return best

Each variant is scored against the same test cases. max() with a key function selects the highest-scoring prompt. This is automated prompt engineering where the LLM both writes and evaluates prompts.

Prompt optimization pipeline

Ordering exercise: Evaluation pipeline steps

Loading practice…

Checkpoint: Checkpoint: agentic prompting

Loading practice…

Timed quiz: Speed round: agentic prompting

Loading practice…

Validation checklist: Prompt optimization checklist

Loading practice…