Ship an AI agent like an API with timeouts and traces
Ship an AI agent like an API. Frameworks like LangGraph, Google ADK or the Claude SDK get you started, but you still own how the agent runs. That means timeouts, tracing, monitoring and evals. Learn to set a timeout per agent, read traces by hand, weigh token cost against quality and rerun evals on every release.
What you will learn: - Pick a framework and still own how each agent runs - Give every agent a timeout, tracing and monitoring - Run evals where your traces show risk - Weigh token cost against quality, and rerun evals on every release
#aiagents #llmevals #observability
Chapters
- Pick a framework, but own the run
- Give every agent a timeout
- Run evals where your traces show risk
- Weigh token cost against quality
- Rerun evals on every release
Transcript
Pick a framework, but own the run
So most of the time when you are building an agentic AI system, oftentimes what you see in demo demos and prototypes or in examples in GitHub or more around what can be built with a framework like LangGraph or or with Google ADK, those are really good. again, like they they abstracted a lot of complexity within you know building agentic loop and so on. so those are really really good. Most of those SDK, for example, I am a big fan of Claude SDK, and they they provide a very good you know, you can build your harness, a solid harness on top of it very easily. So those SDKs are good, but then you are the one who is going to build that harness as well as you are the one who is going to build that system, how this is going to be invoked and how long this is going to be invoked, whether it is a long-running process, whether it is a short-running process, what is the timeout we need to give. We have done this before as well.
Give every agent a timeout
Like in an API, do you ever say hey, my API will you know run for 15 minutes? No, usually you will provide a timeout, like okay, 60 seconds. Probably that is the longer period for maybe for uploads and those kind of endpoints, and for remaining of endpoints, it will be one second, two seconds based on your latency. You will define the same thing you need to define for your AI agents as well. When it is running in the cloud, it got stuck and it is calling the API again and again and again, it is looping now without a proper tracing or without a proper observability. How do you know there is 10 different agents which is running in the cloud which gone rogue without a monitoring layer, you can't keep up. It's applied for a normal API, it will be applied for any agentic API as well.
Run evals where your traces show risk
So these are all the places in your trace, you need to do evals, like you need to read your traces. Like Hamel Husain, who is right now I saw yesterday. Mainly, if you read his blog post, mainly he says, you know, as a human, we need to start reading our traces. The main thing you will start to understand what are all the traces you are missing. That is one thing. Most of the time, when you are not logging it, then you don't know what is the system which is running. When you start to read it, then you will understand the system, what is going on, where the token is getting wasted.
Weigh token cost against quality
Token economics is very important. Like when you are you need to balance fast and quality at the same time, then automatically you will be able to achieve the end result in a in both fast iteration as well as with quality. So, how to balance that? You need to start reading those traces, and you need to evaluate like what is going on, what needs to be tweaked the next time. So these are the learning parameter, like okay, I land from this experience. So next time I if I tweak these, these, these and I will put this in my golden set. So let's evaluate again with a rerun in non-production, and I will be able to get okay, it is improving.
Rerun evals on every release
So every release, you are not releasing your AI agent with you know, okay, I I don't have a metric, but I feel this AI agent the next release is going to be good. Then you are not basically improving, you don't know because when in an API system, in a deterministic API, you can easily say, hey, last time my latency is 0.6 seconds, and now I have tested with my locust and I was hitting with you know this machine a thousand times and my concurrency with the same concurrency I was able to achieve 0.6. Unless and until such deterministic numbers metrics you are actually capturing and you are measuring, then only you will know whether it is actually improving or not improving. So that's that's the that's the way like building first time is very easy, but beyond that, every iteration improving the system further will become more and more complex.
This lesson comes with Pro, $149 a year: Get Pro. This lesson is cut from the free Lightning session Build a Multi-Agent System with MCP. The Live Labs build the same system with you, free, on Thursdays: see the Live Labs.