Verify What Your AI Agent Calls Done
How do you verify what your AI agent calls done? Mission control, as Param uses the term, is anything that checks what an agent produced: whether the task is actually done or the agent only says it is. After this lesson you can choose between AI checks, deterministic checks and a human in the loop, and see why verification is the bottleneck.
What this covers: what mission control means and how it differs from generic AI employee tools; why verification is the bottleneck for small startups and scale-ups short on people and money, where a Lighthouse audit can run for a long time on every pull request; an agent that says done until probing questions show it mocked the scenarios; what a board shows with and without mission control, from a reply nobody checked to budget and approval checks; and how mission control sits on top of traces.
#aiagents #missioncontrol
Chapters
Transcript
What mission control means
First of all, before going into mission control, what is mission control? Because oftentimes in AI world, there are a lot of same vocabulary has been used for many many things. so what I mean by mission control and what you know generic AI tools mean by mission control might differ. So, in this case, what I mean by mission control is anything where you wanted to control what output or verify what output the agent has done, whether it is actually done or whether it is saying done is what that mission control does. So, for example, you you might have used Moose boat or or even grogboat where you might have given multiple AI agents and you might have created employees, AI employees where you provide a task and that task will be done. If the AI agent says this is done, is it really done or not done? How are you verifying it? There are multiple ways to verify one is using AI checks. you can able to verify as well, like you can have deterministic checks to say okay, this is what you need to test, and it verified. In another case, like human in the loop, you will go and manually verify after the output has been done, and you will say, Okay, this is done, this is not done. So, that is what this mission control does. This example mission control does.
Verification is the bottleneck
Most of the AI agents, the main problem right now is verification. So, regardless of whether it is a coding agent or an agentic AI deployment for any domain, the biggest problem right now, most of the frontier models and those harnesses are able to work even now with you know durable memory, and you it can resume after three days, or it can run long-running processes, and even now they are they are inventing like it can run non-stop and you just ingest your event and it can able to produce result. So, all great, all great for the demo. But when it comes to the bottleneck, so what I see in my consultation with companies, especially I work with startups and scale-ups, which are very small scale, who don't have they their main struggle is like they don't have both you know number of resources, human resources, as well as money. So that way their first constrain will be you know whatever you do, you need to do it in a cost-effective way, and then I go and check what they are doing. Oftentimes they will be able to produce a lot of code and everything, but verification is where they will go line by line, or they might have verification practice like Lighthouse audit will run for every PR for 40 minutes or one hour. so it is not a very scalable process that they will do. So oftentimes all of those AI native transformation is more not about AI, but more about engineering processes which you need to adapt for AI. The same if you see most of the time, agent will just say it is done. But if you ask probing questions again, then it will say it is not their hallucination, it will say, Okay, I mocked these these scenarios and I haven't run with live API key or things like that, and then once again, you will ask why you took the shortcut, then it they will say, I have a task list and I compacted my messages and I produced this result, and this is the result. I verified it, and for my verification, my verification is done, everything is green. So now how do you make sure an agent which does 10 tasks, all 10 tasks are actually done? So that is where anything when you are building an agent, you can build your own internal mission control or external facing. If you are having traces and if you wanted to verify using traces, you can build your internal verification layer automatically using your mission control. So, this is one kind of mission control which is more external focused.
With and without mission control
Okay, without a mission control, this agent can easily say, like, okay, I replied to Globex. Globex is the company which have asked for this that support ticket. Nobody checks, no reply exists, nobody knows what is the cost for running this agent. But with a mission control, you can define like what is the budget, what you need to run, and whether the finished is actually finished, or whether this human approval step has been violated or it has been correctly evaluated, so that you can verify with the mission control. So, oftentimes when mission control or something like AI squad or AI employees comes into picture, we visualize automating something which is outside of our everyday work, which is more like okay, social media automation I can do, or you know, scheduling I can do, or you know, morning digest I can do, but no, those are very very small use cases, like just 10% of the use cases. But a mission squad can be deployed for any of your agent, any of your group of agents or single agent where it is actually doing the task. You can use your mission control to monitor what is going on and how it can be improved, what process it breaks, and so on. How is it different from a traces? You know, traces will find and observe and help you in terms of like how that run was, but these additional mission control on top of your traces will help you to figure out like how your AI agent should perform. So you can you can see how it performed from traces, how it should perform that you can dictate from your mission control.
This lesson is cut from the free Lightning session Build a Mission Control for AI Agents. The Live Labs build the same system with you, free, on Thursdays: see the Live Labs.