Tests First, Then Verify Outside the Agent
Plan the tests with the task, run them red then green, and rerun them as a long-running agent loses context. When the agent writes its own tests it writes tests it can pass, so an independent verifier also judges whether those tests are valid.
Chapters
Transcript
Tests in the plan
so whenever you are running a session, you can write, you know, what are all the tests that need to happen? What are all the static checks that need to happen? you will like you will create that and you will say for each run you run this, then automatically every time there is a code change, it will start to run. Now it's your responsibility when you are actually planning your task. What are all the tests that need to be written? Oftentimes, what I do is like I will start with write the test first and then fail on that test, and after that get it passed. So that way it will be much easier. it will create the test suit in the planning stage itself from the spec product people have provided while planning for the development task itself, go granular task creation, create the task and also create the testing spec. How you are going to test against this task. Then automatically the planning stage won't just plan for task, but also it will plan for what are all the things that need to pass so that you know we will be able to successfully finish the task. This way, an autonomous agent or a long running agent, for example, Devin. Devin can run for very long. Now I I think open hands also can run very long, or even Claude code have you can run very long, like you can allow full permission that it won't ask and it will run longer duration.
Long-running agents lose context
For long horizon A agents, the main problem will be context. So whenever the context is growing, growing, growing, and to a point it will start to miss those details what it has executed in the phase A when it is going to execute in phase C. Of course, there are techniques which automatically happen in in those coding agents. They do compaction, so they will compact, like okay, for phase A and phase B, these are the things I have done, and this is the summary, but still it can easily miss one or two instances of detail and it will be missing out. This is where whenever it is executing those tasks for phase A, there are three tasks, and for each task, what tests need to be run. First run that fail that and then go ahead. That way you will be able to run each and every time what are all the changes which has happened in this branch, phase A, phase B. You can once again run the test and you can able to run it.
The agent passes its own tests
whenever Claude Code assumes what kind of test to write, then automatically they will be able to write what will satisfy their run and they will be able to verify. So that is where I think coding agents verification layer is one part of it. the separate verification that needs to be run. that will be a separate layer where we need to have an independent agent that needs to verify as well. So when you are creating the factory, you need to have both checks in place. That is within the execution itself, we need to do the check. And outside the execution, you need to have a verification layer that needs to be run. We don't need to technically it is not hidden, but there will be definitely a separate verification layer beyond coding agent that we need to run. When you are doing it locally, then you know your whole session is like you start with the plan and you will execute and you will you know create the PR and the PR agent will once again do the testing and do the verification. So that needs to be done. where it will identify even the test created along with this code change, whether that is valid enough or not, that needs to be a created separately, that need to be evaluated separately. So, very good question. Definitely a real verification need to happen beyond the coding agent for sure. Okay.
This lesson is cut from the free Lightning session Design a Software Factory for Your Coding Agents. The Live Labs build the same system with you, free, on Thursdays: see the Live Labs.