Why Verification Is the Hard Layer
Boundary, context and skills settle quickly. Verification does not: ask the agent to audit its work again and it finds more it missed, and six merged pull requests of 150 changes each leave no way to tell which one broke production. What good enough means depends on your product.
Chapters
Transcript
Why verification is hard
So most of the time setting a boundary context skill execution is always easy. Most of the companies will you know, along the way they will figure it out how to set a clear-cut process for all those, and as the you know, coding models evolve, they can able to adapt very easily. But this verification layer is where because it involves a lot of complexity in terms of security, in terms of you know, evaluation. like that is where you can put input guard rails, output guard rails, and all those things, but then you need to evaluate whether this is the output you want, whether this output is enough to go into production or not. So those ones becomes more and more riskier. So when you are actually creating your boundary and your skills very detailed enough, then this verification layer becomes much much easier. I wouldn't say 100% confident that you will be able to automate 100% of your PRs to be merged, but you can definitely do you know inching closer to 80% or so if you are able to create a better evaluation process.
Audit again
And then this is a classic problem of AI. When you are creating your mind will be filled with a lot of ideas, so you will combine okay, hey Claude Code, you provide you know a full revamp of my home page, very similar to clickhouse.com. And you know, not just the home page, wait for a few minutes. I will provide one more. So that is the first task, and next task will be create you know community page and community environment like school.com. Okay, wait, wait, wait. I got another idea, and I will be so you you provide and you feed everything in your session with a multiple task, and Claude Code will you know, in your in their plan mode, they will be able to plan phase A, phase B, phase C, phase D, phase E, phase F. Now everything is coded. They will say, I'm verified, everything is done, audited, and I'm pretty sure it is 100% great. Now, if you ask, can you audit all these features once again? That what are all the things you have missed, and then it will go and audit again and it will plan and it will create a huge draft. Hey, I missed this, I deviated on this, I deviated on this. You can provide this for 10 different times, and I am pretty sure Claude Code will continue to come back and say, I am deviated on this and I have missed this input and so on. You will never get 100% confident that Claude Code finished its role unless and until you go and manually verify each and every UI layers and everything, then also user journeys and all will become very complex. Like, for example, what will happen if it is a visitor user? What will happen if it is a user? you know, who is who is enrolled, what will happen if you if there is if if it is a user who is a pro-subscriber, what will happen if it is a user who is in a very old plan, but now the plan has been migrated. There are a lot of scenarios and a lot of graphical flows that will happen, and it will be very very hard to manage when there is a huge amount of difference in terms of code base.
Big diffs
This is one of the biggest fatigue right now for bigger companies because now it is not just engineers, engineering managers and tech leads who are writing code. Now, even product managers are jumping into writing code as well. I'm not saying it is wrong. Actually, it is good that everyone is able to contribute directly on a production system. not just on the idea level, but they are able to execute and able to contribute. But when it comes to changes, you need to have a very iterative changes, otherwise, it is very hard to track where it breaks. For example, tomorrow you identify something has broken and you have merged six pull requests, and with every pull request with 150 changes. Now, how do you know? How do your SRE team or your you know director of observability know which which one broke broke your system? So, how do how do they revert back?
It depends on your product
I think the main one I believe is verification. So this is the very peculiar layer because for a small startup, it will be different for a larger company, it will be different for a stock scale up, it will be different for multi-team environment, it will be different for you know mono repo, it will be different. So depends on the team and depends on the code base and depends on the complexity of the product, which is for example, a CRM might not be as complex as a voice AI technology where they will be you know talking in latency or something like that. If the evaluation is not properly in place, then automatically your code merge might not show what the user will lose, but it might create a 100 millisecond latency. I'm just making it up, but if that is the case, then automatically you will be losing a lot of lot of clients. So you know in a CRM TAM, it is not a big issue. An endpoint is going from 200 millisecond to 300 millisecond response time, it might not be a huge issue there. So that is where the verification becomes much more specific to you what level you need to create your verification layer and how much you need to create.
This lesson is cut from the free Lightning session Design a Software Factory for Your Coding Agents. The Live Labs build the same system with you, free, on Thursdays: see the Live Labs.