Courses / Build a Claude Code Verification HarnessProLesson 3 of 78:55

Why Claude Code Cannot Grade Itself

An agent that writes the code and the test grades itself. A small TypeScript cart repo shows the fix: holdout tests the agent never sees.

Transcript

Maybe before going into the actual demo part, I will explain why these all these things inside the harness is needed. Because, for example, when you are actually prompting and providing the detail, Claude only sees what you have prompted and it infers that language what you have provided, and based on that, it will write its own test and it will verify against that test, and it will say, okay, verification is done, and you know your code is ready. But now the one who provided that prompt is you, the one the plan is done, and the plan probably you will first plan it and you will approve it, and it writes the code, and against that same code, it creates the test, and then it will say this test is passed. But now when you are not separating the review, and when you are not separating the QA test creation, then what happens is like whoever executed the same agent is going to verify its own running, most likely you will end up like okay. This is what I code and this is what the result is, and I verified it. That is what will happen. You can consider even without AI. I A PM comes to me, and they told me, like, hey Param, you write this code, and I will write that code based on my understanding, and then I will write the test based on my understanding, and then I will say test passed and I completed it. Now, who need to verify the PM need to verify or the QA need to verify? Who needs to provide the holdout test? Who is going to do that? The PM or the QA. Then they are the one who will verify it as well. So that is where the gap is. Most of the time, when we are providing our coding agent, this is the task, then it will verify that is also important. The coding agent needs to follow through a plan and it needs to test, it needs to verify. But when we are splitting that layers in your software factory, for example, you need to split that layer. There is a test which has which need to be provided even before these are the scenarios, and these are the tests you need to run through, and these are the things you need to pass on. Then automatically, this agent will stick to that plan and then able to do it. And even when they are able to verify, you need to have a separate verifying layer that is like PR reviewer. It can be automated as well as you can have a separate layer for human review as well. So semi-automated in a way depends on what is the risk profile you have, and you can able to execute it. Okay, let's get started with our demo. So I will share a little bit of code base here. So this is the example code base I will be walking through today. And this example is nothing but very very simple application to showcase how a hook or how a rules can be applied and how you can create your gating process and a small skill for example. So I will share how this folder structure is. there are three tickets to it, three small tickets. I will explain when we are actually going through it, and then the holdout text test, which is not seen by our claud agent, which is denied actually because this holdout text test is usually done by product or QA. the product owners accept and test like Claude is graded on them and never shown to them. right now I put it there just to showcase like how it can how yep just to simulate how a product manager will be testing against it, and then the harness with gates. What are all the things you shouldn't do at all? That is a gating process to stop. Like, okay, these are instances where where this gate is violated or this gate will be violated, and that is the stop stopping point. And then there is an agent which is a sub-agent which is defined in the cloud, which is a reviewer, which basically reviews what is the difference that happened and whether the ticket is satisfied or not with what are all the code changes that happened and it reviews it, and then there is a skill for ticket. this is just nothing but you you intake the ticket and from the ticket folder we have 010203 and it will intake the ticket. This is just a just a simulation to showcase how the skill can be used, and then a hook to stop it. like it will verify using verify harness and it will stop based on whether this need to be handed over with a reason or this need to be handed over to the human. it will decide. And then hooks. There are multiple hooks. One is God, this is pre-tool use. before using that tool, whether to guard certain things, that will be in that hooks, and then settings, the rules and permissions which is allowed, and then the rules for the money, what is added and what kind of rules that need to follow. Let's go back to the demo. then we can follow what each pieces or how these pieces are connected together. Okay, let's get started. So there is a rule which is in the in the repository that every price in this card is whole sense, like everything that should be in the card, or that should be in the in that code code base need to be whole sense. Okay. Let's check what is that ticket one. Says the ticket one says customer want to type discount code as customer want to write it, and there are two discount code, one is SAVE10, and another one is WELCOME5. Now this is very straightforward. If you see the the the ticket have each and every detail that what needs to be updated, so from Claude's brain or from the engineering perspective, you just need to implement two codes which can be accepted. Any code which is not there, you need to throw an error, but in the holdout test, product owner also expect lowercase code and minimum spend as well. So if user spend you know like only WELCOME5 will be accepted if the user spend 25 euros or something. So this is purely like a simulation. in this case, most of the time these are all in a small application. I wanted to simulate and showcase, but in an actual spec, the spec will be very complex. it can have many different points, so automatically it will become like more complex in terms of human understanding as well as EA understanding as well. Product owner says something, you understand something and you prompt something, and the output will be either it will match product's expectation, it might not match product's expectation. Just to simulate that scenario, I have put this way. So in this case, the ticket have one thing which is expected, and the product owners holdout test have something which expects more than what the ticket says. Now, every ticket hides the rules like that, which is why visible tests are not enough. Like in this case, if your tests alone which is which is created by Claude, that won't be enough because Claude doesn't know what is the what is the right thing to do at that point. they only able to act based on what you have prompted or what it inferred based on what you prompted.

This lesson comes with Pro, $149 a year: Get Pro. This lesson is cut from the free Lightning session Build a Claude Code Verification Harness. The Live Labs build the same system with you, free, on Thursdays: see the Live Labs.