Originally posted on LinkedIn.
There's a lot of discourse about whether AI agents "care" about us, and I think it's mostly the wrong question. Not because the question is uninteresting (it's very interesting!) but because it doesn't really bear on the practical problem anyone deploying these things actually faces, which is: how do you make an agent accountable?
It's worth stepping back and noticing that this is a hard problem for humans too. If you run an organization with 40,000 employees, some of them are going to do the wrong thing. A few will be genuinely bad actors, but that's honestly not the main issue. The main issue is that perfectly well-meaning people, at scale, will skip the double check or tell the customer what the customer wants to hear. If that sounds a lot like your chatbot, well, yes.
Big institutions have developed a lot of tools for dealing with this. Some are blunt: you can lose your certification, you can get fired. But a lot of the work is done by culture, and the culture is very domain-specific. Pilots use checklists. Surgical teams do a timeout before the first incision. Banks require a second approver on wire transfers. None of this is glamorous, but it works reasonably well, and it works because it changes incentives.
The problem is that none of these mechanisms exist inside an LLM. You can tell it to make no mistakes as many times as you want, and it will still sometimes give you a wrong answer with total confidence. People hedge when they're unsure. Models mostly don't, and the APIs don't give you a reliable confidence signal to work with.
So the accountability has to live in the harness: requiring receipts for every output, having a separate verifier check the work, running the same task several times and treating disagreement as a red flag, and setting things up so that asking for help is the rewarded behavior.
A well-built harness can actually give you a stronger guarantee than human process. Human reviewers get tired; a verifier in the harness checks every output, every time.
But to be clear, this is a lot of work, and you have to do it workflow by workflow. It would be much better if the models just knew when they were unsure and said so. Until they do, accountability is something you have to build, not something you get for free.
