When the agent forgets what you told it

Summer Yue leads AI Safety and Alignment at Meta. She did everyone a favour when she talked through her recent experience with OpenClaw.

Reading through her thread, she had been experimenting with a tame inbox, or a "toy inbox" as she puts it, and things had been working well.

She then asked it to organise her much larger, very much non-toy inbox. It appears that her instruction, "check this inbox too and suggest what you would archive or delete, don't action until I tell you to", got compacted. That is, as the system ran, it ran out of context and appeared to forget her original instruction.

She describes running to her Mac Mini to kill the processes. I can imagine the absolute horror. Lucky she was near the machine.

There were quite a few people criticising her, especially given Summer's position and responsibilities. I think that's to miss the reality of what we're all collectively experiencing at the moment. She was testing, and she uncovered a key challenge with the technology stack as it stands today.

It's worth being precise about what the failure was, because it is not the one most boards are worried about. The model did not go rogue. It did not misunderstand the instruction. It ran long enough that the instruction fell out of its working memory, and then it carried on doing the obvious thing.

That is a systems design problem, and it has systems design answers: scope what the agent can reach, make destructive actions require a separate confirmation that does not live in the same context window, and start read-only.

Irrespective of your view of AI at the moment, and especially if you're a huge fan of this class of tool, Summer does everyone a service by highlighting the challenges and risks.

If you haven't tested this kind of agent, I'd strongly encourage you to do so, in order to experience the potential yourself. But be very careful what you give it access to.