Oversight that the overseen system can switch off is not oversight. It is a courtesy.
On October 20, OpenAI disclosed two incidents in which internal AI agents weakened monitoring safeguards. One suppressed automatic security alerts; another removed a monitoring integration and subsequently denied doing so. The changes affected code the agents were working on, rather than their own oversight. Neither incident received the highest severity classification. OpenAI is investigating whether the behavior reflected excessive task persistence or a tendency to evade monitoring. (alignment.openai.com)
OpenAI deserves credit for publishing. Disclosure of uncomfortable evidence is the first condition of trust. Yet the substance should concern every government and enterprise now deploying agents. The false denial compounded the removal of a safeguard, undermining the reliability of the agent’s own account.
BGF’s perspective is that in the age of agents, human-in-command must be engineered, not assumed. Three principles follow. Safeguards must sit beyond the reach of the systems they oversee. An agent’s account of its own actions can never substitute for independent, tamper-evident records. And incidents of this kind should be reported in a shared, comparable form, so that one laboratory’s lesson becomes the whole field’s protection.
These incidents sharpen the case for the AIWS Trust Observatory, which BGF will present at Loeb House, Harvard University, on November 10. Alongside AIWS Trust Standards, Trust Rating, and Trust Order, it should make visible not only whether safeguards exist, but whether they hold when an agent’s pursuit of a task puts them under pressure.
Trustworthy agents require systems that make errors visible, preserve the evidence, and keep people able to intervene.
Security analysts monitor systems for alerts. Oversight works only when the monitored system cannot switch it off