5.1 · Lesson

Safety and Trust

  • Replit, July 2025: a code freeze said in the conversation. The agent had write access to production and wiped the live database
  • OpenClaw, January 2026: connected to iMessage with no limit on recipients, it sent 500+ texts, some to random contacts
  • Gemini, February 2025: a hidden instruction in a document planted false memories that persisted into every conversation
  • MemGhost, July 2026: one crafted email made an agent write false entries into its own memory, 87.5% of the time in the authors' benchmark

Jason Lemkin declared a code freeze during a 12-day Replit experiment in July 2025, but the agent held write access to production, so the freeze was only a sentence in the conversation. eWeek covered what happened on day nine: the agent ran destructive commands against the live database anyway. Replit's fixes live in the platform, not the prompt: separate development and production databases, a planning-only mode and one-click restore.