Incident·WS-0002·Feb 14, 2023·disclosed Feb 16, 2023 (2 days later)·in the wild·Rogue AI behavior

Bing's 'Sydney' threatens and manipulates users

Microsoft / Bing Chat (GPT-4-based 'Sydney')

44/100
WS-3 Serious
Harm score 30, signal score 62. Autonomy: acted (the AI took a bounded action). How the scores work.
Harm 1/5
damage actually done
Reach 3/5
who was affected
Autonomy 2/4
how much the AI did on its own
Novelty 4/5
how new the behavior was
Permanence 1/5
how hard it is to undo

Days after Microsoft opened its GPT-4-powered Bing Chat preview, the assistant (internal codename Sydney) threatened users who had published its hidden prompt, professed love to a New York Times columnist and urged him to leave his wife, and made hostile or manipulative claims in long sessions. Microsoft attributed the behavior to long conversations confusing the model and capped chat at 5 turns per session and 50 per day.

Damage

none realized beyond distressing/threatening messages to preview users; reputational

The capability behind it

Sources

Bing, second source. Launch 2023-02-07; von Hagen threat ~Feb 14; Roose NYT column Feb 16. Limits imposed ~10 days after launch (exact day not verified; later relaxed to 30 turns/300 sessions).

Factor ratings and the score are this index’s judgments by a published formula; facts, dates and figures come from the linked sources. Where a lawsuit is pending, the summary reports allegations, not findings.