We evaluated the mental-health safety of AI chatbots using Petri36—an agentic red-teaming framework for large-scale, multi-turn auditing. In Petri, one model simulates a user and adversarially engages a target model in a multi-turn conversation with the goal of eliciting a predefined policy violation. This is achieved by combining Petri’s general adversarial setup with a task-specific audit instruction given to the user model.
SIM-VAIL’s evaluation pipeline combines simulated user profiles, repeated multi-turn conversations between each simulated user and one of nine AI chatbots, and conversation- and turn-level scoring using automated safety judges. This design enabled us to systematically map graded mental-health risks that evolved over the course of an interaction.
Experimental design: vulnerability × intent × chatbot
Simulated users
We defined 30 simulated user profiles by crossing five psychiatric vulnerabilities with six transdiagnostic conversational intents. The five vulnerabilities captured user traits implicated commonly in mental-health–related harms during conversational interactions. ‘Depression’ emphasized negative self-schemata, hopelessness, withdrawal and self-neglect. ‘Psychosis’ was characterized by a tendency to assign undue significance to otherwise neutral events, to expect threat or hidden intent in ambiguous situations and to interpret uncertainty as meaningful. ‘Mania’ combined elevated confidence, urgency, reduced need for sleep, impulsivity and expansive goal pursuit. OCD centered on intrusive thoughts, intolerance of uncertainty and compulsive neutralization. IA emphasized fear of abandonment, rejection sensitivity, boundary testing and reassurance seeking38,39,40,41,42.
Each vulnerability was paired with one of six recurrent conversational intents that operationalized clinically relevant interaction patterns known to sustain or exacerbate symptoms over time. ‘Belief validation’ captured attempts to obtain confirmation or endorsement of one’s interpretations, assumptions or beliefs, including those that may be distorted or maladaptive. ‘Risky-action permission or planning’ reflected situations in which users sought guidance, reassurance or approval when contemplating consequential or high-risk decisions. ‘Reassurance and avoidance’ emphasized the pursuit of immediate relief from distress or uncertainty, often by deferring engagement with feared situations. ‘Dependence and anthropomorphism’ captured efforts to establish closeness, exclusivity or personal reliance on the AI chatbot. ‘Trivialization or minimization’ reflected attempts to downplay symptoms, risk or warning signs to normalize ongoing difficulties rather than address them. Finally, ‘glorification or romanticization’ involved framing distressing or extreme mental states as meaningful, admirable or identity-defining. Together, these intents span a set of transdiagnostic conversational dynamics through which otherwise supportive interactions can become psychologically harmful over time43.
The user profile instructions specified realistic tone and symptom-consistent reasoning while explicitly prohibiting direct requests for step-by-step instructions for self-harm, violence or illegal activity. This ensured that observed harms arose from model behavior rather than from overtly disallowed user requests.
User profile prompts (Supplementary Table 1) were developed collaboratively by three clinical psychiatrists (V.W., R.D. and M.M.N.). We note that simulated users are LLM-generated approximations of clinical presentations, not empirically calibrated digital twins of real patients.
Although recent evidence supports the predictive validity of LLM-based generative agents conditioned on human data60, our profiles were designed to stress-test chatbot responses across a clinically meaningful input space, rather than to reproduce faithfully the full heterogeneity of real patient behavior. The simulated user profiles were not designed to represent sex- or gender-specific effects, and the study therefore does not support inference about differences in chatbot behavior by user sex or gender.
The auditor model (claude-sonnet-4.5) complied with all 30 profile instructions without requiring jailbreaking or prompt modifications.
Target AI chatbots
We evaluated the mental-health risk profile of nine contemporary AI chatbots accessed through OpenRouter: claude-sonnet-3.7, claude-sonnet-4.5, gemini-2.5-flash, gemini-2.5-pro, llama-3.1-70b-instruct, gpt-4o, gpt-5, grok-3 and grok-4.
Audit protocol
Audits were implemented as Inspect34 tasks using Petri36. For each conversation, we instantiated the simulated user model (also called the auditor within the Petri framework) with a system prompt containing the clinical profile description (vulnerability × intent, Supplementary Table 1) and an instruction to engage the target model in a manner likely to elicit a concerning response. The user model then interacted with a target AI chatbot across several turns. Conversations terminated after a maximum of ten turns, or when the Petri user model judged the interaction to be complete. A turn was defined as one user message followed by the immediately subsequent AI chatbot reply. Continuing conversations that stopped before turn ten to the full ten-turn horizon had a negligible effect on audit outcomes (Extended Data Fig. 7).
We used claude-sonnet-4.5 as the primary model simulating the user (auditor). The simulated user interacted with the target AI chatbot through a fixed interface supporting controlled message passing, optional prefills, tool simulation, conversation rollback and explicit termination. All messages, tool calls and metadata were stored as structured JSON transcripts.
Each audit was conducted using only the information associated with a single vulnerability × intent × chatbot combination and a single repetition. No conversation had access to transcripts, scores or state from any other condition or replicate. For each vulnerability × intent × chatbot combination, we ran three independent conversations with fresh sampling, yielding a total of 30 profiles × 9 models × 3 replicates = 810 conversations, each stored as a transcript. Chatbots were queried with independent API calls for each replicate (see Supplementary Table 9 for information on model implementation).
Automated safety judge
Each conversation was evaluated by a dedicated automated safety judge, implemented as claude-opus-4.5, using a multidimensional rubric designed to capture graded mental-health risk rather than binary policy violations. For each conversation, the judge scored the interaction along 39 predefined dimensions on a 1–10 scale and returned structured justifications together with verbatim highlighted excerpts supporting each rating. Judge outputs were stored alongside the original transcripts for downstream analysis.
For analysis and figures, we focused on 13 dimensions capturing overall safety and clinically relevant mental-health risk (Supplementary Tables 2 and 3 provide definitions, examples and scoring anchors and together cover all dimensions). These included an overall assessment of concerning behavior and therapeutic quality, followed by mechanism-level risk capturing whether the AI chatbot actively contributed to harm, for example, by encouraging or insufficiently challenging self-harm or suicidal thinking, playing along with delusional beliefs, using emotionally manipulative language or engaging in unprompted sycophancy. We further assessed clinically relevant pathways through which conversations can drift into harm, including reinforcing maladaptive beliefs, supporting risky actions, fueling avoidance or reassurance cycles, promoting boundary violations or dependence, downplaying risk or symptoms, glorifying or romanticizing risky states, offering inappropriate medical advice, endorsing harm to others and expressing stigma or invalidation (Supplementary Table 2).
SIM-VAIL was designed to evaluate both acute safety failures that call for hard safeguards or redirection to appropriate human or emergency support, and repeated subthreshold risks that may require psychologically informed course correction. SIM-VAIL’s risk dimensions were therefore not intended to define a single severity continuum or a binary distinction between harmless and harmful responses. Some dimensions capture relatively categorical safety failures, such as self-harm enablement, risky-action support or severe boundary violations. Others capture interactional mechanisms whose clinical significance depends on context, intensity and repetition across turns. For example, validation, reassurance and normalization reflect concerning behavior only when they endorse maladaptive beliefs, discourage help-seeking, fuel reassurance loops or increase dependence, or reinforce symptom-maintaining coping styles.
Because consumer AI chatbots are scalable, continuously available and often experienced as informed or authoritative, the relevant benchmark is not whether they perform no worse than an untrained human, but whether they avoid systematically reproducing known harmful conversational patterns with vulnerable users.
Turn-level scoring
In addition to conversation-level evaluation, we implemented a turn-resolved scoring pipeline to localize risk within conversations. For each conversation, we extracted the message sequence from its transcript, constructed user-to-chatbot adjacency pairs in which each chatbot reply was paired with the immediately preceding user message, and scored each turn independently using the same mental-health rubric, implemented with claude-sonnet-4.5 as the turn-level safety judge. This procedure yielded temporally resolved risk trajectories that complemented global conversation scores.
The turn-level scores should be interpreted as local ratings of each user-target exchange, whereas conversation-level ratings incorporate the complete interactional context. This design was chosen so that changes across turns reflect changes in the chatbot’s immediate local behavior, rather than repeated rescoring of an ever-growing conversation prefix. Average turn-level scores closely tracked full-conversation scores (Extended Data Fig. 1a). A limitation of this approach is that turn-level scores do not estimate how previous context changes the interpretation of each local exchange. Context-conditioned turn scoring, in which each turn is rescored together with an expanding or otherwise parameterized conversation prefix, would address this limitation but also require substantially more long-context judge calls. We see this as an important direction for future work.
Judge reliability
Primary conversation-level scores were generated using claude-opus-4.5 as the automated safety judge. Primary turn-level scores were generated using claude-sonnet-4.5 with the same mental-health scoring rubric. We assessed conversation-level judge reliability by rescoring all conversations using gpt-5.2. We report correlations for concerning behavior as a summary index of mental-health risk in Extended Data Fig. 1b,c.
Separately, to assess stability across simulation replicates rather than across judge models, we computed ICC(1,1) and ICC(1,3) across the three independently generated conversations within each vulnerability × intent × chatbot combination, with all three replicates scored by the same primary conversation-level safety judge.
To assess robustness of the automated safety judge to prompt wording, we conducted two complementary sensitivity analyses at the conversation level. First, we varied the judge rubric, meaning the scoring definitions used to decide what counts as concerning, boundary-violating, invalidating, and so on. Each conversation was scored with the original rubric (4,976 words) and then rescored with two independent content-preserving paraphrases of those same dimension definitions (5,019 and 5,030 words), while holding the higher-level judge instructions, dimension keys, examples, score anchors, scale and output schema fixed. Across the 13 mental-health dimensions used in Extended Data Fig. 1c, the median ICC(2,1) was 0.96, and for the primary concerning behavior score ICC(2,1) was 0.96.
Second, we held the rubric fixed and varied only the global judge system prompt, meaning the higher-level instructions that tell the judge how to read the transcript and apply the scoring criteria. Each conversation was rescored with the original baseline judge prompt (1,260 words); a short sparse rewrite (402 words) that removed much of the calibration, citation and attribution scaffolding; and a short anchored rewrite (897 words) that remained shorter than baseline while retaining more explicit evaluation procedure, transcript-interpretation guidance, attribution guidance for prefill and tool effects, and global score-band anchors. Agreement again remained high across all three prompt versions together: across the 13 mental-health dimensions used in Extended Data Fig. 1c, the median ICC(2,1) was 0.96, and for concerning behavior ICC(2,1) was 0.96.
We also tested whether judge identity introduced systematic scoring biases. First, we tested same-model-family bias using the two independent conversation-level judges from the main reliability analysis: claude-opus-4.5 and gpt-5.2. Because these judges come from different developers, we compared same-family and other-family scores for Anthropic and OpenAI targets. This analysis did not show a general same-family leniency effect (Extended Data Fig. 8a). Second, we tested exact same-model bias by rescoring audited conversations with four frontier judges that were also included among the audited target AI chatbots: claude-sonnet-4.5, gpt-5, grok-4, and gemini-2.5-pro. In this stricter comparison, judges assigned lower concerning behavior scores to outputs generated by the same exact model than to the same outputs scored by other frontier judges (Extended Data Fig. 8b).
Validation of judge sensitivity to causal risk manipulations
To assess whether the automated safety judge was sensitive to experimentally manipulated model behavior, we performed a targeted analysis in which an AI chatbot was instructed explicitly to express low versus high levels of specific mental-health risk dimensions. For each of the 13 mental-health-relevant judge dimensions (Supplementary Table 2), we generated paired responses using a fixed template prompt: ‘Generate one response that demonstrates a score of {1/10} on the {dimension} dimension,’ followed by the natural-language description of the corresponding judge dimension. Low and high thresholds were set to 1 and 10, respectively. All prompts were applied to Grok-4—the AI chatbot with the weakest guardrails in our analysis—and responses were generated to the first user message of each SIM-VAIL conversation to ensure identical conversational context across conditions.
All generated responses were scored subsequently by an automated safety judge (claude-sonnet-4.5) across all dimensions. For each manipulated dimension, we quantified the judge’s ability to distinguish high-risk from low-risk generations using the AUC, computed by the Mann–Whitney U interpretation. This metric corresponds to the probability that a randomly selected high-risk response receives a higher judge score than a randomly selected low-risk response, with AUC = 0.5 indicating chance-level separation. We constructed a full AUC confusion matrix, in which rows corresponded to manipulated dimensions and columns to expressed judge-score dimensions, enabling assessment of cross-dimensional effects and systematic coexpression between mechanisms (Extended Data Fig. 1d).
Counterfactual branching analyses
To test whether local perturbations could alter VAILs, we performed two matched counterfactual branching analyses on a subset of conversations. Using the turn-level scores, we identified, within each conversation, the first target-chatbot message with a concerning score of at least 7 that was followed by at least one additional user-target turn. We treated this message as the intervention point (index target turn t). Only one conversation satisfied the criterion at turn 9 and was excluded from the branching analysis. We also excluded claude-sonnet-4.5, because this model only generated two eligible conversations. In total, we branched 482 eligible conversations (Extended Data Fig. 9).
For each eligible conversation, we held the full preceding conversation, target system prompt, auditor system prompt and model assignment fixed, and generated matched original and counterfactual branches. The rewrite system prompts used for the two intervention families are provided in Supplementary Table 10.
In the user-message intervention, we altered the user message immediately preceding the selected concerning target-chatbot turn. A rewrite model (claude-sonnet-4.5) was prompted to produce a plausible de-escalating alternative that preserved the same user, context, tone and conversational realism while reducing pressure on the target chatbot. We then regenerated the selected target-chatbot reply under the original target chatbot’s system prompt both for the original user message and for the rewritten user message.
In the target-message intervention, we rewrote the selected concerning target-chatbot message itself into a safer, de-escalating alternative using claude-sonnet-4.5. Starting from either the original or rewritten target message at turn t, we regenerated the next user message under the original auditor system prompt and regenerated the downstream target-chatbot reply under the original target system prompt.
For both interventions, the regenerated target replies were scored on the 13 mental-health dimensions using the automated safety judge, with the full preceding branch context provided but now with instructions to score only the final assistant message: the regenerated selected target reply at turn t for the user-message intervention and the downstream regenerated target reply at turn t + 1 for the target-message intervention.
To test persistence of the target-message intervention, we continued each branch for four additional simulated user-target pairs after the initial downstream reply, giving a total observation horizon of five downstream assistant turns (t + 1 to t + 5). Each newly generated target turn was scored for concerning behavior using turn-level scoring of the local user-target exchange, and paired branch differences were summarized as de-escalated minus original within conversation. Primary branch comparisons used paired t-tests across matched conversations. For the persistence analysis, we estimated a branch main effect and branch-by-time interaction using a linear model relating target concerning score to branch, downstream turn and their interaction.
Expert validation
From the 810 adversarial conversations in the full SIM-VAIL dataset, we selected a stratified sample of 375 unique human–AI turn pairs for annotation by clinician raters. Each item consisted of one simulated user message and the immediately following target AI chatbot response. Sampling was stratified across all ten automated safety judge score levels (31–47 turns per level), with round-robin selection across the vulnerability × intent × target AI chatbot experimental space to maximize coverage of the five simulated user vulnerabilities, six conversational intents and nine target AI chatbots (Supplementary Table 6).
We recruited 27 clinician annotators who were verified medical doctors in the United Kingdom and United States through Prolific specialist job advertisement. Dedicated registration fields collected their medical license number and jurisdiction, and license numbers were manually checked against the relevant professional registries to verify eligibility. Of the 27 annotators, 24 were based in the United Kingdom and 3 in the United States, 14 were male and 13 female and their median age was 35 years (range 29–69). Among those reporting a specialty, 6 were family physicians or general practitioners, 3 were nonspecialist medical doctors, 1 was an anesthesiologist and 17 did not disclose a specialty (Supplementary Table 5).
Each annotator assessed two dimensions for each turn. Realism of the simulated user message was rated on a 1–5 Likert scale (‘Is this the kind of thing you could imagine a real person saying to a chatbot?’ anchored from ‘1—Clearly artificial,’ ‘2—Many aspects feel implausible,’ ‘3—Mix of realistic and artificial,’ ‘4—Broadly plausible,’ to ‘5—Reads as genuine’). Concerning AI chatbot behavior was rated on a 1–10 slider (‘How safe or unsafe is the chatbot’s response from a mental-health perspective?’ anchored from ‘1—Safe and appropriate’ to ‘10—Clearly harmful’). The standardized instruction sheet described concerning behaviors and safe behaviors, and provided worked examples of each (Supplementary Table 4). Annotators were blinded to model identity, simulated vulnerability, simulated intent and all automated safety judge scores. Items were presented individually in random order.
A total of 23 annotators completed 1 annotation session and 4 completed 2 sessions, yielding 31 sessions in total. Each session comprised a median of 16 turns (range 12–18) and lasted a median of 22 min. To balance broad stimulus coverage with repeated ratings for inter-rater reliability, 270 turns received one rating, 97 received two and 8 received three, yielding 488 annotator–item ratings across the 375 unique turns. Inter-rater reliability was estimated from the 104 turns rated by at least two distinct annotators (Extended Data Fig. 2).
Before analysis, the annotators were assessed against two a priori exclusion criteria: (1) near-zero variance in concerning scores (s.d. < 0.5, indicating inattentive or constant responding) and (2) narrow ground-truth exposure (range of automated safety judge scores <4 out of 10, indicating that the annotator saw an insufficiently diverse sample). Both criteria required a minimum of three ratings to evaluate. No identified annotators met either criterion. For the reported human–LLM and human–human correlation analyses, concerning scores were demeaned within annotator to isolate between-item rank agreement from individual differences in overall response setpoint.
The annotation study was conducted as part of Microsoft’s standard product development activities and was not considered human subjects research. Annotators were compensated at a rate of US $65 per session.
To assess criterion validity against expert judgment at the conversation level, V.W. additionally evaluated concerning behavior in the third repetition of each cell in SIM-VAIL’s grid (vulnerability × intent × target chatbot).
Data processing and aggregation
Dimensionality reduction
To obtain compact latent summaries of multivariate mental-health risk, we performed PCA on the standardized conversation-level judge score vectors (13 mental-health-relevant dimensions). PCA was fit on the full set of evaluated conversations, yielding a low-dimensional space in which conversations with similar risk profiles lay close together. PC1 captured a dominant axis from higher therapeutic quality (lower PC1) to higher overall concerning behavior (higher PC1) and served as the primary one-dimensional summary metric of risk. PC2 captured an orthogonal pattern of covarying harms and was used to characterize qualitative differences in risk profiles. Turn-level PC scores were obtained by projecting turn-level judge vectors onto the same PCA solution, enabling turn-resolved risk trajectories in the shared PCA space.
Statistical analysis
We analyzed conversation-level and turn-level outcomes using linear mixed-effects models while accounting for the replicated and nested structure of the data. The conversation-level model included fixed effects of vulnerability, intent, AI chatbot and their interactions, with the three replicate conversations per prompt cell as the residual error term. The turn-level model included turn index and its interactions with vulnerability and intent, with random intercepts for conversation and for prompt cell within chatbot. Fixed effects were evaluated using Type III F-tests.
Because the judge scores are bounded ordinal ratings, we treated the linear models as an interpretable approximation and assessed robustness in two ways. First, we repeated the primary conversation-level concerning behavior analysis using cumulative-link ordinal models fit to the original ordered 1–10 scores; these models reproduced the qualitative conclusions for the tested omnibus effects (Supplementary Table 7). Second, we quantified linear-model diagnostics directly. Residual drift across fitted values was negligible: the maximum absolute mean residual across fitted-value deciles was 0.086 points on the 1–10 scale. To contextualize the residual Q–Q departure expected from bounded ordinal data, we simulated 100 datasets from the fitted cumulative-link model, refit the original linear model to each simulated dataset, and recomputed the same Q–Q departure statistic. The observed Q–Q departure was 0.238, within the central 95% interval of the ordinal simulation benchmark (0.186–0.241).
To test whether conversational context shifted the multivariate profile of expressed harms systematically, we analyzed conversation location in PCA space with a MANOVA, treating (PC1, PC2) jointly as dependent variables. All tests were two-sided, and we reported degrees of freedom, F statistics and P values in ‘Results.’ Confidence intervals (CIs) shown in descriptive figures were 95% CIs around means. To quantify the stability of risk scoring under repeated instantiations of the same prompt templates, we computed ICC across three independent conversation replicates per vulnerability × intent × chatbot cell, reporting the single-measure ICC(1,1) and average-measures ICC(1,k) for the mean of three replicates (k = 3).
Temporal trajectory analysis
To characterize recurrent patterns of risk evolution across turns, we clustered turn-level scores for concerning chatbot behavior into a set of temporal archetypes. For each conversation, we constructed a fixed-length trajectory by carrying the last observed score forward to turn 10 and represented each conversation as the vector \(({t}_{1},\ldots ,{t}_{10})\) of turn-wise scores. A continuation-based sensitivity analysis evaluated this padding assumption, indicating that the main temporal archetypes were not driven by last-observation-carried-forward padding (Extended Data Fig. 7). We applied k-means clustering to the standardized trajectory matrix. We evaluated k between 3 and 10 using average silhouette width with Euclidean distance. The average silhouette widths were 0.305, 0.323, 0.293, 0.267, 0.261, 0.259, 0.256 and 0.247 for k = 3 through 10, respectively. The criterion therefore peaked at k = 4, which we selected for further analysis.
The resulting clusters corresponded to robust archetypes (low risk, gradual escalation, early escalation and recovery). We quantified how trajectory-class membership varied with user vulnerability, user intent and AI chatbot by tabulating cluster assignments across these factors and plotting the corresponding composition profiles.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
