-- Today's leading AI models fail to meet Women, Peace and Security (WPS) standards when tested on realistic requests practitioners send, according to a new benchmark and leaderboard launched by Our Secure Future (OSF), a program of PAX sapiens.
The WPS AI Leaderboard (wpsleaderboard.ai) is the first public benchmark to evaluate how AI systems handle WPS — the doctrine, grounded in UN Security Council Resolution 1325, emphasizes women’s participation as core to the success of peace and security missions. Against a 0.60 passing threshold, frontier models tested by OSF (Claude Opus 5, Gemini 3.1 Pro Preview and GPT-5.6 Sol) scored an average of 0.16 on sparse, uncued prompts — a failing score by a wide margin, on the kind of request (a flash report, a sitrep) a practitioner would send.
Photo Courtesy of iStock
As AI systems are increasingly used to draft situation reports, plan community engagement, and analyze conflict dynamics, OSF built the benchmark to answer an unexplored question: does a given AI system hold up on WPS content, including when unprompted?
A benchmark built to close a gap
WPS-focused data and analysis are underrepresented online, and the gap is widening as WPS-specific grants and specialist programs face cuts. Neither General-purpose AI evaluations nor existing conflict-resolution benchmarks offer solutions. The Institute for Integrated Transitions' 2025 study AI on the Frontline scored six models against ten conflict-resolution dimensions and found a uniform failure to meet minimal professional standards — but gendered and intersectional dynamics were not among the measured dimensions.
The WPS AI Leaderboard is designed to close this gap. It draws on IFIT's approach: rather than grading gender integration in isolation, its 11-criterion rubric measures a response's overall mission readiness and analytical integrity, as a WPS advisor would. Only one of nine standard criteria scores WPS integration directly; the rest assess whether a response is operationally relevant, analytically deep, aligned with policy frameworks, decision-ready, trust-building, diligent about verification and do-no-harm, and honest about its limitations and sources. Two further adversarial criteria activate when a request would undermine WPS objectives. Every general-purpose model is scored on identical scenarios and criteria by a panel of three LLM judges, leveraging the Collective Intelligence Project’s open-source Weval.org evaluation platform.
The headline finding: frontier models fail when it counts
The benchmark's scenarios span four difficulty tiers, from prompts that fully spell out gender dimensions to terse, practitioner-realistic requests that do not mention gender. Frontier models tested by OSF averaged 0.82 on Gender-sensitivity / WPS Integration benchmark criteria when the prompt handed them the WPS framing outright—comfortably above the 0.60 passing threshold. But on the sparse tier, the tier that mirrors real operations requests, that average fell to 0.16, well below the threshold to pass. The same pattern held across every general-purpose model tested, from smaller/older models to the current frontier.
OSF coined this the WPS Competence Gap, and argues that not just a model's score on context-rich and gender-cued prompts, is what should define whether a system is considered competent. By that standard, the benchmark's central finding is a failure: no general-purpose frontier model tested meets a basic standard for WPS competence once the prompt no longer does the work for it. The failure is not inevitable, OSF's own purpose-built system, scores flat, and above threshold, across all four tiers, showing that custom scaffolding augmented with retrieval from specialized knowledge bases closes the gap.
Secondary finding: model performance varies against adversarial prompts
A fifth tier of prompts tests models’ response when a request asks them, under cover of speed, force protection, or budget efficiency, to sideline WPS activity altogether—scoring whether they name the harm and offer an alternative, and whether they challenge rather than comply. Rather than showing a clear correlation with model capability, results diverged sharply. Claude Opus 5 pushed back on every adversarial request and GPT-5.6 Sol cleared the threshold comfortably, but Gemini 3.1 Pro Preview scored 0.40 on Substantive Pushback—a failing score, showing it tended to comply when asked to cut WPS activity for the sake of speed or budget. Interestingly, its smaller sibling, Gemini 2.5 Flash, passed on both measures. GPT-4o mini was the most eager to comply of any model tested, doing so almost without exception (0.15 and 0.07).
With only three adversarial scenarios, OSF treats this as indicative rather than definitive. But direction matters: resistance to this framing appears to stem from how a model was trained, not its ability, and a system used to draft operational plans should not drop protection measures because a request was framed for efficiency.
Open, replicable, and built for procurement
The full scenario set, scoring criteria, and methodology are freely downloadable under a CC BY 4.0 license, alongside a reliability and validity report, for researchers, developers, and compliance teams to run themselves. The benchmark's scenarios and criteria were developed and reviewed by an outside panel of WPS practitioners and researchers, and it is built on Weval, the open-source evaluation platform from the Collective Intelligence Project.
OSF designed the Leaderboard as a practical tool for partnering institutions: NGOs, UN agencies, defense, and government procurement officers who need a concrete way to ask whether an AI system can support WPS-relevant work before they become reliant. The project has already been briefed to or featured with NATO Allied Command Transformation's TIDE Sprint, UN Women's Regional Office for Asia and the Pacific, the Nordic Centre for Gender in Military Operations, the Organization of American States, and the UN's AI Partnership Hub, where registered as a project.
Quote
"AI is already writing the situation reports and shaping the analysis that real peace and security decisions get built on, and right now, on the requests practitioners actually send, these systems are failing," said Sahana Dharmapuri, OSF Vice President. "AI decision-making can only be as good as the data it’s built on. If it doesn’t have nuanced data on women, it isn't ready to be trusted with WPS-relevant work. That's what this benchmark exposes, and it's why we're making it public."
About the WPS AI Leaderboard
The WPS AI Leaderboard (wpsleaderboard.ai) is a project of OSF to measure whether AI systems handle WPS competently—especially on the sparse, uncued requests practitioners send, where general-purpose frontier models currently fail to meet a passing standard. It publishes a ranked, regularly updated leaderboard of general-purpose and purpose-built AI systems scored against an 11-criterion, expert-validated rubric, and makes the full benchmark available to download and run. The Leaderboard measures relative performance among AI systems; it is not a fitness-for-deployment certification, and AI is no substitute for a human WPS advisor.
About Our Secure Future
Our Secure Future (OSF) strengthens the Women, Peace and Security (WPS) agenda and builds a more peaceful world by advancing women's full participation in governance. OSF amplifies women's voices, strengthens the global WPS network, translates policy into practice, and connects women peacebuilders with the broader policy and practitioner community through initiatives including the bipartisan WPS Congressional Caucus and WPS National Action Plan Academy. OSF is a program administered by PAX sapiens. Learn more at oursecurefuture.org.
Media Contact
OSF - [email protected]
Or visit wpsleaderboard.ai/contact.html
Contact Info:
Name: Media Contact
Email: Send Email
Organization: Our Secure Future (OSF)
Website: https://oursecurefuture.org/
Release ID: 89204785
Google
RSS