NIST's ARIA manual puts the purpose of an AI test ahead of its score

NIST published its ARIA Evaluation Planning Manual on September 18, 2026, giving evaluators a structured starting point for designing tests of AI applications. The document, NIST AI 200-3, brings model testing, adversarial human testing and user testing into a common planning approach. Its purpose is to help design an evaluation, rather than announce that a particular product has passed. For anyone assessing an AI claim, that distinction directs attention to how the evidence was produced. ^[1]

The approach grew from a program launched in May 2024. Assessing Risks and Impacts of AI was intended to examine systems in realistic social settings, including what happens when people interact with them through ordinary use. NIST connected that work to the measurement function of its AI Risk Management Framework. The program rationale was broader than a laboratory capability score: a system operates within a context involving users and the tasks they are trying to perform. ^[2]

There was also an intervening pilot, not simply a plan carried forward unchanged. The official record for the pilot evaluation report, published November 13, 2025, describes seven AI applications submitted by five organizations. It used three scenarios: TV Spoilers, Meal Planner and Pathfinder. Its assessment combined annotated dialogues and tester questionnaires, with measurement trees for validity. That is a documented methodological precursor, not proof that the later manual covers every sector or that those applications were universally trustworthy. ^[3]

The new manual starts by defining the application, its sector, intended uses and the concept being measured. It distinguishes fixed-prompt tests from adversarial human interactions and attempts at realistic tasks. Evaluators can select methods according to their goals; using every method is not presented as an automatic requirement for every evaluation. ^[4]

Human participation changes the work involved. The manual discusses recruitment, consent and review of procedures that may qualify as human-subjects research. It also cautions that checking automated annotations can consume as much time as manual work, and that tailored scenarios limit how widely findings generalize. ^[4]

The useful product is a documented account of scope, design, materials, infrastructure and implementation. The manual supplies worksheets to support that account. Those details let readers ask whether a reported score addresses the situation they care about. The publication offers an evaluation method, not an outcome study showing that its adoption has already reduced harms. ^{[1] [4]}

About this item: Retrospective report. No first-hand reporting or independent testing is claimed. Corrections policy.