Risk management frameworks and regulation

The earlier units treated a detector as a technical object with a true positive rate, a false positive rate and an alert budget. An organization that deploys one also has to answer questions that no metric answers by itself. Who owns the system and its failures? Whom do its errors affect? How is it tested before and during use, and what happens when it degrades? Which laws apply to it? This lesson introduces the documents that structure those questions as of October 2026: the NIST AI Risk Management Framework and its profile for generative AI, NIST's taxonomy of adversarial machine learning, the management system standard ISO/IEC 42001:2023 and the European Union's AI Act. It then works through what they mean for a security detector.

Risk as probability and consequence

The NIST AI Risk Management Framework defines risk as the composite measure of an event's probability of occurring and the magnitude of its consequences (Tabassi, 2023). A simple way to make that operational for a detector is to list the events that matter, estimate how often each happens and what each costs, and add the products,

where \(p_i\) is the expected number of events of type \(i\) per period and \(c_i\) their cost in a common unit. This is the expected-cost reasoning of Lesson 1.2, applied to an organization rather than to a single decision.

Consider a detector that locks accounts after suspicious sign-ins, at a service with 20,000 sign-ins a day of which 0.1%, or 20, are account-takeover attempts. At a threshold with a true positive rate of 0.9 and a false positive rate of 0.5%, it locks 18 attackers out and misses 2, and it wrongly locks 99.9 legitimate users a day, about 25 hours of help-desk and user time at 15 minutes each. If a missed takeover costs 400 times as much as a wrongful lockout, \eqref{eq:risk} gives \(R = 99.9 \times 1 + 2 \times 400\), or 899.9 units a day. A stricter threshold with a true positive rate of 0.8 and a false positive rate of 0.1% gives \(R = 19.98 + 4 \times 400\), or 1,619.98 units, so cutting false alerts by a factor of five nearly doubles the expected cost. Two features of the calculation recur throughout this unit. Part of the cost of a false positive falls on a specific person, whose account is locked, and not only on the organization in general. And the numbers that drive the decision, the base rate and the two costs, are estimates that someone has to own and revisit.

The NIST AI Risk Management Framework

NIST published the AI Risk Management Framework (AI RMF 1.0) as NIST AI 100-1 in January 2023 (Tabassi, 2023). It describes itself as voluntary, rights-preserving, non-sector-specific and use-case agnostic, so it applies to an intrusion detector as much as to a hiring tool. It names the characteristics of trustworthy AI systems as valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. Validity and reliability is drawn as the base on which the others rest, and NIST calls it a necessary condition of trustworthiness, so a detector that does not work as intended is not made trustworthy by being explainable.

The framework's core organizes risk management into four functions, shown in Figure 11.1. Govern cultivates a culture of risk management and puts in place the policies, roles, accountability and processes for the whole lifecycle; NIST describes it as a cross-cutting function that informs and is infused throughout the other three. Map establishes the context in which risks arise: the system's purpose, the people and processes it affects, and its dependencies. Measure uses quantitative, qualitative or mixed methods to analyze, assess, benchmark and monitor risks and impacts, and NIST states that AI systems should be tested before deployment and regularly while in operation. Manage allocates resources to the mapped and measured risks and plans how to respond to, recover from and communicate about incidents. The framework states that the actions in its core do not constitute a checklist and are not necessarily an ordered set of steps; after governance is in place, most users start with Map and continue to Measure or Manage, iterating as needed.

The framework is general by design. NIST adapts it to particular settings through profiles. NIST AI 600-1, published in July 2024, is a cross-sectoral profile of the AI RMF for generative AI (National Institute of Standards and Technology, 2024a). It lists twelve risks that generative AI creates or exacerbates, among them confabulation, information security, and value chain and component integration (Lesson 10.3 lists them all), and suggests actions for each function of the core. For adversarial threats specifically, NIST AI 100-2e2025, published in March 2025, gives a taxonomy and terminology of attacks and mitigations (Vassilev et al., 2025). It classifies attacks on predictive AI into evasion, poisoning and privacy attacks, and attacks on generative AI into poisoning, direct prompting and indirect prompt injection, and it discusses the limitations of existing mitigations. In AI RMF terms, the taxonomy is a tool for the Map function (what can an adversary do to this system?) and the Measure function (which of those attacks has been tested?).

ISO/IEC 42001:2023

ISO/IEC 42001:2023, published in December 2023 and developed by ISO/IEC JTC 1/SC 42, the artificial intelligence subcommittee of the joint technical committee of ISO and IEC, specifies requirements for establishing, implementing, maintaining and continually improving an AI management system within an organization (International Organization for Standardization and International Electrotechnical Commission, 2023). A management system is the set of policies, objectives and processes through which an organization controls an activity. The standard is meant for organizations that provide or use AI-based products or services. Where the AI RMF offers voluntary guidance that an organization adapts, ISO/IEC 42001 states requirements, which makes it a reference against which an organization's processes can be assessed. The two fit together. An organization can use the AI RMF's functions and NIST's profiles to decide what to do about the risks of a particular system, and the management system to show that it does so consistently for all its systems.

The EU AI Act

The European Union's Regulation (EU) 2024/1689, the Artificial Intelligence Act, was published in the Official Journal on 12 July 2024 (European Parliament and Council of the European Union, 2024). Unlike the documents above, it is law, binding on providers and deployers of AI systems placed on the market or used in the Union. It takes a risk-based approach that the European Commission summarizes in four levels (European Commission, 2026).

Unacceptable risk. Article 5 prohibits certain practices outright, among them manipulative techniques that materially distort behavior, social scoring, predicting that a person will commit a crime based solely on profiling or personality traits, untargeted scraping of facial images to build recognition databases, and inferring emotions in the workplace and in education except for medical or safety reasons.

High risk. Article 6 makes an AI system high-risk if it is a safety component of a product covered by the Union legislation in Annex I, or is itself such a product, and that product needs third-party conformity assessment, or if it falls into a use case listed in Annex III, which includes biometrics, critical infrastructure, education, employment and workers' management, access to essential private and public services, law enforcement, migration and the administration of justice. Under the derogation of Article 6(3), an Annex III system escapes the classification if it poses no significant risk of harm, for example because it performs only a narrow procedural or preparatory task, but an Annex III system that profiles natural persons is always high-risk. High-risk systems must meet requirements on risk management, data governance, technical documentation, record-keeping, transparency to deployers and human oversight, and Article 15 requires an appropriate level of accuracy, robustness and cybersecurity, including measures against data poisoning, model poisoning, adversarial examples and confidentiality attacks.

Transparency risk. Systems that interact with people or generate content carry disclosure obligations, for example telling people that they are talking to a machine and marking synthetic content.

Minimal or no risk. All other systems fall into this category, which according to the Commission covers the vast majority of AI systems in use and includes spam filters. The Act introduces no specific rules for these systems, although the AI literacy duty of Article 4 applies to providers and deployers of all AI systems (European Parliament and Council of the European Union, 2024; European Parliament and Council of the European Union, 2026).

Providers of general-purpose AI models, such as the language models of Unit 10, have separate obligations on transparency and copyright, and providers of models that may pose systemic risks must also assess and mitigate those risks.

The Act applies in stages. Article 113 of the original text set the dates, and Regulation (EU) 2026/1744 of 8 July 2026 (the Digital Omnibus on AI), published in the Official Journal on 24 July 2026, amended some of them (European Parliament and Council of the European Union, 2026). The European Commission's policy page, last updated on 3 August 2026, summarizes the result (European Commission, 2026). The dates in the table below were checked against both regulations and that page on 15 October 2026, and Figure 11.2 draws them.

The table lists when obligations start, not when an organization should start preparing. A provider of a high-risk system needs its risk management process, data governance and technical documentation in place, and tested, before the application date.

What this means for a security detector

The Act classifies AI systems by their use, so the same model can fall in different categories depending on what it decides and about whom. Its text addresses several security uses directly. Recital 55 states that components intended to be used solely for cybersecurity purposes should not qualify as safety components of critical infrastructure, so an intrusion detector protecting a utility's network is not high-risk merely because the network is critical infrastructure (European Parliament and Council of the European Union, 2024). Annex III lists AI systems that evaluate creditworthiness as high-risk, with the explicit exception of systems used to detect financial fraud, and recital 42 notes that the prohibition on predicting crime by profiling does not touch risk analytics that assess the likelihood of financial fraud by undertakings on the basis of suspicious transactions. Remote biometric identification is high-risk, but Annex III excludes biometric verification whose sole purpose is to confirm that a person is who they claim to be, and recital 54 adds that biometric systems used solely to enable cybersecurity and personal data protection measures should not be considered high-risk.

Other security uses move in the opposite direction. Insider-threat monitoring that evaluates employees' behavior resembles the Annex III use case of AI systems that monitor and evaluate the performance and behavior of persons in work-related relationships, and if it profiles individuals the derogation of Article 6(3) cannot apply. An insider-threat tool that infers employees' emotions from video or voice would fall under the workplace prohibition of Article 5. Classifying a real system requires the full facts and legal advice; the point here is that a security team cannot assume its tools are exempt because their purpose is protective.

For most network and malware detectors, the Act imposes no specific obligations, and the frameworks are what structure good practice. The four functions of the AI RMF translate into concrete work that the earlier units developed, as the table below shows for an intrusion detector.

Summary

The AI RMF treats risk as probability combined with consequence and organizes its management into Govern, Map, Measure and Manage, around trustworthiness characteristics of which validity and reliability come first. NIST AI 600-1 profiles the framework for generative AI and NIST AI 100-2e2025 names the adversarial attacks to map and measure, while ISO/IEC 42001:2023 states requirements for the management system that applies such practices across an organization. The EU AI Act is law and classifies AI systems by use, prohibiting a few practices, imposing requirements on high-risk systems and transparency duties on some others, with application dates staged from February 2025 to August 2028 after the 2026 amendment. Most network and malware detectors fall outside its high-risk categories, while systems that monitor employees or identify people can fall inside, and in every case the quantities that the earlier units taught you to measure are the evidence that the frameworks ask for.