Unit 11 exercises

There are 200 frauds and 99,800 legitimate transactions a day. At \(t_1\), false positives are \(0.01 \times 99{,}800 = 998\) and missed frauds \(0.15 \times 200 = 30\), so \(R_1 = 2 \times 998 + 150 \times 30\), which is \(1{,}996 + 4{,}500 = 6{,}496\) units. At \(t_2\), false positives are \(0.002 \times 99{,}800 = 199.6\) and missed frauds \(0.30 \times 200 = 60\), so \(R_2 = 399.2 + 9{,}000 = 9{,}399.2\) units. Threshold \(t_1\) is cheaper in expectation. The organization's figure hides who pays. Nearly a thousand legitimate customers a day have a payment declined at \(t_1\), against about two hundred at \(t_2\), while missed frauds cost the issuer (and, until refunded, the victims). A risk register in the Map function of the AI RMF should list both, and the Measure function should check whether the declined customers are spread evenly across customer groups (Lesson 11.2).

(a) Minimal risk; the Commission's description of the risk categories names spam filters as an example of minimal-risk systems, for which the Act introduces no specific rules beyond the AI literacy duty of Article 4. (b) Most likely not high-risk; Annex III point 2 covers safety components in critical infrastructure, but recital 55 states that components intended to be used solely for cybersecurity purposes should not qualify as safety components. (c) Not high-risk under Annex III point 5(b), which lists creditworthiness evaluation but explicitly excepts systems used to detect financial fraud; recital 42 also keeps fraud risk analytics outside the profiling prohibition. (d) Not high-risk as remote biometric identification, because Annex III point 1(a) excludes biometric verification whose sole purpose is to confirm that a person is who they claim to be. (e) Likely high-risk under Annex III point 4(b), which covers systems that monitor and evaluate the behavior of persons in work-related relationships, and the Article 6(3) derogation cannot apply because the tool profiles individuals; the high-risk requirements apply from 2 December 2027. (f) Prohibited under Article 5(1)(f), which bans AI systems that infer the emotions of natural persons in the workplace except for medical or safety reasons (recital 18 lists anger among the emotions concerned); this prohibition has applied since 2 February 2025. Each classification depends on the facts of the deployment and needs legal review; the exercise shows that the same detection techniques fall into different categories by use.

The number of positives is \(\pi N\) and of negatives \((1 - \pi)N\), so \(TP = (1 - \mathrm{FNR})\pi N\). From the definition of precision, \(FP = TP(1 - \mathrm{PPV})/\mathrm{PPV}\). Dividing by the number of negatives, \(FP + TN = (1 - \pi)N\), gives

Every step uses only the confusion matrix of the group, which is why the identity holds for any binary classifier with nonzero precision, and why equal PPV and FNR force unequal FPR when \(\pi\) differs, except for the extreme classifiers of Lesson 11.2 whose precision is 1 or whose true positive rate is 0.

(a) For group a, true alerts are \(0.7 \times 0.01 = 0.007\), false alerts \(0.02 \times 0.99 = 0.0198\), alert rate 0.0268 and precision \(0.007/0.0268 = 0.261\). For group b, true alerts are 0.028, false alerts \(0.02 \times 0.96 = 0.0192\), alert rate 0.0472 and precision 0.593. (b) Precision of 0.261 in group b requires false alerts of \(0.028(1/0.261 - 1) = 0.0792\) of all cases, so \(\mathrm{FPR}_b = 0.0792/0.96 = 0.0825\), over four times group a's rate. (c) The innocent members of group b, who would be flagged at 8.25% instead of 2% so that an alert means the same thing in both groups. This is Chouldechova's identity at work, and it shows why predictive parity is rarely the right target when false alerts lead to actions against people.

(a) The point estimate at site B is higher, but the intervals overlap and each rests on a few dozen alerts, so the audit should report the difference with its uncertainty rather than claim that the sites are treated differently. It should also report the true positive rates, which differ in the same direction, and the precision, which differs mainly because of the base rates. (b) Collect more test days, which also allows resampling whole days (a better sampling unit than single connections, since connections on the same day share conditions), and report per-site rates for each week as they accumulate. A pre-registered target, such as a maximum ratio of false positive rates between sites, turns the monitoring into a test. (c) Thresholds equalized on past data did not keep the rates equal, because the benign traffic changed between periods (the authorized scanner ran at site A only in the training period). The audit plan therefore needs monitoring of per-site rates after deployment, with control limits such as those of Lesson 11.3, and a rule for when thresholds are refitted.

import numpy as np


def group_rates(y, alert, group, day, reps=1000, seed=0):
    """Per-group TPR and FPR with 95% intervals from resampling days."""
    rng = np.random.default_rng(seed)
    days = np.unique(day)
    out = {}
    for g in np.unique(group):
        m = group == g
        yg, ag, dg = y[m], alert[m], day[m]

        def rates(idx):
            yy, aa = yg[idx], ag[idx]
            tpr = aa[yy == 1].mean() if (yy == 1).any() else np.nan
            fpr = aa[yy == 0].mean() if (yy == 0).any() else np.nan
            return tpr, fpr

        point = rates(np.arange(len(yg)))
        by_day = {d: np.flatnonzero(dg == d) for d in days}
        boot = []
        for _ in range(reps):
            pick = rng.choice(days, len(days), replace=True)
            idx = np.concatenate([by_day[d] for d in pick])
            boot.append(rates(idx))
        boot = np.array(boot)
        lo = np.nanpercentile(boot, 2.5, axis=0)
        hi = np.nanpercentile(boot, 97.5, axis=0)
        out[g] = {"tpr": (point[0], lo[0], hi[0]),
                  "fpr": (point[1], lo[1], hi[1])}
    return out


rng = np.random.default_rng(1)
n = 4000
group = rng.choice(["A", "B"], n)
day = rng.integers(10, 14, n)
y = (rng.random(n) < np.where(group == "A", 0.02, 0.05)).astype(int)
alert = np.where(y == 1, rng.random(n) < 0.6,
                 rng.random(n) < 0.01).astype(int)
for g, r in group_rates(y, alert, group, day).items():
    print(g, {k: tuple(round(float(x), 3) for x in v)
              for k, v in r.items()})

Connections on the same day share conditions (the same attack campaign, the same scanner run, the same configuration change), so they are not independent. Resampling connections treats them as independent and understates the variation of a rate from one period to the next. Resampling days keeps each day's connections together, so the intervals reflect day-to-day variation, at the cost of being crude when there are few days; with four days, every resample is drawn from only four clusters, and the intervals should be read as indicative. The function returns nan for a rate whose denominator is empty in a resample, and the percentiles ignore those draws, which a report should mention.

The standard deviation of the weekly rate is \(\sqrt{0.004 \times 0.996/12{,}000}\), which is 0.000576, so the upper control limit is \(0.004 + 3 \times 0.000576 = 0.00573\), or about 68.7 false alerts a week, against an expected 48. Weeks with 52 and 61 are within the limit; the week with 74 exceeds it and calls for a review. The rising trend over three weeks is itself worth noting. The first checks are whether the branch's normal traffic changed (a new application, a scanner, a backup job, a change in the number of connections), whether the alerts concentrate on a few hosts or users, and whether the threshold or model version changed. Only then would one consider refitting the threshold, and the refit would be documented in the model card.