Unit 2 exercises

(a) There are 4 attacks and 6 normal records, so 24 pairs. The attack at rank 1 is above all 6 normal records, the one at rank 2 also above 6, the one at rank 4 above 5 (ranks 5, 6, 7, 9, 10), and the one at rank 8 above 2 (ranks 9 and 10). The count is \(6 + 6 + 5 + 2 = 19\), so \(\mathrm{AUC} = 19/24 \approx 0.792\).

(b) The precision at the rank of each attack is \(1/1\), \(2/2\), \(3/4\) and \(4/8\). AP is their mean, \((1 + 1 + 0.75 + 0.5)/4\), which equals 0.8125. (Joining the points of the precision-recall curve with straight lines from \((0, 1)\) and applying the trapezoidal rule gives about 0.793. The course uses the step-wise definition because, as Lesson 2.3 explains, straight-line interpolation between points in precision-recall space is incorrect.)

(c) A random ranking has expected precision equal to the base rate, \(4/10 = 0.4\), at every cutoff, and for a large set of records its AP is close to that value. With only ten records the expected AP is higher, about 0.53 (the mean over all 210 equally likely placements of the four attacks), because AP evaluates precision at the rank of each attack, a cutoff that always includes that attack. The value approaches the base rate as the number of records grows.

(d) A threshold of 0.70 flags ranks 1 to 3. Two of the three alerts are attacks, so precision is \(2/3\) and recall is \(2/4 = 0.5\).

(a) By Bayes' rule, \(p = P(x \mid A)\,b / P(x)\) and \(1 - p = P(x \mid N)(1-b)/P(x)\), so the odds are \(p/(1-p) = \Lambda(x)\, b/(1-b)\) with the likelihood ratio \(\Lambda(x) = P(x \mid A)/P(x \mid N)\). The same holds at \(b'\) with the same \(\Lambda(x)\), because the class-conditional distributions are unchanged. Dividing one expression by the other eliminates \(\Lambda(x)\) and gives \eqref{eq:ex-odds}. It is the odds form of Theorem 2 of Elkan (2001), the base-rate correction of predicted probabilities in Lesson 2.3.

(b) With \(b = 0.5\) the denominator \(b/(1-b)\) is 1 and \(b'/(1-b') = 0.002/0.998\). An output of 0.5 has odds 1, giving \(p' = 0.002\). An output of 0.9 has odds 9, giving odds \(0.01804\) and \(p' \approx 0.0177\). An output of 0.99 gives \(p' \approx 0.166\), and 0.999 gives \(p' \approx 0.667\).

(c) Deployment odds of \(0.1/0.9 = 0.1111\) require balanced odds of \(0.1111 \times 0.998/0.002\), which is 55.4, so \(p = 55.4/56.4 \approx 0.982\).

(a) At \(t_1\) there are 100 false and 18 true alerts, 118 in all, and precision is \(18/118 \approx 0.153\). At \(t_2\) there are 30 false and 15 true alerts, 45 in all, and precision is \(15/45 \approx 0.333\). At \(t_3\) there are 10 false and 10 true alerts, 20 in all, and precision is 0.5.

(b) \(t_2\). It is the threshold with the highest recall whose expected volume fits within the budget of 50. \(t_1\) would more than double the analysts' workload, so in practice many of its alerts would not be investigated at all. \(t_3\) leaves budget unused and misses a third of the attacks that \(t_2\) would catch (5 of 15 a day).

(c) Rates measured on validation days carry sampling error (at a false positive rate of \(3 \times 10^{-5}\), the estimate rests on a few dozen false alerts), and traffic changes. A software update, a new scanner or a new attack family can shift the score distribution, and a burst of attacks adds true alerts, as the denial-of-service traffic on day 11 does for the fixed threshold of Lab 2. The alert count has to be monitored.

(a) \(\bar y = 5/10 = 0.5\). For curl, \(s_c = 3\) and \(n_c = 4\), so \((3 + 0.5)/5 = 0.7\). For firefox, \((1 + 0.5)/5 = 0.3\). For python, \((1 + 0.5)/3 = 0.5\).

(b) Rows in fold 1 use fold 2 (rows 2, 4, 6, 8, 10), where \(\bar y = 2/5 = 0.4\). Curl has \(s_c = 2\), \(n_c = 2\), giving \((2 + 0.4)/3 = 0.8\). Firefox has \(s_c = 0\), \(n_c = 2\), giving \(0.4/3 = 0.133\). Python has \(s_c = 0\), \(n_c = 1\), giving \(0.4/2 = 0.2\). Rows in fold 2 use fold 1 (rows 1, 3, 5, 7, 9), where \(\bar y = 3/5 = 0.6\). Curl has \(s_c = 1\), \(n_c = 2\), giving \(1.6/3 = 0.533\). Firefox has \(s_c = 1\), \(n_c = 2\), giving \(0.533\). Python has \(s_c = 1\), \(n_c = 1\), giving \(1.6/2 = 0.8\).

(Here the prior \(\bar y\) is computed from the rows used for the encoding. Using the overall attack share of 0.5 instead changes the numbers slightly but not the conclusion.)

(c) Row 7 is a firefox attack in fold 1. Under (a) its encoding, 0.3, includes its own label; under (b) it is 0.133, computed only from rows 6 and 8, both normal, and the prior of fold 2. In training, the encoding must look the way it will look on new data, where the record's own label is unknown. Encoding from statistics that include the row's own label makes the attacks of each category look slightly more like attacks than they will at prediction time, and with small categories that difference is large enough for a model to exploit.

One solution, which uses dedup and make_model from the lab,

from sklearn.model_selection import (StratifiedGroupKFold,
                                     cross_val_score)

cv = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(make_model("hgb"), dedup, dedup["label"],
                         groups=dedup["src_host"], cv=cv,
                         scoring="average_precision")
print("grouped 5-fold AP per fold:", scores.round(3),
      "mean", scores.mean().round(3))

by_day = dedup.sort_values("timestamp").reset_index(drop=True)
folds = [(np.flatnonzero(by_day["day"] < d),
          np.flatnonzero(by_day["day"] == d)) for d in range(8, 14)]
scores_t = cross_val_score(make_model("hgb"), by_day, by_day["label"],
                           cv=folds, scoring="average_precision")
print("forward-chaining AP per test day 8 to 13:", scores_t.round(3))

cross_val_score accepts an explicit list of (training indices, test indices) pairs as cv, which makes custom temporal folds easy. In the grouped cross-validation, one fold scores far below the others. That is the fold whose test hosts include A010, the IT scanner, for the reason explained in Lesson 2.2; the mean over folds hides the problem, and the per-fold scores reveal it. In the forward-chaining folds, the score drops on day 10, when the beacon family first appears, and recovers on days 11 to 13, because from then on day 10 is part of the training data. The temporal split of the lab, which trains only on days 0 to 9, corresponds to a detector that is never retrained; the forward-chaining scores correspond to one retrained daily with labels from the previous day. That assumption needs analysts to label yesterday's traffic, which is itself an operational cost.

(a) The number of user reports and the quarantine flag both accumulate after delivery and partly reflect the label, so they are target leakage. The gateway spam score at delivery is legitimate if it is the score recorded at the time of receipt; it would not be if it was recomputed later with updated rules. Any feature computed over a window (for example, how many messages a sender sent that week) must use only messages received before the one being classified.

(b) Split by time, for example training on January 2024 to December 2024, using January to March 2025 for validation and threshold selection, and testing on April to June 2025. In addition, group by campaign or by sender domain so that near-identical messages of one campaign do not appear on both sides; a temporal split alone does not prevent this when a campaign spans the boundary. Remove exact duplicates first, since the same message is often delivered to many users.

(c) Report the precision-recall curve and average precision on the test months, together with the base rate of phishing in those months, and the ROC curve for comparison with other datasets. Choose the threshold on the validation months from the team's alert budget (or from the costs of a missed phish and of a quarantined legitimate message), and report the recall, precision and daily alert volume at that fixed threshold on the test months.