Unit 1 exercises

(a) The base rate is \(\pi = 1/20{,}000 = 5 \times 10^{-5}\), so \(200{,}000 \times 5 \times 10^{-5} = 10\) attacks occur per day. The detector raises \(10 \times 0.95 = 9.5\) true alerts on average and \(199{,}990 \times 0.002 = 399.98\), or about 400, false alerts.

(b) Precision is \(9.5/(9.5 + 399.98) \approx 0.023\). About one alert in 43 is real.

(c) Precision reaches one half when true and false alerts are equal. By the bound of Lesson 1.2 this requires

That is about 42 times smaller than the current rate of 0.002, and it allows about 9.5 false alerts a day, matching the 9.5 true ones.

(a) With 499,950 normal connections and 50 attacks, the expected alerts are the false alerts \(499{,}950 \times \mathrm{FPR}\) plus the true alerts \(50 \times \mathrm{TPR}\), as the table below shows.

(b) Only the threshold 0.9 keeps the expected load, about 40 alerts, within the budget of 60. Its recall is 0.60, so 20 of the 50 daily attacks go unnoticed. The thresholds 0.5 and 0.7 would detect more attacks on paper, but the analysts could examine only 60 of the roughly 245 and 90 alerts they raise each day, so about three quarters of the detections at 0.5 and a third of those at 0.7 would never be read. If the analysts read 60 alerts in arbitrary order, they would find about 11 attacks at 0.5 and 27 at 0.7, fewer than the 30 found at 0.9. The table also shows that the alert load changes across thresholds almost entirely through the false alerts, which fall from about 200 to 10 while the true alerts fall only from 45 to 30.

(c) At a true positive rate of 0.80 the detector raises 40 true alerts, which leaves room for \(60 - 40 = 20\) false alerts. The false positive rate must then satisfy \(499{,}950 \times \mathrm{FPR} \le 20\), so \(\mathrm{FPR} \le 4.0 \times 10^{-5}\), two and a half times smaller than the current \(10^{-4}\) at that threshold.

(a) By inclusion and exclusion, \(P(S \cup L) = p_{\mathrm{s}} + p_{\mathrm{l}} - P(S \cap L)\). The intersection is at least \(\max(0, p_{\mathrm{s}} + p_{\mathrm{l}} - 1)\), because the union cannot exceed 1, and at most \(\min(p_{\mathrm{s}}, p_{\mathrm{l}})\), so

The lower bound is reached when one detector fires on every event on which the other fires (the events caught by the weaker detector are a subset of those caught by the stronger), so the second detector adds nothing. When \(p_{\mathrm{s}} + p_{\mathrm{l}} \le 1\), the upper bound is reached when the two never fire on the same event, so their detections simply add. When the sum exceeds 1, the bound of 1 is reached when every event triggers at least one of the detectors. Independence, \(P(S \cap L) = p_{\mathrm{s}} p_{\mathrm{l}}\), lies in between.

(b) For variants the union lies between 0.60 and 0.70, and independence gives 0.64. For normal events it lies between \(1.00 \times 10^{-4}\) and \(1.02 \times 10^{-4}\). The gain in recall depends strongly on how the two detectors' misses are related, from nothing at all to 0.10, while the cost in false alarms is close to the sum of the two rates whatever their relationship, because the signature system's false positive rate is so small. In practice the attacks that evade one detector often evade the other for the same reason, so the realistic gain is nearer the lower bound than the independence calculation suggests, and it should be measured on data rather than assumed.

The following code, run from the folder labs/unit-01 like the lab, does the three steps.

import pandas as pd

path = "../../data/flows.csv"
flows = pd.read_csv(path)
unique = flows.drop_duplicates()
n_dup = len(flows) - len(unique)
print(len(flows), len(unique), n_dup)
rate_all = flows["label"].mean()
rate_unique = unique["label"].mean()
print("base rate with duplicates:   ",
      f"{rate_all:.4f}")
print("base rate without duplicates:",
      f"{rate_unique:.4f}")
by_site = unique.groupby("site")
stats = ["size", "sum", "mean"]
print(by_site["label"].agg(stats))
family = unique["attack_type"]
site = unique["site"]
print(pd.crosstab(family, site))

Removing the 1,103 duplicates leaves 36,770 records, of which 1,040 are attacks. The base rate moves from 2.81% to 2.83%. Whether it rises or falls depends on whether duplicates are more or less common among attacks than among normal records. Here 26 of the 1,103 duplicates are attacks, a lower share than in the data as a whole, so removing them raises the base rate a little. Because the generator duplicated records at random, without regard to their label, the best prediction was "almost no change". The per-site table shows that site B keeps a base rate more than twice that of site A, and the cross-tabulation shows which families are concentrated at site B. Duplicates matter far more for how a split is made than for the base rate, as Lesson 1.3 explains.

(a) By the expected-alerts formula of Lesson 1.3, the site has 100 attacks and 99,900 normal connections a day. The detector raises \(100 \times\) recall \(=\) 31.4 true alerts and \(99{,}900 \times \mathrm{FPR} =\) 76.9 false alerts, about 108 in total.

(b) Precision is the share of true alerts, 0.290, which is the value Lab 1 prints for a base rate of 0.1%. Most alerts would be false, although on the test days, where the base rate is 4.1%, the same detector had a precision of 0.95.

(c) For deployment, the load fits within the budget, and the detector is cheap to run. Against it, it misses more than two attacks in three, including every command-and-control beacon, and most of the alerts analysts would read are false. A better course is to lower the threshold until the alert load approaches the budget and measure the recall gained, which is the method of Lesson 2.3, and to treat the missed beacon family as evidence of drift that retraining on recent data must address.

(a) Five problems stand out.

1. The column attack_type is the label in another form, so the model can read the answer directly. It must be removed.
2. The column ticket_opened is filled in after an analyst investigates, so it is not available when the detector decides. It is target leakage and must be removed.
3. The columns src_host, timestamp and day let the model learn which hosts and which days were attacked instead of what attacks look like. Hosts at site B appear in attack records more than twice as often, relative to their traffic, as hosts at site A, so host identity correlates with the label without explaining it, a spurious correlation in the sense of Arp et al. (2022). Such identifiers should be removed or replaced by behavioral features.
4. A random split puts future records, including some of the c2beacon family, and copies of duplicated records into training. The split should be by time (train on early days, test on later ones), with duplicates removed or kept on one side.
5. The default threshold of 0.5 ignores the costs and the alert budget. The threshold should be chosen on a validation period from the budget or from the costs of errors, and recall, the false positive rate, precision and expected alerts per day should be reported at that operating point.

(b) With mean 80, the standard deviation of a Poisson count is \(\sqrt{80} \approx 8.9\), so \(z = (112 - 80)/8.9 \approx 3.6\). A day that far above the mean is unusual under the Poisson model. Before suspecting an attack campaign, check whether the alerts are concentrated on one host or one service, which would point to a misconfiguration, a new benign application or a broken data feed, and whether the mix of traffic changed that day. Because alerts from one host are not independent, the Poisson model understates normal variation, so the comparison with the same weekday in previous weeks is also worth making.