Unit 2. Data, leakage and evaluation

This unit covers three stages of an evaluation at which mistakes can make a security model look better or worse than it will be in deployment. Lesson 2.1 turns raw flows, logs and text into features without letting the test data shape them, covering deduplication, missing values, categorical encodings including target encoding, scaling, log transforms and term frequency-inverse document frequency (TF-IDF) weighting. Lesson 2.2 shows how to split data so that the test resembles deployment, by time and by host, and how duplicates, leaked columns, preprocessing and model selection inflate scores. Lesson 2.3 explains ROC and precision-recall curves, why average precision depends on the base rate, how to set a threshold from an alert budget or from costs, and how to check and repair calibration.

After the unit you will be able to build a leakage-free scikit-learn pipeline for a security dataset, choose a split that matches how a detector will be used, compute the area under the ROC curve (AUC) and average precision by hand and read both curves correctly, and translate a detector's scores into a daily alert volume. Lab 2 measures on the course dataset what duplicates, the choice of split, a leaked column and preprocessing fitted on all the data do to a detector's score, and computes the curves, alert-budget thresholds and reliability diagrams of Lesson 2.3. The unit assumes Lessons 1.2 and 1.3 (base rates, the confusion matrix and the course dataset). Plan on about eight to ten hours, including the lab and the exercises.

Lessons

1. 2.1 Preparing security data
2. 2.2 Splits, leakage and cross-validation
3. 2.3 ROC and precision-recall curves, and choosing a threshold

Labs

Practice

• Exercises with worked solutions
• Self-check quiz