Unit 4. Unsupervised learning and anomaly detection

Most security data arrives without labels. This unit covers the methods that work without them. Lesson 4.1 develops k-means from its objective, Lloyd's algorithm and k-means++ seeding, shows how to choose the number of clusters with the silhouette, and contrasts it with density-based clustering (DBSCAN and HDBSCAN) and hierarchical clustering. Lesson 4.2 derives principal component analysis (PCA) from the covariance matrix and the singular value decomposition, works a two-dimensional example exactly, and uses the reconstruction error as an anomaly score, as in the subspace method for network-wide traffic anomalies. Lesson 4.3 presents the isolation forest, the local outlier factor and the one-class support vector machine, explains how scores become alerts under an alert budget, and measures how often unusual activity turns out to be benign and how often attacks look ordinary.

After this unit you will be able to run and interpret clustering, PCA and anomaly detectors on network flows, choose their parameters with a stated rule, evaluate a detector by average precision, precision at k and per-family recall, and explain to a colleague why an anomaly score alone cannot tell a backup from exfiltration. The unit assumes Units 1 to 3, in particular base rates and the alert budget (Lessons 1.2 and 2.3), feature scaling (Lesson 2.1) and time-ordered splits (Lesson 2.2). Plan on about 9 hours, including Lab 4 and the exercises.

Lessons

1. 4.1 Clustering
2. 4.2 Principal component analysis
3. 4.3 Anomaly detection and the alert budget

Labs

Practice

• Exercises with worked solutions
• Self-check quiz