Python primer

This primer covers the parts of Python and its scientific libraries that the labs use: core Python, reading errors, virtual environments, NumPy, pandas, Matplotlib and scikit-learn, and the use of random seeds. It assumes you have written some Python before. The examples use the course dataset, data/flows.csv, which Lesson 1.3 describes, so by the end you will have loaded it, explored it and trained a first detector on it.

Run the code yourself as you read, in a Colab notebook or in Jupyter on your own computer (see How to use this course). The blocks build on each other, so run them in order. The output of each line is shown as a comment after it, or in a comment block below the code. These outputs were produced with the package versions in requirements.txt; with other versions the last digits of some numbers may differ.

Core Python

Lists hold ordered sequences and are indexed from 0. Negative indices count from the end, and a slice a[i:j] takes the items from position i up to, but not including, position j. Dictionaries map keys to values.

ports = [22, 80, 443, 3389]
print(ports[0])           # 22
print(ports[-1])          # 3389
print(len(ports))         # 4
print(ports[1:3])         # [80, 443]

service = {"ssh": 22, "http": 80, "https": 443}
print(service["https"])   # 443
service["rdp"] = 3389
print(sorted(service))    # ['http', 'https', 'rdp', 'ssh']

A loop and a list comprehension can build the same list. The comprehension can also filter.

squares = []
for p in ports:
    squares.append(p * p)
print(squares)                         # [484, 6400, 196249, 11485321]
print([p * p for p in ports])          # [484, 6400, 196249, 11485321]
print([p for p in ports if p > 100])   # [443, 3389]

Functions take positional and keyword arguments, and keyword arguments can have defaults. An f-string formats values inside text, and a format specification after a colon controls how a number is shown, for example :, for thousands separators or :.2% for a percentage with two decimals.

def is_large(byte_count, threshold=1_000_000):
    """Return True if a transfer is larger than the threshold."""
    return byte_count > threshold

n = 2_500_000
print(f"{n:,} bytes large? {is_large(n)}")
# 2,500,000 bytes large? True
print(f"{n:,} bytes large at 5 MB? {is_large(n, threshold=5_000_000)}")
# 2,500,000 bytes large at 5 MB? False

Libraries are imported once at the top of a notebook, usually under short conventional names: import numpy as np, import pandas as pd and import matplotlib.pyplot as plt.

Reading errors

When something goes wrong, Python stops and prints a traceback. Read it from the bottom up. The last line names the type of error and gives a message, and the lines above it show the chain of calls that led there, ending with the line where the error was raised, which in the example below is a line of your own code. Asking a dictionary for a key it does not contain produces this traceback (file paths shortened).

Traceback (most recent call last):
  File "ports.py", line 2, in <module>
    print(service["telnet"])
          ~~~~~~~^^^^^^^^^^
KeyError: 'telnet'

The last line says that the key 'telnet' is missing, and the line above points at the expression that asked for it. Misspelling a column name in pandas gives the same kind of error, KeyError: 'src_byte', after a longer traceback that passes through the internals of pandas. In that case skip the frames inside the library and look for the last frame that shows your own code. Three other errors are common when working in notebooks. A NameError means a variable was used before it was defined, often because a notebook cell above it was not run. A ModuleNotFoundError means a package is not installed in the environment the notebook is using. A ValueError or TypeError usually means a function received data of the wrong shape or type, and printing the .shape or type() of its inputs usually shows which.

Virtual environments and pinned versions

A virtual environment is a folder that holds its own Python interpreter and its own installed packages, so that the packages of one project do not interfere with those of another. On your own computer, create one for this course in the top folder of the repository with python -m venv .venv and activate it with .venv\Scripts\activate (on macOS and Linux, create it with python3 -m venv .venv and activate it with source .venv/bin/activate). Once it is active, install the packages with python -m pip install -r requirements.txt. The file requirements.txt pins the exact version of each package, such as numpy==2.5.2. Pinning matters because a new release of a library can change a default or a numerical routine and with it the printed results. To see what is installed, run python -m pip list, and to check one package from Python, print its __version__ attribute, for example np.__version__.

NumPy

NumPy's array holds many numbers of one type and applies arithmetic to all of them at once, without a Python loop. This is called vectorization, and it is both shorter to write and much faster than a loop.

import numpy as np

sizes = np.array([120, 540, 80, 1500, 240])
print(sizes.shape)            # (5,)
print(sizes.dtype)            # int64
print(sizes * 2)              # [ 240 1080  160 3000  480]
print(sizes.mean())           # 496.0
print(round(sizes.std(), 1))  # 527.2

A two-dimensional array is a table with rows and columns, which is how a dataset of several features is stored. The index X[i] selects row i, X[:, j] selects column j, and the argument axis=0 makes an aggregate run down the rows, giving one value per column.

X = np.array([[1, 120],
              [5, 540],
              [0, 80]])
print(X.shape)         # (3, 2)
print(X[0])            # [  1 120]
print(X[:, 1])         # [120 540  80]
print(X.mean(axis=0))  # [  2.         246.66666667]

A comparison on an array gives a boolean mask, an array of True and False values. Indexing with the mask keeps the elements where it is True. Because True counts as 1 and False as 0, the sum of a mask counts the matches and its mean gives their share.

big = sizes > 500
print(big)          # [False  True False  True False]
print(sizes[big])   # [ 540 1500]
print(big.sum())    # 2
print(big.mean())   # 0.4

Random numbers in the course come from a generator object created with np.random.default_rng(seed). The seed fixes the sequence, so the same seed gives the same numbers on every run, while successive calls on one generator continue the sequence.

rng = np.random.default_rng(42)
print(rng.integers(0, 100, size=5))   # [ 8 77 65 43 43]
print(rng.integers(0, 100, size=5))   # [85  8 69 20  9]

rng = np.random.default_rng(42)
print(rng.integers(0, 100, size=5))   # [ 8 77 65 43 43]

rng = np.random.default_rng(0)
# 10,000 draws, each True with probability 0.03
coin = rng.random(10_000) < 0.03
print(coin.mean())                    # 0.0297

Older code calls np.random.seed(42) and then functions such as np.random.rand, which use a hidden global generator. Do not mix the two styles in one program, since it becomes unclear which stream a result came from. The labs use generator objects only, and pass the generator, or a seed, to every function that needs randomness.

pandas

pandas stores tables as a DataFrame, whose columns are named and may have different types. Each column is a Series. The labs load the course dataset from a local copy if it exists and from GitHub otherwise, so the same code runs in Colab and on your own computer.

import os
import pandas as pd

LOCAL = "../../data/flows.csv"
URL = ("https://raw.githubusercontent.com/aalbata/"
       "machine-learning-for-cybersecurity/main/data/flows.csv")
flows = pd.read_csv(LOCAL if os.path.exists(LOCAL) else URL)
print(flows.shape)               # (37873, 21)
print(flows.columns.tolist())    # output wrapped here to fit the page
# ['timestamp', 'day', 'hour', 'site', 'src_host', 'protocol',
#  'service', 'duration', 'src_bytes', 'dst_bytes', 'packets',
#  'failed_logins', 'count', 'srv_count', 'serror_rate',
#  'rerror_rate', 'same_srv_rate', 'dst_host_count',
#  'ticket_opened', 'attack_type', 'label']

The first things to do with a new table are to look at a few rows, the column types and summary statistics. Double brackets select several columns as a smaller DataFrame. In pandas 3, text columns have the type str.

print(flows[["timestamp", "site", "service", "src_bytes",
             "attack_type", "label"]].head(3))
#               timestamp site service  src_bytes attack_type  label
# 0  2025-03-03T00:03:05Z    B   https    8539765      normal      0
# 1  2025-03-03T00:04:24Z    A     dns         65      normal      0
# 2  2025-03-03T00:05:51Z    A     dns         37      normal      0

print(flows.dtypes.value_counts())
# int64      11
# str         6
# float64     4
# Name: count, dtype: int64

print(flows[["duration", "src_bytes", "failed_logins"]]
      .describe().round(1))
#        duration    src_bytes  failed_logins
# count   37873.0      37873.0        37873.0
# mean       19.4     211119.4            0.0
# std       113.4    2839253.2            0.3
# min         0.0          2.0            0.0
# 25%         0.3        106.0            0.0
# 50%         1.1        447.0            0.0
# 75%         2.9       1210.0            0.0
# max      7926.4  201223301.0            6.0

The summary already says something about security data. The mean of src_bytes is far above its median and its maximum is enormous, because a few connections, such as backups, transfer vastly more than the rest. Such skewed features are usually transformed with a logarithm before modeling.

The next question for any detection dataset is how rare the attacks are. The method value_counts counts each distinct value, and the mean of a 0/1 column is the share of ones.

print(flows["label"].value_counts())
# label
# 0    36807
# 1     1066
# Name: count, dtype: int64
print(f"{flows['label'].mean():.2%} of records are attacks")
# 2.81% of records are attacks
print(flows["attack_type"].value_counts())
# attack_type
# normal        36807
# scan            338
# bruteforce      266
# c2beacon        185
# dos             165
# exfil           112
# Name: count, dtype: int64

Rows are filtered with a boolean mask, just as in NumPy. Conditions are combined with & (and), | (or) and ~ (not), and each condition must be wrapped in parentheses.

ssh = flows[flows["service"] == "ssh"]
print(len(ssh))                                       # 1728

night_uploads = flows[(flows["src_bytes"] > 100_000)
                      & (flows["hour"] < 5)]
print(len(night_uploads))                             # 430
print(night_uploads["attack_type"].value_counts())
# attack_type
# normal    397
# exfil      33
# Name: count, dtype: int64

Large uploads at night are mostly normal records, the nightly backups described in Lesson 1.3, with some exfiltration among them. A rule that alerted on every large night-time upload would bury the analysts in backups.

The parentheses are needed because & binds more tightly than > and < in Python. Without them, flows["src_bytes"] > 100_000 & flows["hour"] < 5 is read as a chained comparison around 100_000 & flows["hour"], and pandas stops with ValueError: The truth value of a Series is ambiguous. The message is confusing the first time you see it, and missing parentheses are a frequent cause.

The indexer .loc selects by row label and column name at the same time, and it also accepts a boolean mask in place of row labels. After read_csv, the row labels are the integers 0, 1, 2 and so on. With .loc, a slice of labels includes its end point, unlike a Python slice.

print(flows.loc[0, "service"])                       # https
print(flows.loc[:2, ["site", "service", "label"]])
#   site service  label
# 0    B   https      0
# 1    A     dns      0
# 2    A     dns      0
mask = flows["failed_logins"] >= 5
cols = ["src_host", "service", "failed_logins", "attack_type"]
print(flows.loc[mask, cols].head(4))
#      src_host service  failed_logins attack_type
# 2525     B036     ssh              5  bruteforce
# 4496     A026     ssh              6  bruteforce
# 4919     A089     ssh              5  bruteforce
# 5114     B014     ssh              6  bruteforce

Use .loc whenever you assign values to a subset of rows, as in flows.loc[mask, "column"] = value. Assigning through two separate indexing steps, such as flows[mask]["column"] = value, modifies a temporary copy and leaves flows unchanged; pandas 3 warns with a ChainedAssignmentError when this happens.

The method groupby splits the rows by the values of a column, applies an aggregate to each group, and combines the results.

print(flows.groupby("site")["label"]
      .agg(["size", "sum", "mean"]).round(4))
#        size  sum    mean
# site
# A     25512  504  0.0198
# B     12361  562  0.0455
print(flows.groupby("attack_type")["src_bytes"].median())
# attack_type
# bruteforce        776.5
# c2beacon          419.0
# dos                29.0
# exfil         3415089.5
# normal            454.0
# scan               59.5
# Name: src_bytes, dtype: float64

The share of attacks, which Lesson 1.2 calls the base rate, is more than twice as high at site B as at site A, a difference Unit 11 returns to. New columns are computed from existing ones, which is how features are engineered. Adding 1 before taking a logarithm or dividing avoids the infinite values that zeros would otherwise produce.

flows["log_src_bytes"] = np.log10(flows["src_bytes"] + 1)
flows["bytes_ratio"] = flows["src_bytes"] / (flows["dst_bytes"] + 1)
cols = ["src_bytes", "dst_bytes", "log_src_bytes", "bytes_ratio"]
print(flows[cols].head(3).round(3))
#    src_bytes  dst_bytes  log_src_bytes  bytes_ratio
# 0    8539765        682          6.931    12503.316
# 1         65        165          1.820        0.392
# 2         37        428          1.580        0.086

Finally, check for missing values and duplicates. The course dataset has no missing values, but it does contain records that were logged twice.

print(flows.isna().sum().sum())    # 0
print(flows.duplicated().sum())    # 1103

Real data usually has gaps. The method isna marks them, and fillna replaces them, here with the median of the column. Whether to fill, drop or flag missing values is a modeling decision that Unit 2 discusses.

small = pd.DataFrame({"host": ["A001", "A002", "A003"],
                      "duration": [1.5, None, 0.2]})
print(small.isna().sum())
# host        0
# duration    1
# dtype: int64
print(small["duration"].fillna(small["duration"].median()))
# 0    1.50
# 1    0.85
# 2    0.20
# Name: duration, dtype: float64

Matplotlib

Matplotlib draws figures through a figure object and one or more axes. The following code compares the distribution of the logarithm of bytes sent for normal and attack records. Because there are far more normal records, density=True scales each histogram to unit area so that the shapes can be compared.

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(6, 3))
for label, name in [(0, "normal"), (1, "attack")]:
    values = flows.loc[flows["label"] == label, "log_src_bytes"]
    ax.hist(values, bins=40, density=True, histtype="step", label=name)
ax.set_xlabel("log10(src_bytes + 1)")
ax.set_ylabel("density within class")
ax.legend()
plt.show()

The two distributions overlap over most of their range, which is why a detector cannot rely on this feature alone. Lesson 1.3 shows the same comparison as a figure. A bar chart shows counts per category, here the number of records of each attack family.

counts = flows.loc[flows["label"] == 1, "attack_type"].value_counts()
fig, ax = plt.subplots(figsize=(6, 3))
ax.bar(counts.index, counts.values)
ax.set_xlabel("attack family")
ax.set_ylabel("records")
plt.show()

Label every axis with the quantity and its unit, and avoid titles that repeat what the axes already say.

scikit-learn

Every scikit-learn classifier follows the same pattern. You create it with its settings, call fit on training data, and call predict or predict_proba on new data. The features are a table X with one row per example, and the labels are a vector y.

A detector learns from the past and is used on the future, so the course splits the data by time. Here the first ten days are used for training and the last four for testing. The features are eleven numeric measurements of each connection, all known when the connection ends. They leave out ticket_opened, which an analyst fills in later, the identifiers and times (src_host, timestamp, day and hour), the categorical columns, and of course label and attack_type.

features = ["duration", "src_bytes", "dst_bytes", "packets",
            "failed_logins", "count", "srv_count", "serror_rate",
            "rerror_rate", "same_srv_rate", "dst_host_count"]
train = flows[flows["day"] <= 9]
test = flows[flows["day"] >= 10]
X_train, y_train = train[features], train["label"]
X_test, y_test = test[features], test["label"]
print(X_train.shape, X_test.shape)          # (27039, 11) (10834, 11)
print(f"{y_train.mean():.4f} {y_test.mean():.4f}")   # 0.0231 0.0408

A pipeline chains preprocessing steps and a model into one object. Here it takes the logarithm of one plus each feature (np.log1p), standardizes the result to mean 0 and variance 1, and fits a logistic regression. Because the scaling is part of the pipeline, its means and variances are learned from the training data only.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler

model = make_pipeline(FunctionTransformer(np.log1p), StandardScaler(),
                      LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
pred = model.predict(X_test)               # 0 or 1 for each test record
# estimated probability of an attack for each test record
score = model.predict_proba(X_test)[:, 1]
print(pred[:10])                           # [0 0 0 0 0 0 0 0 0 0]
print(score[:3].round(4))                  # [0.0026 0.0033 0.053 ]

The label in this dataset depends on the features, because the generator gives each kind of behavior its own distributions, so the model has something real to learn. Accuracy would say little here (Lesson 1.2), so the evaluation uses the confusion matrix, recall, precision and the false positive rate.

from sklearn.metrics import (confusion_matrix, precision_score,
                             recall_score)

tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
print(f"TP={tp} FN={fn} FP={fp} TN={tn}")
# TP=146 FN=296 FP=13 TN=10379
print(f"recall    {recall_score(y_test, pred):.3f}")
# recall    0.330
print(f"precision {precision_score(y_test, pred):.3f}")
# precision 0.918
print(f"FPR       {fp / (fp + tn):.4f}")
# FPR       0.0013

The method predict applies a threshold of 0.5 to the score. A lower threshold catches more attacks and raises more false alerts.

pred_low = (score >= 0.1).astype(int)
tn, fp, fn, tp = confusion_matrix(y_test, pred_low).ravel()
print(f"TP={tp} FN={fn} FP={fp} TN={tn}")
# TP=205 FN=237 FP=148 TN=10244
print(f"recall {tp / (tp + fn):.3f}, precision {tp / (tp + fp):.3f}")
# recall 0.464, precision 0.581

Choosing the threshold from the costs of errors or from an alert budget is the subject of Lessons 1.2 and 2.3. This model differs slightly from the one in Lab 1, which also uses the categorical columns protocol and service and the hour of the day, so its numbers differ too.

Reproducibility with seeds

Many models and utilities use randomness, and each accepts a random_state argument that fixes it. With the same seed, the same data and the same package versions, a random forest is identical from run to run. A different seed gives a slightly different model, which may or may not show in a given measure, so the loop below prints both recall and precision.

from sklearn.ensemble import RandomForestClassifier

for seed in [0, 0, 1]:
    forest = RandomForestClassifier(n_estimators=50,
                                    random_state=seed, n_jobs=1)
    forest.fit(X_train, y_train)
    pred_rf = forest.predict(X_test)
    rec = round(recall_score(y_test, pred_rf), 4)
    prec = round(precision_score(y_test, pred_rf), 4)
    print(seed, rec, prec)
# 0 0.4774 0.9769
# 0 0.4774 0.9769
# 1 0.4774 0.9814

Seeds 0 and 1 happen to give the same recall, but the two forests flag different test records and their precision differs. The variation across seeds is a reminder that a single run is one sample. When two models differ by less than the variation across seeds, the difference should not be reported as an improvement.

The function train_test_split makes random splits and also takes random_state. Its argument stratify keeps the share of each class nearly equal in both parts, as close as whole numbers of records allow.

from sklearn.model_selection import train_test_split

X_a, X_b, y_a, y_b = train_test_split(
    flows[features], flows["label"], test_size=0.3,
    random_state=42, stratify=flows["label"])
print(X_a.shape, X_b.shape)                  # (26511, 11) (11362, 11)
print(f"{y_a.mean():.4f} {y_b.mean():.4f}")  # 0.0281 0.0282

A random split like this is reproducible, but it is the wrong split for this dataset. It mixes future days into training, so the test overstates how the detector would perform, and it places copies of duplicated records on both sides of the split. Use it only when the examples really are independent of time, and prefer a split by time for detection data (Lesson 1.3 and Unit 2). The seed fixes which records fall on each side of the split, but it does not remove either problem. In PyTorch, used from Unit 5 on, the equivalent call is torch.manual_seed(seed).

Quick reference

The table below collects the operations of this primer, in the order they appear, with the code that performs each. In it, df stands for any DataFrame and g for a grouped DataFrame.