Module 10 · Core

Missing values and honest cleaning

Find missing entries and choose a treatment that fits the question.

Before you start: You can convert columns and use Boolean masks.

Python loads from this site on your first run.

Each run starts with fresh data. Download any exported files before another run. Your written answers and code drafts are saved on this device; export your work to keep a backup.

M10C01 · Explore · predict · practice

Missing is not the same as empty or zero

isna() gives True where a value is missing. Zero is a recorded value, and an empty string is text unless you explicitly treat it as missing. Counting True values with .sum() counts missing entries.

ReadWhat type of values does isna() return?

PredictHow many missing entries will be counted?

ExplainWhy is the empty string not automatically missing?

Read the example

values = pd.Series([0, None, "", pd.NA, np.nan], dtype="object")
result = values.isna()
print(result.tolist())
print("Missing count:", result.sum())

Verified example output · pandas 2.2.3

[False, True, False, True, True]
Missing count: 3

Watch for this. Use .isna(), not == np.nan, to detect missing values.

Your turn

Treat the empty string as missing, but keep zero. Store the cleaned Series in result and count missing entries.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

values = pd.Series([0, None, "", pd.NA, np.nan], dtype="object")
result = values.isna()
print(result.tolist())
print("Missing count:", result.sum())

Show a hint

Use .replace("", pd.NA).

Show one worked solution

values = pd.Series([0, None, "", pd.NA, np.nan], dtype="object")
result = values.replace("", pd.NA)
print(result)
print("Missing count:", result.isna().sum())

Four values are now missing. The recorded zero is still zero.

Compare your explanations

Read. True/False values, one for each entry.
Predict. Three: None, pd.NA, and np.nan.
Explain. It is a stored text value. A cleaning rule must say whether blank text represents missing data.

One quick check

Which original value is known rather than missing?

What you should be able to explain. Define what missing means in your dataset.

Your own assessment, separate from code checks

M10C02 · Explore · predict · practice

Drop rows only for required fields

dropna(subset=[...]) removes rows missing the specified fields. It does not remove a row just because an unrelated field is missing. Keep a count of excluded records and consider how the exclusion affects the answer.

ReadWhich column controls row removal in this cell?

PredictWill S006 remain even though Study_Hours is missing?

ExplainWhy not use df.dropna() without thinking?

Read the example

df = raw_students()
df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
result = df.dropna(subset=["Score"])
print(result[["Student_ID", "Score", "Study_Hours"]])
print("Rows kept:", len(result))

Verified example output · pandas 2.2.3

   Student_ID  Score Study_Hours
0        S001   88.0           8
1        S002   72.0           5
2        S003   91.0           9
4        S005   79.0           6
5        S006   95.0        None
6        S007   84.0           7
8        S009   76.0           6
9        S010   89.0           8
10       S011   68.0           5
11       S012   82.0           7
12       S003   91.0           9
Rows kept: 11

Watch for this. Removing incomplete records can change the composition of your data; report what was excluded.

Your turn

Keep only rows with both Score and Study_Hours available. Leave the intentional duplicate for the next module.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

df = raw_students()
df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
result = df.dropna(subset=["Score"])
print(result[["Student_ID", "Score", "Study_Hours"]])
print("Rows kept:", len(result))

Show a hint

Pass both required column names to subset.

Show one worked solution

df = raw_students()
df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
result = df.dropna(subset=["Score", "Study_Hours"])
print(result[["Student_ID", "Score", "Study_Hours"]])
print("Rows kept:", len(result))

Ten rows remain out of thirteen. S004, S008, and S006 were excluded for the stated requirements.

Compare your explanations

Read. Score only.
Predict. Yes. S006 has a score, so it passes the Score-only requirement.
Explain. It may remove rows for fields the analysis does not need, changing the sample unnecessarily.

One quick check

What does subset specify?

What you should be able to explain. Match required fields to the analysis question.

Your own assessment, separate from code checks

M10C03 · Explore · predict · practice

Fill a label without inventing a measurement

fillna() replaces missing values with a value you specify. A label such as "Unknown" can make a missing category visible in a report. The same idea does not justify inventing a score or age.

ReadWhat is being filled here: a measured number or a category label?

PredictHow many Unknown labels will appear?

ExplainDoes Unknown tell us the actual campus?

Read the example

campuses = pd.Series(["RAK", None, "Dubai", None], dtype="string")
result = campuses.fillna("Unknown")
print(result.tolist())

Verified example output · pandas 2.2.3

['RAK', 'Unknown', 'Dubai', 'Unknown']

Watch for this. Do not use zero as a universal replacement for missing numbers.

Your turn

Use the same Series, but first record a Boolean Was_missing flag. Store a DataFrame named result with the filled Campus column and the Was_missing flag.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

campuses = pd.Series(["RAK", None, "Dubai", None], dtype="string")
result = campuses.fillna("Unknown")
print(result.tolist())

Show a hint

Call .isna() before replacing missing values.

Show one worked solution

campuses = pd.Series(["RAK", None, "Dubai", None], dtype="string")
result = pd.DataFrame({"Campus": campuses.fillna("Unknown"), "Was_missing": campuses.isna()})
print(result)

The flag preserves which labels were originally missing, even after every cell contains text.

Compare your explanations

Read. A category label.
Predict. Two.
Explain. No. It makes the missing information explicit; it does not recover the campus.

One quick check

Why retain a missingness flag?

What you should be able to explain. Keep missingness visible when replacements are used.

Your own assessment, separate from code checks

M10C04 · Explore · predict · practice

Compare a fill policy with keeping missing data

Preview of Module 16: the groupby, transform, and named aggregation syntax in this cell will be explained there. First focus on the difference between a technical replacement and a justified data policy. A group median is one possible imputation rule, not a universal answer. transform() returns a group result for every original row. This small example compares a proposed fill with the original values so its effect is visible.

ReadWhy does transform return five values instead of two group summaries?

PredictWhich values fill the missing AI and IT hours?

ExplainWhy is filling the IT value with 2.0 a weakly supported choice?

Read the example

df = pd.DataFrame({"Program": ["AI", "AI", "AI", "IT", "IT"], "Hours": [4.0, 8.0, np.nan, 2.0, np.nan]})
group_median = df.groupby("Program")["Hours"].transform("median")
result = df.assign(Hours_filled=df["Hours"].fillna(group_median), Was_missing=df["Hours"].isna())
print(result)

Verified example output · pandas 2.2.3

  Program  Hours  Hours_filled  Was_missing
0      AI    4.0           4.0        False
1      AI    8.0           8.0        False
2      AI    NaN           6.0         True
3      IT    2.0           2.0        False
4      IT    NaN           2.0         True

Watch for this. In predictive modeling, fit any imputation rule on training data only; do not learn it from held-out test data.

Your turn

Use the same original table. Keep the missing values and store a summary in result with Program as its index. Name the known Hours count Known_hours, the row count Rows, and the median Median_hours.

For the code check, keep these result variable names: df, result. Different valid approaches are welcome.

Your Python code

df = pd.DataFrame({"Program": ["AI", "AI", "AI", "IT", "IT"], "Hours": [4.0, 8.0, np.nan, 2.0, np.nan]})
group_median = df.groupby("Program")["Hours"].transform("median")
result = df.assign(Hours_filled=df["Hours"].fillna(group_median), Was_missing=df["Hours"].isna())
print(result)

Show a hint

Use named aggregation; count counts known Hours, while size counts rows.

Show one worked solution

df = pd.DataFrame({"Program": ["AI", "AI", "AI", "IT", "IT"], "Hours": [4.0, 8.0, np.nan, 2.0, np.nan]})
result = df.groupby("Program").agg(Known_hours=("Hours", "count"), Rows=("Hours", "size"), Median_hours=("Hours", "median"))
print(result)

AI has 2 known values in 3 rows; IT has 1 in 2. The report shows the missing information rather than replacing it.

Compare your explanations

Read. It repeats each group median on the rows belonging to that group.
Predict. AI gets 6.0, the median of 4 and 8; IT gets 2.0, its only observed value.
Explain. It is based on only one observed IT value and can hide uncertainty or reduce apparent variation.

One quick check

Should every missing measurement be filled with a mean or median?

What you should be able to explain. Explain the assumption behind every fill policy.

Your own assessment, separate from code checks

Previous: Data types and safe conversion Next: Duplicates and data quality

Further detail: official documentation for this topic.