Module 25 · Apply

Debugging clinic

Find the reason a pandas result is wrong, then repair it.

Before you start: Modules 1–20; revisit selection, cleaning, and joins.

Python loads from this site on your first run.

Each run starts with fresh data. Download any exported files before another run. Your written answers and code drafts are saved on this device; export your work to keep a backup.

M25C01 · Explore · predict · practice

A label is not a row position

A table can use student IDs as its index. Position 0 still means the first row, but the label of that row is now S001. Read the index before choosing a selector.

ReadWhat labels does df use now?

PredictWhy do both selections contain two rows?

ExplainWould df.loc[0] mean the first row here?

Read the example

df = students().set_index("Student_ID")
by_position = df.iloc[0:2][["Name", "Score"]]
by_label = df.loc["S001":"S002", ["Name", "Score"]]
print(by_position)
print("Same rows:", by_position.equals(by_label))

Verified example output · pandas 2.2.3

             Name  Score
Student_ID              
S001        Amira     88
S002        Bilal     72
Same rows: True

Watch for this. An integer inside loc is a label, even when it looks like a row number.

Your turn

Select S003 through S005 by label, then select the same three students by position. Print both and confirm equality.

For the code check, keep these result variable names: by_label, by_position. Different valid approaches are welcome.

Your Python code

df = students().set_index("Student_ID")
by_position = df.iloc[0:2][["Name", "Score"]]
by_label = df.loc["S001":"S002", ["Name", "Score"]]
print(by_position)
print("Same rows:", by_position.equals(by_label))

Show a hint

The first selected row has position 2. The excluded stop must be one position after S005.

Show one worked solution

df = students().set_index("Student_ID")
by_label = df.loc["S003":"S005", ["Name", "Score"]]
by_position = df.iloc[2:5][["Name", "Score"]]
print(by_label)
print(by_position)
print("Same rows:", by_label.equals(by_position))

Both results contain Chloe, Dana, and Ethan. Labels and positions happen to select the same records here, but they express different rules.

Compare your explanations

Read. The Student_ID values, such as S001, are index labels.
Predict. iloc excludes stop position 2; loc includes both existing endpoint labels S001 and S002.
Explain. No. It asks for label 0, which is absent, so it raises KeyError.

One quick check

Which stop is excluded?

What you should be able to explain. Choose a selector from the meaning of the request, not from habit.

Your own assessment, separate from code checks

M25C02 · Explore · predict · practice

Repair a condition with two rules

A filter must produce one True or False value per row. Combine pandas conditions with & for AND or | for OR, with parentheses around each comparison. The broken expression is a comment so this example can run.

ReadWhat does each True in mask represent?

PredictWill Lina be included?

ExplainWhy is and wrong for these Series?

Read the example

df = students()
# Broken: df[(df["Score"] >= 80) and (df["Attendance"] >= 90)]
mask = (df["Score"] >= 80) & (df["Attendance"] >= 90)
result = df.loc[mask, ["Student_ID", "Score", "Attendance"]]
print(result)

Verified example output · pandas 2.2.3

   Student_ID  Score  Attendance
0        S001     88          95
2        S003     91          98
5        S006     95          99
6        S007     84          92
9        S010     89          96
11       S012     82          94

Watch for this. Do not remove the parentheses around each comparison.

Your turn

Find students whose Score is below 70 OR whose Attendance is below 80. Print their IDs and both values.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

df = students()
# Broken: df[(df["Score"] >= 80) and (df["Attendance"] >= 90)]
mask = (df["Score"] >= 80) & (df["Attendance"] >= 90)
result = df.loc[mask, ["Student_ID", "Score", "Attendance"]]
print(result)

Show a hint

Use | and keep both comparisons in parentheses.

Show one worked solution

df = students()
mask = (df["Score"] < 70) | (df["Attendance"] < 80)
result = df.loc[mask, ["Student_ID", "Score", "Attendance"]]
print(result)

S004, S008, and S011 meet at least one condition. S011 qualifies by score even though attendance is 82.

Compare your explanations

Read. A student meeting both the score and attendance conditions.
Predict. Yes. Her score is 82 and attendance is 94.
Explain. and expects a single truth value. Each comparison returns a Series of truth values, which & combines element by element.

One quick check

A student meets only one of two conditions. Which operator keeps that student?

What you should be able to explain. Translate AND, OR, and NOT into &, |, and ~.

Your own assessment, separate from code checks

M25C03 · Explore · predict · practice

Change the table you actually intend to change

A filtered table and the original table are different objects. Use a single .loc assignment when the original must change. Use .copy() when creating a separate working table, and inspect both afterward.

ReadWhich object receives the assignment?

PredictWhat score will S004 have now?

ExplainWhy should you avoid the commented chained assignment?

Read the example

df = students()
mask = df["Student_ID"].eq("S004")
# Avoid: df[mask]["Score"] = 70
df.loc[mask, "Score"] = 70
result = df.loc[mask, ["Student_ID", "Score"]]
print(result)

Verified example output · pandas 2.2.3

  Student_ID  Score
3       S004     70

Watch for this. A method returning a new object is not an instruction to update the old one.

Your turn

Start with a fresh df = students() and create a separate copy of its RAK rows named rak. Add 5 points to scores in rak. Show that Amira's copied score is 93 while her score in df remains 88.

For the code check, keep these result variable names: df, rak. Different valid approaches are welcome.

Your Python code

df = students()
mask = df["Student_ID"].eq("S004")
# Avoid: df[mask]["Score"] = 70
df.loc[mask, "Score"] = 70
result = df.loc[mask, ["Student_ID", "Score"]]
print(result)

Show a hint

Start with df.loc[df['Campus'].eq('RAK')].copy().

Show one worked solution

df = students()
rak = df.loc[df["Campus"].eq("RAK")].copy()
rak["Score"] = rak["Score"] + 5
print("Copied score:", rak.loc[rak["Student_ID"].eq("S001"), "Score"].iloc[0])
print("Original score:", df.loc[df["Student_ID"].eq("S001"), "Score"].iloc[0])

The working copy changes; the original data stays available for comparison. For these numeric columns, .copy() makes the intended separation explicit.

Compare your explanations

Read. df itself, through one .loc operation.
Predict. 70. The original example value was 65.
Explain. It writes through an intermediate object. Under pandas Copy-on-Write it cannot update the original DataFrame.

One quick check

Which expression clearly updates df?

What you should be able to explain. Make every data change explicit.

Your own assessment, separate from code checks

M25C04 · Explore · predict · practice

A correct total can still hide a bad merge

Two tables may contain the same key more than once. Matching repeated keys can multiply records. Validate the intended relationship and inspect unmatched IDs before calculating a report.

ReadWhat does validate='one_to_one' check?

PredictWhich IDs will be unmatched?

ExplainWhy is a left merge alone insufficient to find extra assessment IDs?

Read the example

left = students()[["Student_ID", "Name"]]
right = assessment()
result = left.merge(right, on="Student_ID", how="outer",
                    validate="one_to_one", indicator=True)
print(result.loc[result["_merge"].ne("both"), ["Student_ID", "_merge"]])
print("Rows:", len(result))

Verified example output · pandas 2.2.3

   Student_ID      _merge
11       S012   left_only
12       S013  right_only
Rows: 13

Watch for this. Do not remove duplicate keys without understanding why they exist.

Your turn

Keep only enrolled students with a left merge stored in result. Keep the merge indicator column, and store the count of students with no assessment in missing_count.

For the code check, keep these result variable names: missing_count, result. Different valid approaches are welcome.

Your Python code

left = students()[["Student_ID", "Name"]]
right = assessment()
result = left.merge(right, on="Student_ID", how="outer",
                    validate="one_to_one", indicator=True)
print(result.loc[result["_merge"].ne("both"), ["Student_ID", "_merge"]])
print("Rows:", len(result))

Show a hint

Use how='left', validate='one_to_one', and indicator=True.

Show one worked solution

result = students().merge(assessment(), on="Student_ID", how="left",
                          validate="one_to_one", indicator=True)
missing_count = result["_merge"].eq("left_only").sum()
print(result[["Student_ID", "Quiz", "_merge"]])
print("Enrolled students without assessment:", missing_count)

The result has 12 enrolled students and one missing assessment, S012. The earlier outer audit also exposed S013 for follow-up.

Compare your explanations

Read. The merge keys must be unique in each input table.
Predict. S012 appears only in students; S013 appears only in assessment.
Explain. It preserves left-side students but discards right-only records, including S013.

One quick check

What can a many-to-many merge do?

What you should be able to explain. Audit the join before trusting the summary.

Your own assessment, separate from code checks

Previous: Plot and interpret Next: End-to-end classroom project

Further detail: official documentation for this topic.