Module 29 · Apply

Analyze a real public dataset

Produce a traceable descriptive analysis with justified exclusions and visible denominators.

Before you start: You can clean, group, plot, and export data; review the public-data README and field dictionary.

Python loads from this site on your first run.

Each run starts with fresh data. Download any exported files before another run. Your written answers and code drafts are saved on this device; export your work to keep a backup.

M29C01 · Explore · predict · practice

Check the source and audit missingness

Begin a public-data analysis by identifying the source, version, and field definitions. A checksum can confirm that the local source file matches the recorded bytes, while a missingness audit tells you what is available to analyze. These checks do not prove that every scientific measurement is correct.

Read the example

import hashlib
folder = DATA_DIR / "public"
metadata = json.loads((folder / "provenance.json").read_text(encoding="utf-8"))
digest = hashlib.sha256((folder / "penguins.csv").read_bytes()).hexdigest()
assert digest == metadata["artifacts"]["penguins.csv"]["sha256"]
df = pd.read_csv(folder / "penguins.csv", na_values=["NA"])
result = pd.DataFrame({"Missing": df.isna().sum(), "Known": df.notna().sum()})
print("Dataset:", metadata["dataset"])
print("Source commit:", metadata["upstream"]["commit"])
print("Shape:", df.shape)
print(result)
Verified example output · pandas 2.2.3
Dataset: Palmer Penguins, simplified penguins table
Source commit: 8957207b78d6ccd1b4654a9dd9c9041b657478ab
Shape: (344, 8)
                   Missing  Known
species                  0    344
island                   0    344
bill_length_mm           2    342
bill_depth_mm            2    342
flipper_length_mm        2    342
body_mass_g              2    342
sex                     11    333
year                     0    344

Watch for this. A row position is a way to trace the source file, not an animal identifier.

Your turn

Read the source into df. Store a dictionary named result with keys total_rows, rows_with_any_missing, complete_rows, and rows_with_known_mass. Calculate those four counts across the source fields without deleting rows from df.

For the code check, keep these result variable names: df, result. Different valid approaches are welcome.

Show a hint

Use isna().any(axis=1), dropna() only for counting, and body_mass_g.notna().sum().

Show one worked solution
df = pd.read_csv(DATA_DIR / "public" / "penguins.csv", na_values=["NA"])
result = {
    "total_rows": len(df),
    "rows_with_any_missing": int(df.isna().any(axis=1).sum()),
    "complete_rows": len(df.dropna()),
    "rows_with_known_mass": int(df["body_mass_g"].notna().sum()),
}
print(result)
print("The original DataFrame still has", len(df), "rows.")

There are 344 rows, 11 with at least one missing field, 333 complete rows, and 342 with known body mass. Complete-case filtering would discard nine additional known body masses because another field is missing.

Compare your explanations
  1. Read. The local file has the exact bytes recorded in the provenance manifest, assuming the manifest itself is trusted.
  2. Predict. sex has 11 missing entries, compared with two missing body masses.
  3. Explain. A file can be an exact copy and still contain measurement limitations, sampling constraints, or source errors.

One quick check

Which statement is justified by a matching recorded checksum?

What you should be able to explain. Document source identity and data availability before reporting results.

M29C02 · Explore · predict · practice

Choose exclusions to match the question

A body-mass summary needs a known body mass and a grouping label. It does not automatically need recorded sex. Keep an exclusion audit and retain original units; this exercise does not replace missing measurements with invented values.

Read the example

df = pd.read_csv(DATA_DIR / "public" / "penguins.csv", na_values=["NA"])
df.insert(0, "source_row", np.arange(1, len(df) + 1))
required = ["species", "body_mass_g"]
missing_required = df[required].isna().any(axis=1)
audit = df.loc[missing_required, ["source_row"] + required].copy()
result = df.loc[~missing_required].copy()
assert result["body_mass_g"].gt(0).all()
print("Rows retained:", len(result))
print("Excluded rows:")
print(audit)
print("Retained rows with unrecorded sex:", result["sex"].isna().sum())
Verified example output · pandas 2.2.3
Rows retained: 342
Excluded rows:
     source_row species  body_mass_g
3             4  Adelie          NaN
271         272  Gentoo          NaN
Retained rows with unrecorded sex: 9

Watch for this. Report exclusions and consider whether the remaining observations differ systematically; these checks do not establish random missingness.

Your turn

Change the question to body-mass summaries by species and sex. Require species, sex, and body_mass_g. Recreate the retained result and exclusion audit from the original file; do not reuse the earlier filtered table.

For the code check, keep these result variable names: audit, result. Different valid approaches are welcome.

Show a hint

Add sex to required and rebuild the missing-required mask.

Show one worked solution
df = pd.read_csv(DATA_DIR / "public" / "penguins.csv", na_values=["NA"])
df.insert(0, "source_row", np.arange(1, len(df) + 1))
required = ["species", "sex", "body_mass_g"]
missing_required = df[required].isna().any(axis=1)
audit = df.loc[missing_required, ["source_row"] + required].copy()
result = df.loc[~missing_required].copy()
print("Rows retained:", len(result))
print("Rows excluded:", len(audit))
print(audit)

The revised question retains 333 records and excludes 11. The decision changed because the requested grouping now needs sex; it is not a rule that every analysis must use complete cases.

Compare your explanations
  1. Read. species and body_mass_g.
  2. Predict. No. Nine retained rows have a known mass but missing sex.
  3. Explain. It traces a record to its one-based position in the source table. It is not a biological identity or a repeated-measurement ID.

One quick check

Why keep some rows with missing sex for the original mass-by-species question?

What you should be able to explain. Define the required fields and preserve an exclusion audit.

M29C03 · Explore · predict · practice

Report group means with their denominators

A group mean uses the known measurements in that group. Show total observations, known measurements, and missing measurements beside the mean so readers can see its denominator. The summaries describe these observations; they do not establish causal species or island effects.

Read the example

df = pd.read_csv(DATA_DIR / "public" / "penguins.csv", na_values=["NA"])
result = df.groupby("species", as_index=False).agg(Observations=("species", "size"), Known_mass=("body_mass_g", "count"), Mean_mass_g=("body_mass_g", "mean"))
result["Missing_mass"] = result["Observations"] - result["Known_mass"]
print(result.round({"Mean_mass_g": 2}))
print("Total observed records:", result["Observations"].sum())
print("Total known body masses:", result["Known_mass"].sum())
Verified example output · pandas 2.2.3
     species  Observations  Known_mass  Mean_mass_g  Missing_mass
0     Adelie           152         151      3700.66             1
1  Chinstrap            68          68      3733.09             0
2     Gentoo           124         123      5076.02             1
Total observed records: 344
Total known body masses: 342

Watch for this. Sample sizes and species composition differ across islands. A descriptive comparison is not an adjusted causal analysis.

Your turn

Group by island instead of species. Store result with island as a normal column, sorted in ascending island order, and report Observations, Known_mass, Missing_mass, and Mean_mass_g. Reconcile the total observations and known measurements with the full table.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Show a hint

Change the grouping key and keep the matching size/count denominators.

Show one worked solution
df = pd.read_csv(DATA_DIR / "public" / "penguins.csv", na_values=["NA"])
result = df.groupby("island", as_index=False).agg(Observations=("island", "size"), Known_mass=("body_mass_g", "count"), Mean_mass_g=("body_mass_g", "mean"))
result["Missing_mass"] = result["Observations"] - result["Known_mass"]
assert result["Observations"].sum() == len(df)
assert result["Known_mass"].sum() == df["body_mass_g"].count()
print(result.round({"Mean_mass_g": 2}))

Biscoe has 168 observations and 167 known masses; Dream has 124 and 124; Torgersen has 52 and 51. Their mean masses are approximately 4,716.02 g, 3,712.90 g, and 3,706.37 g. Different species composition is one reason to avoid interpreting these as an isolated island effect.

Compare your explanations
  1. Read. 151 known body masses, not all 152 Adelie observations.
  2. Predict. Chinstrap: all 68 observations have known body mass.
  3. Explain. No. They are descriptive summaries; causal explanations require an appropriate design and relevant evidence.

One quick check

What should accompany a mean when values are missing?

What you should be able to explain. Pair every summary with its relevant count and interpretation.

M29C04 · Explore · predict · practice

Create a chart and an auditable export

Combine the source, missingness policy, grouped summary, chart, and export into one repeatable cell. Keep counts in the exported table and label the plotted units. A chart can summarize the sampled measurements without claiming a population estimate or a causal finding.

Read the example

df = pd.read_csv(DATA_DIR / "public" / "penguins.csv", na_values=["NA"])
result = df.groupby("species", as_index=False).agg(Observations=("species", "size"), Known_mass=("body_mass_g", "count"), Mean_mass_g=("body_mass_g", "mean"))
result["Missing_mass"] = result["Observations"] - result["Known_mass"]
result["Mean_mass_kg"] = result["Mean_mass_g"] / 1000
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(result["species"], result["Mean_mass_kg"], color="#326f79")
for i, row in result.iterrows():
    ax.text(i, row["Mean_mass_kg"] + 0.08, f"n={int(row['Known_mass'])}", ha="center")
ax.set(title="Mean observed penguin body mass, 2007–2009", xlabel="Species", ylabel="Mean body mass (kg)", ylim=(0, 6))
fig.tight_layout()
csv_path = OUTPUT_DIR / "public_penguins_mass_summary.csv"
plot_path = OUTPUT_DIR / "public_penguins_mass_summary.png"
result.to_csv(csv_path, index=False)
fig.savefig(plot_path, dpi=160)
print(result.round({"Mean_mass_g": 2, "Mean_mass_kg": 3}))
print("Source: Palmer Penguins; DOI 10.5281/zenodo.3960218. Means use known body masses.")
print("These summaries describe sampled observations, not causal effects or a population census.")
plt.show()
Verified example output · pandas 2.2.3
     species  Observations  Known_mass  Mean_mass_g  Missing_mass  Mean_mass_kg
0     Adelie           152         151      3700.66             1         3.701
1  Chinstrap            68          68      3733.09             0         3.733
2     Gentoo           124         123      5076.02             1         5.076
Source: Palmer Penguins; DOI 10.5281/zenodo.3960218. Means use known body masses.
These summaries describe sampled observations, not causal effects or a population census.
Verified example figure for Create a chart and an auditable export

Watch for this. Do not imply uncertainty intervals or representativeness that the chart has not calculated or established.

Your turn

Build result as a summary by species in ascending species order, with columns species, Observations, Known_length, Median_flipper_mm, and Missing_length. Plot the medians with known-count labels on axes named ax, using the y-axis label "Median flipper length (mm)". Set csv_path and plot_path to public_penguins_flipper_summary.csv and public_penguins_flipper_summary.png in OUTPUT_DIR, then export the table and chart.

For the code check, keep these result variable names: ax, csv_path, plot_path, result. Different valid approaches are welcome.

Show a hint

Change the measurement, summary method, labels, and filenames together; do not reuse the mass units.

Show one worked solution
df = pd.read_csv(DATA_DIR / "public" / "penguins.csv", na_values=["NA"])
result = df.groupby("species", as_index=False).agg(Observations=("species", "size"), Known_length=("flipper_length_mm", "count"), Median_flipper_mm=("flipper_length_mm", "median"))
result["Missing_length"] = result["Observations"] - result["Known_length"]
fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(result["species"], result["Median_flipper_mm"], color="#326f79")
for i, row in result.iterrows():
    ax.text(i, row["Median_flipper_mm"] + 3, f"n={int(row['Known_length'])}", ha="center")
ax.set(title="Median observed penguin flipper length, 2007–2009", xlabel="Species", ylabel="Median flipper length (mm)", ylim=(0, 250))
fig.tight_layout()
csv_path = OUTPUT_DIR / "public_penguins_flipper_summary.csv"
plot_path = OUTPUT_DIR / "public_penguins_flipper_summary.png"
result.to_csv(csv_path, index=False)
fig.savefig(plot_path, dpi=160)
print(result)
print("Source: Palmer Penguins; DOI 10.5281/zenodo.3960218. Medians use known lengths.")
print("The chart describes the available study observations; it is not a causal analysis.")
plt.show()

The median flipper lengths are 190 mm for Adelie, 196 mm for Chinstrap, and 216 mm for Gentoo, based on 151, 68, and 123 known measurements. The table and chart use the same units and denominators.

Compare your explanations
  1. Read. The known body masses used to calculate that species mean.
  2. Predict. No. Every mean is divided by the same positive factor of 1,000.
  3. Explain. They preserve the analysis denominator and make it easier to audit how the result was calculated.

One quick check

Which label correctly describes the plotted output?

What you should be able to explain. Deliver the result with its source, units, counts, and limits.

Further detail: official documentation for this topic.