Module 18 · Core

Concatenate and align

Combine rows or columns while respecting index labels.

Before you start: Understand DataFrame indexes and missing values.

Python loads from this site on your first run.

Each run starts with fresh data. Download any exported files before another run. Your written answers and code drafts are saved on this device; export your work to keep a backup.

M18C01 · Explore · predict · practice

Stack row batches

concat stacks tables when axis=0, the default. ignore_index=True assigns new position labels. This changes the index, not the Student_ID values.

ReadWhat is the list inside concat?

PredictHow many student rows will result have?

ExplainDoes ignore_index remove duplicate students?

Read the example

df = students()
first = df.iloc[:6]
second = df.iloc[6:]
result = pd.concat([first, second], ignore_index=True)
print(result[["Student_ID", "Name"]])

Verified example output · pandas 2.2.3

   Student_ID    Name
0        S001   Amira
1        S002   Bilal
2        S003   Chloe
3        S004    Dana
4        S005   Ethan
5        S006  Fatima
6        S007   Grace
7        S008  Hassan
8        S009     Ivy
9        S010   Jamal
10       S011  Khalid
11       S012    Lina

Watch for this. Build a list of batches and concatenate once instead of repeatedly growing a DataFrame in a loop.

Your turn

Stack the second batch before the first, and reset the index in the combined result.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

df = students()
first = df.iloc[:6]
second = df.iloc[6:]
result = pd.concat([first, second], ignore_index=True)
print(result[["Student_ID", "Name"]])

Show a hint

Reverse the order of the two DataFrames passed to concat.

Show one worked solution

df = students()
result = pd.concat([df.iloc[6:], df.iloc[:6]], ignore_index=True)
print(result[["Student_ID", "Name"]])

S007 is first because concat keeps the order of the input batches.

Compare your explanations

Read. It contains the two DataFrames to combine.
Predict. 12, the 6 rows from each batch.
Explain. No. It only creates a fresh integer index; duplicate data would still remain.

One quick check

What changes with ignore_index=True?

What you should be able to explain. Use concat for batches with the same meaning and schema.

Your own assessment, separate from code checks

M18C02 · Explore · predict · practice

Align columns by labels

concat with axis=1 places objects side by side. It matches index labels, even when their order differs. This is why an ID index is safer than assuming row positions match.

ReadWhy are the two Series given names?

PredictWhich score will Amira receive?

ExplainWhy is assigning scores.to_numpy() risky here?

Read the example

names = pd.Series(["Amira", "Bilal"], index=["S001", "S002"], name="Name")
scores = pd.Series([72, 88], index=["S002", "S001"], name="Score")
result = pd.concat([names, scores], axis=1)
print(result)

Verified example output · pandas 2.2.3

       Name  Score
S001  Amira     88
S002  Bilal     72

Watch for this. Resetting indexes before concatenating unrelated tables can hide a mismatched-key problem.

Your turn

Add score 91 for S003, then concatenate using join="inner" so only shared IDs remain.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

names = pd.Series(["Amira", "Bilal"], index=["S001", "S002"], name="Name")
scores = pd.Series([72, 88], index=["S002", "S001"], name="Score")
result = pd.concat([names, scores], axis=1)
print(result)

Show a hint

Construct a three-value scores Series and set join="inner".

Show one worked solution

names = pd.Series(["Amira", "Bilal"], index=["S001", "S002"], name="Name")
scores = pd.Series([72, 88, 91], index=["S002", "S001", "S003"], name="Score")
result = pd.concat([names, scores], axis=1, join="inner")
print(result)

S003 is excluded because its label is absent from names.

Compare your explanations

Read. Their names become output column labels.
Predict. 88, because both values use index label S001.
Explain. It discards labels, so the first score, 72, would be assigned by position to Amira.

One quick check

What does axis=1 concatenate side by side?

What you should be able to explain. Alignment follows labels, not visual row order.

Your own assessment, separate from code checks

M18C03 · Explore · predict · practice

Request a specific index order

reindex requests specific labels in a specific order. A requested label that does not exist produces missing values. It can also omit existing labels that were not requested.

ReadWhat controls the row order?

PredictWhat will appear for S099?

ExplainHow is this different from loc with a missing label?

Read the example

df = students().set_index("Student_ID")
result = df.reindex(["S003", "S001", "S099"])[["Name", "Score"]]
print(result)

Verified example output · pandas 2.2.3

             Name  Score
Student_ID              
S003        Chloe   91.0
S001        Amira   88.0
S099          NaN    NaN

Watch for this. Reindexing generally requires unique labels on the axis being reindexed.

Your turn

Reindex to S012, S002, S001 and print the Name and Score columns in that order.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

df = students().set_index("Student_ID")
result = df.reindex(["S003", "S001", "S099"])[["Name", "Score"]]
print(result)

Show a hint

Supply a different list of labels to reindex.

Show one worked solution

df = students().set_index("Student_ID")
result = df.reindex(["S012", "S002", "S001"])[["Name", "Score"]]
print(result)

All three labels exist, so this result has no missing values.

Compare your explanations

Read. The label list supplied to reindex.
Predict. Missing Name and Score values because the ID does not exist.
Explain. With a list of IDs, loc raises KeyError if any requested ID is absent. reindex can create rows with missing values for absent labels. Label slices have different endpoint rules.

One quick check

What does an absent label create during reindex?

What you should be able to explain. Reindex is an explicit request for labels and order.

Your own assessment, separate from code checks

M18C04 · Explore · predict · practice

Fill from a second source

combine_first fills missing values from matching labels in another object. Existing non-missing values in the first source win. Agree which source has priority before applying this rule.

ReadWhich source has priority?

PredictWill S001 become 85?

ExplainWhy is this different from an unconditional overwrite?

Read the example

primary = pd.Series([88, pd.NA, 91], index=["S001", "S002", "S003"], dtype="Int64")
backup = pd.Series([85, 72, 90], index=["S001", "S002", "S003"], dtype="Int64")
result = primary.combine_first(backup)
print(result)

Verified example output · pandas 2.2.3

S001    88
S002    72
S003    91
dtype: Int64

Watch for this. An unmatched backup label can expand the index; inspect the result size.

Your turn

Reverse the source priority by calling combine_first on backup. Explain the S001 result.

For the code check, keep these result variable names: result. Different valid approaches are welcome.

Your Python code

primary = pd.Series([88, pd.NA, 91], index=["S001", "S002", "S003"], dtype="Int64")
backup = pd.Series([85, 72, 90], index=["S001", "S002", "S003"], dtype="Int64")
result = primary.combine_first(backup)
print(result)

Show a hint

Place backup before the dot.

Show one worked solution

primary = pd.Series([88, pd.NA, 91], index=["S001", "S002", "S003"], dtype="Int64")
backup = pd.Series([85, 72, 90], index=["S001", "S002", "S003"], dtype="Int64")
result = backup.combine_first(primary)
print(result)

Backup now wins wherever it has a value; S001 is 85.

Compare your explanations

Read. primary, because combine_first is called on it.
Predict. No. Its primary score 88 is already present.
Explain. Only missing primary entries are filled; existing values remain.

One quick check

Which primary values are replaced by combine_first?

What you should be able to explain. Filling from another source is a policy as well as a method.

Your own assessment, separate from code checks

Previous: Merge related tables Next: Reshape tables

Further detail: official documentation for this topic.