Pandas Library

pandas is numpy with labels. a Series is a 1-d array married to an index; a DataFrame is a dict of such columns sharing one row index. the single organising idea — the one that explains both the magic and the bugs — is that every operation aligns on labels first and computes second. everything on this page runs against pandas 3.0 (outputs are real). 𐃏

the model: alignment is the core idea

positional arithmetic is what numpy already does; pandas instead matches labels, outer-joining the two indices and inserting NaN where either side is silent:

import pandas as pd

s1 = pd.Series([1.0, 2.0, 3.0], index=["a", "b", "c"])
s2 = pd.Series([10.0, 20.0, 30.0], index=["b", "c", "d"])
print(s1 + s2)

a     NaN
b    12.0
c    23.0
d     NaN
dtype: float64

nothing lined up by position: b met b and c met c regardless of where they sat, and the unmatched labels surfaced as missing data instead of silently pairing wrong values. this is the entire value proposition — merge two months of data with different row orders, subtract a baseline keyed by id, divide by a per-group total: alignment does the bookkeeping that positional code gets subtly wrong. the price: an unexpected NaN usually means your labels disagreed, not that data vanished — check indices before reaching for fillna.

indexing: loc, iloc, and the trap

three access idioms, strictly separated:

df.loc[label_rows, label_cols] — by label (slices are inclusive of the endpoint).
df.iloc[i, j] — by position, numpy semantics.
df[mask] / df.loc[mask, cols] — boolean filtering.

import warnings
import pandas as pd

df = pd.DataFrame({"x": [1, -2, 3], "y": [10, 20, 30]},
                  index=["r0", "r1", "r2"])
print(df.loc["r1", "y"], df.iloc[1, 1])     # label vs position: same cell
print(df.loc[df.x > 0, "y"].tolist())       # boolean + column, one step

# the chained-assignment trap (pandas 3 = copy-on-write):
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    df[df.x > 0]["y"] = 99                  # two steps: filter THEN assign
    print("warned:", w[0].category.__name__)
print("y unchanged:", df.y.tolist())

df.loc[df.x > 0, "y"] = 99                  # one step: works
print("y now      :", df.y.tolist())

20 20
[10, 30]
warned: ChainedAssignmentError
y unchanged: [10, 20, 30]
y now      : [99, 20, 99]

the infamous SettingWithCopyWarning era ended with pandas 3: under copy-on-write, df[mask] always behaves as a copy, so chained assignment never writes through — you get a ChainedAssignmentError warning and no effect, instead of the old maybe-it-worked lottery. 𐃏 the rule was and remains: one indexing operation per assignment, df.loc[rows, cols] = value.

groupby: split, apply, combine

split-apply-combine: the frame is partitioned on the key, a function hits each part, and the parts are stitched back. agg shrinks each group to a row; transform returns a like-indexed frame; filter keeps or drops whole groups.

the three verbs differ in output shape, and choosing the wrong one is the classic groupby bug:

agg — group \(\to\) scalar(s). result has one row per group.
transform — group \(\to\) like-length column, broadcast back to the original index (the tool for “divide by group total” features).
filter — group \(\to\) boolean. keeps or drops whole groups, original shape otherwise untouched.

import numpy as np
import pandas as pd

sales = pd.DataFrame({
    "shop":  list("AABBBCC"),
    "item":  ["tea", "coffee", "tea", "coffee", "tea", "tea", "coffee"],
    "units": [30, 21, 44, 12, 20, 52, 33],
})

g = sales.groupby("shop")["units"]
print(g.agg(["sum", "mean"]))                    # agg: one row per group
sales["shop_share"] = g.transform(lambda u: u / u.sum())  # transform: same length
print(sales.head(4))
big = sales.groupby("shop").filter(lambda d: d.units.sum() > 60)
print("filter keeps shops:", sorted(big.shop.unique()))

      sum       mean
shop
A      51  25.500000
B      76  25.333333
C      85  42.500000
  shop    item  units  shop_share
0    A     tea     30    0.588235
1    A  coffee     21    0.411765
2    B     tea     44    0.578947
3    B  coffee     12    0.157895
filter keeps shops: ['B', 'C']

merge and join

relational algebra over frames. merge is the general tool; the four joins differ only in which unmatched keys survive:

import pandas as pd

left  = pd.DataFrame({"key": ["a", "b", "c"], "l": [1, 2, 3]})
right = pd.DataFrame({"key": ["b", "c", "d"], "r": [20, 30, 40]})

for how in ["inner", "left", "right", "outer"]:
    m = left.merge(right, on="key", how=how)
    print(f"{how:>6}: keys={m.key.tolist()}")

# validation + provenance
m = left.merge(right, on="key", how="outer", validate="one_to_one", indicator=True)
print(m)

 inner: keys=['b', 'c']
  left: keys=['a', 'b', 'c']
 right: keys=['b', 'c', 'd']
 outer: keys=['a', 'b', 'c', 'd']
  key    l     r      _merge
0   a  1.0   NaN   left_only
1   b  2.0  20.0        both
2   c  3.0  30.0        both
3   d  NaN  40.0  right_only

two habits that pay for themselves: validate="one_to_one" (or "one_to_many" etc.) makes the merge assert its own cardinality — a silently duplicated key is the most expensive bug in data work, because row counts explode after the join and every downstream aggregate doubles; 𐃏 and indicator=True records where each row came from, which turns “why is this NaN” from archaeology into a groupby.

reshape: pivot, melt, stack

wide and tall are the two habitable phases of tabular data; these operators move between them:

import pandas as pd

sales = pd.DataFrame({
    "shop":  list("AABBBCC"),
    "item":  ["tea", "coffee", "tea", "coffee", "tea", "tea", "coffee"],
    "units": [30, 21, 44, 12, 20, 52, 33],
})
wide = sales.pivot_table(index="shop", columns="item",
                         values="units", aggfunc="sum", fill_value=0)
print(wide)
tall = wide.reset_index().melt(id_vars="shop", value_name="units")
print(tall.head(3))
print(wide.stack().head(4))     # columns -> inner index level

item  coffee  tea
shop
A         21   30
B         12   64
C         33   52
  shop    item  units
0    A  coffee     21
1    B  coffee     12
2    C  coffee     33
shop  item
A     coffee    21
      tea       30
B     coffee    12
      tea       64
dtype: int64

pivot_table = groupby + reshape (it aggregates duplicates; plain pivot refuses them). melt is its inverse, back to tidy one-observation-per-row form. stack and unstack are the same moves phrased as index-level rotation — a pivot is just groupby + unstack in a trenchcoat.

time series

the index earns its keep hardest when it is a DatetimeIndex: resampling is “groupby over time bins”, rolling is a sliding window with alignment handled for you.

import numpy as np
import pandas as pd
rng = np.random.default_rng(0)

idx = pd.date_range("2026-01-01", periods=90, freq="D")
ts = pd.Series(50 + np.arange(90) * 0.3 + rng.normal(0, 4, 90), index=idx)

print(ts.resample("MS").mean().round(2))          # downsample to month starts
roll = ts.rolling(window=7, center=True).mean()   # 7-day smoother
print("rolling covers", roll.notna().sum(), "of", len(ts), "days")
print("jan mean vs mar mean: %.1f -> %.1f" %
      (ts["2026-01"].mean(), ts["2026-03"].mean()))

2026-01-01    53.90
2026-02-01    64.77
2026-03-01    72.79
Freq: MS, dtype: float64
rolling covers 84 of 90 days
jan mean vs mar mean: 53.9 -> 72.8

note the partial-string indexing (ts["2026-01"] selects the whole month) and the honest edges: a centred 7-day window cannot cover the first and last three days, and rolling says so with NaN rather than inventing data.

method chaining style

mutate-in-place scripts accumulate mystery state; the chaining idiom makes each frame a value and the pipeline a single readable expression — the fp instinct (see functional programming) applied to tables:

import numpy as np
import pandas as pd
rng = np.random.default_rng(0)

raw = pd.DataFrame({
    "shop": rng.choice(["A", "B", "C"], 6),
    "units": rng.integers(1, 50, 6),
    "item": ["Tea ", "coffee", "TEA", " Coffee", "tea", "coffee "],
})

clean = (raw
         .assign(item=lambda d: d.item.str.strip().str.lower())
         .query("units > 5")
         .pipe(lambda d: d.groupby(["shop", "item"], observed=True)
                          .units.sum()
                          .reset_index()))
print(clean)

  shop    item  units
0    A  coffee     85
1    A     tea     32
2    B     tea      9

assign adds derived columns without mutation, query filters with a readable mini-language, pipe slots any frame-to-frame function into the chain. debugging tip: comment out the tail of the chain and inspect the prefix — the pipeline structure makes bisection trivial.

performance

import numpy as np, pandas as pd, timeit
rng = np.random.default_rng(0)

n = 1_000_000
cities = pd.Series(rng.choice(["sydney", "melbourne", "brisbane"], n))
as_obj = cities.astype("object")
as_cat = cities.astype("category")
print("object strings: %5.1f MB" % (as_obj.memory_usage(deep=True) / 1e6))
print("category      : %5.1f MB" % (as_cat.memory_usage(deep=True) / 1e6))

t = lambda f: timeit.timeit(f, number=5) / 5 * 1e3
print("upper via .apply(str.upper): %6.1f ms" % t(lambda: as_obj.apply(str.upper)))
print("upper via .str.upper()     : %6.1f ms" % t(lambda: as_obj.str.upper()))

# when pandas overhead dominates, drop to numpy
s = pd.Series(rng.standard_normal(n))
a = s.to_numpy()                       # zero-copy view of the same buffer
t_pd = timeit.timeit(lambda: (s - s.mean()) / s.std(), number=10) / 10
t_np = timeit.timeit(lambda: (a - a.mean()) / a.std(), number=10) / 10
print("standardise: pandas %5.2f ms | numpy %5.2f ms" % (t_pd * 1e3, t_np * 1e3))

object strings:  56.7 MB
category      :   1.0 MB
upper via .apply(str.upper): 107.1 ms
upper via .str.upper()     :  73.5 ms
standardise: pandas  6.09 ms | numpy  2.60 ms

three honest lessons from the measurements:

categoricals are the big win: a low-cardinality string column shrinks \(57\times\) (codes + a tiny dictionary), and groupbys on it get faster too. any id/label/state column qualifies.
vectorised string ops (.str.*) are tidier than .apply and comparable in speed (the ordering flips between runs — string work is loop-bound either way) — do not expect numpy-style \(65\times\) miracles from text.
dropping to numpy (.to_numpy() is zero-copy for a single numeric dtype) roughly halves purely numerical pipelines by shedding index-alignment overhead. inside a tight loop, that factor is the difference between interactive and annoying.

Machine Learning Example with Pandas

2026-06-28

ml pandas

Source: https://www.w3resource.com/python-exercises/pandas/pandas-machine-learning-integration.php

Structure of data.csv:

ID	Name	  Age	Gender	Salary	 Target
1,Sara,25,Female,50000,0
2,Ophrah,30,Male,60000,1
3,Torben,22,Male,70000,0
4,Masaharu,35,Male,80000,1
5,Kaya,NaN,Female,55000,0
6,Abaddon,29,Male,NaN,1
Column Description:

ID: A unique identifier for each record (integer). Name: The name of the individual (string). Age: Age of the individual (numerical, may have missing values). Gender: Gender of the individual (categorical: Male/Female). Salary: The individual’s salary (numerical, may have missing values). Target: The target variable for binary classification (binary: 0 or 1).
Read more >

Mastering Pandas

2026-06-28

sharpen

Source: https://www.w3resource.com/python-exercises/pandas/pandas_100_exercises_with_solutions.php

Exercise 1:

Create a DataFrame from a dictionary of lists.

Exercise 2:

Select the first 3 rows of a DataFrame.

Exercise 3:

Select the ‘X’ column from a DataFrame.

Exercise 4:

Filter rows based on a column condition.

Exercise 5:

Add a new column to an existing DataFrame.

Exercise 6:

Remove a column from a DataFrame.

Backlinks (4)

1. Wiki `/wiki/`

2026-07-10

Knowledge is a paradox. The more one understand, the more one realises the vastness of his ignorance.

2. Matplotlib Library `/wiki/ccs/programming/libraries/matplotlib/`

2026-07-10

figures axes artists visualisation

matplotlib has a reputation for being clunky, and the reputation is earned exactly by people fighting the wrong api. there are two: a stateful pyplot layer that mimics matlab (“draw on whatever was touched last”), and an object-oriented core in which every visible thing is an object you hold a reference to. the second is the real library; the first is sugar for one-liners. write fig, ax = plt.subplots() and stay in object land, and most of the clunk evaporates. 𐃏 all code on this page executed for real; figures were written to /tmp and are described rather than embedded, with true file sizes.

3. Numpy `/wiki/ccs/programming/libraries/numpy/`

2026-07-10

ndarray broadcasting vectorisation strides

numpy is one idea executed ruthlessly: a typed, contiguous block of memory plus enough metadata to reinterpret it without copying. everything else — broadcasting, views, vectorised speed — falls out of that design. learn the memory model and the api stops being a list of functions and becomes a handful of consequences. 𐃏 all timings and outputs on this page are real (python 3, numpy 2.4, apple m-series).

4. Libraries `/wiki/ccs/programming/libraries/`

2026-06-28

Pandas Library

the model: alignment is the core idea#

indexing: loc, iloc, and the trap#

groupby: split, apply, combine#

merge and join#

reshape: pivot, melt, stack#

time series#

method chaining style#

performance#

see also#

Machine Learning Example with Pandas

Mastering Pandas

Exercise 1:#

Exercise 2:#

Exercise 3:#

Exercise 4:#

Exercise 5:#

Exercise 6:#

Backlinks (4) This web pageLive render · backlinks includedTypeset PDFLaTeX pre-render

1. Wiki /wiki/

2. Matplotlib Library /wiki/ccs/programming/libraries/matplotlib/

3. Numpy /wiki/ccs/programming/libraries/numpy/

4. Libraries /wiki/ccs/programming/libraries/

the model: alignment is the core idea

indexing: loc, iloc, and the trap

groupby: split, apply, combine

merge and join

reshape: pivot, melt, stack

time series

method chaining style

performance

see also

Exercise 1:

Exercise 2:

Exercise 3:

Exercise 4:

Exercise 5:

Exercise 6:

Backlinks (4)

1. Wiki `/wiki/`

2. Matplotlib Library `/wiki/ccs/programming/libraries/matplotlib/`

3. Numpy `/wiki/ccs/programming/libraries/numpy/`

4. Libraries `/wiki/ccs/programming/libraries/`