Hashnode note up front: this is a data-wrangling article. It is not portfolio advice.

1. Normalize two quarter tables first

Assume you already parsed the information table XML into rows with at least:

• cusip
• nameofissuer
• value (USD thousands, as filed)
• sshprnamt (share count or principal)
• sshprnamt_type (SH / PRN)
• put_call (nullable)

import pandas as pd

def norm(df: pd.DataFrame, tag: str) -> pd.DataFrame:
    out = df.copy()
    out["cusip"] = out["cusip"].astype(str).str.strip().str.upper()
    out["value"] = pd.to_numeric(out["value"], errors="coerce")
    out["sshprnamt"] = pd.to_numeric(out["sshprnamt"], errors="coerce")
    out["put_call"] = out["put_call"].fillna("").astype(str).str.upper()
    out["leg"] = tag
    return out

q1 = norm(raw_q1, "q1")  # prior quarter
q2 = norm(raw_q2, "q2")  # current quarter

Put/call legs must stay in the key. Otherwise a long equity row collapses into a put row with the same CUSIP.

2. Build a stable join key

KEY = ["cusip", "put_call", "sshprnamt_type"]

def keyed(df: pd.DataFrame) -> pd.DataFrame:
    # One row per key; if duplicates exist, keep the largest value row and log
    d = df.sort_values("value", ascending=False)
    dup = d.duplicated(KEY, keep="first").sum()
    if dup:
        print(f"warning: dropping {dup} duplicate keys")
    return d.drop_duplicates(KEY, keep="first").set_index(KEY)

Duplicates happen when issuers use multiple voting/investment discretion rows. Decide a rule (max value, sum, or keep both with an extra key column). Silent sums hide bugs.

3. Outer-join and classify

a = keyed(q1).add_suffix("_q1")
b = keyed(q2).add_suffix("_q2")
m = a.join(b, how="outer")

def classify(r) -> str:
    v1, v2 = r.get("value_q1"), r.get("value_q2")
    if pd.isna(v1) and not pd.isna(v2):
        return "new"
    if not pd.isna(v1) and pd.isna(v2):
        return "exited"
    if v2 > v1 * 1.05:
        return "increased"
    if v2 < v1 * 0.95:
        return "decreased"
    return "unchanged"

m["state"] = m.apply(classify, axis=1)
m["delta_value"] = m["value_q2"].fillna(0) - m["value_q1"].fillna(0)

The 5% band around “unchanged” absorbs trivial mark-to-market noise when share counts barely move. Tune it; keep it explicit.

4. Share-count check beside value delta

Value can rise while shares fall (price move). For equity legs (SH), compare share deltas too:

eq = m[m.index.get_level_values("sshprnamt_type") == "SH"].copy()
eq["delta_shares"] = eq["sshprnamt_q2"].fillna(0) - eq["sshprnamt_q1"].fillna(0)
print(eq.groupby("state")["delta_value"].agg(["count", "sum"]).sort_index())

A cluster of increased with negative delta_shares is usually price, not a fresh buy. Label those before you write a narrative.

5. Sanity asserts worth keeping

assert m.index.is_unique
assert m["state"].isna().sum() == 0
# Every q2 row should appear
assert keyed(q2).index.difference(m.index).empty
print(m["state"].value_counts())

If new + exited explode between two adjacent quarters, check whether you mixed amendment vs original filings, or whether one file still has put/call blank while the other uses PUT/CALL.

6. Where a tracker saves time

Once the classifier works on one manager, the boring part is fetching the right accession for each quarter and keeping issuer names aligned. A lightweight filing tracker such as https://13f.chat is enough to jump to the latest accepted 13F and confirm you did not pair an amendment with the wrong original. The join logic above still belongs in your notebook; the tracker just reduces “wrong file” mistakes.

Takeaway

Cross-quarter 13F work is an outer join plus an explicit state machine. Keep put/call and share-type in the key, separate price effects from share effects, and assert index uniqueness before you chart anything. The code is short on purpose — most errors are in filing selection, not in pandas.

Education only — not investment advice.

This post shows a compact pandas pipeline: load weekly inventories, compute a trailing z-score, and flag weeks that are unusually high or low. The math is short; the gotchas are calendar alignment and units.

1. Shape of the series

Assume you already have a DataFrame with one row per week:

import pandas as pd

# df columns: date (Timestamp, week ending), invent_kb (thousand barrels)
df = df.sort_values("date").reset_index(drop=True)
df["invent_kb"] = pd.to_numeric(df["invent_kb"], errors="coerce")
print(df.tail())

EIA publishes several stocks series (commercial crude, excluding SPR, Cushing, etc.). Pick one and stick to it. Mixing SPR and commercial stocks is a classic reason two charts disagree.

2. Trailing mean and z-score

A 52-week window matches the seasonal habit of comparing "this week vs the past year":

WINDOW = 52

df["mean_52"] = df["invent_kb"].rolling(WINDOW, min_periods=WINDOW).mean()
df["std_52"] = df["invent_kb"].rolling(WINDOW, min_periods=WINDOW).std(ddof=0)
df["z_52"] = (df["invent_kb"] - df["mean_52"]) / df["std_52"]

Using ddof=0 keeps the z-score aligned with how many dashboard tools define population sigma. If you prefer sample sigma, switch to ddof=1; just be consistent when you compare charts.

3. Flag extremes without over-fitting

df["flag"] = pd.cut(
    df["z_52"],
    bins=[-float("inf"), -2, -1, 1, 2, float("inf")],
    labels=["very_low", "low", "normal", "high", "very_high"],
)

print(df.dropna(subset=["z_52"]).tail(8)[
    ["date", "invent_kb", "z_52", "flag"]
])

A z-score below -2 does not mean "buy oil." It means inventories are unusually low versus the trailing year. Combine it with the weekly change and with products (gasoline, distillate) before drawing a conclusion.

4. A tiny sanity check

assert df["date"].is_monotonic_increasing
assert df["invent_kb"].isna().mean() < 0.02
# After warm-up, z should be roughly centered
warm = df.dropna(subset=["z_52"])
print(warm["z_52"].mean(), warm["z_52"].std())

If the mean of z drifts far from zero, your window is too short or the series has a structural break (methodology change, SPR release programme, etc.).

5. Where to look at the finished chart

Once the pipeline runs, it helps to eyeball the same series on a maintained chart before you trust a trading narrative. I cross-check the weekly crude inventory path on CommoditiesChart (the crude inventories view) against my z_52 flags. If the chart shows a steep drawdown and my z-score is still near zero, I usually have a units or series-id bug rather than a market insight.

Takeaway

Rolling z-scores turn a noisy weekly inventory print into a relative statement: how unusual is this level? Keep the window explicit, keep SPR and commercial stocks separate, and verify the plot against a known chart before acting on a flag.

Education only — not trading advice.

Computing a ratio sounds like a one-liner. In practice most of the work is in getting two price series to line up properly, choosing a lookback window, and not fooling yourself with a backtest. This post walks through all three with pandas.

1. Load two price series

Any daily source works. For a reproducible example I'll use the front-month futures from yfinance, but the code below only assumes you end up with two pd.Series indexed by date.

import pandas as pd
import yfinance as yf

raw = yf.download(["GC=F", "SI=F"], start="2005-01-01", auto_adjust=False, progress=False)
close = raw["Close"].rename(columns={"GC=F": "gold", "SI=F": "silver"})
print(close.tail())

If you have spot prices from a CSV instead (LBMA fixes, a broker export, etc.), read them with parse_dates and index_col so the index is a real DatetimeIndex. That matters for every step below.

2. Align before you divide

The first trap: the two series don't always have the same trading days. Holidays differ between exchanges and data vendors, and some sources skip days with no fix. If you divide two misaligned series, pandas aligns on the index and silently produces NaN where either side is missing. That's actually the safe behaviour. The dangerous version is forward-filling everything first.

both = close.dropna(how="any")          # keep only days where both prices exist
ratio = (both["gold"] / both["silver"]).rename("gs_ratio")

missing = close.isna().any(axis=1).sum()
print(f"dropped {missing} days with a missing leg")

My rule of thumb:

• Drop days where either leg is missing for daily analysis.

• Forward-fill at most one day (ffill(limit=1)) if you need a continuous series for charting.

• Never forward-fill across long gaps. A stale silver price from a week ago divided by today's gold price is a made-up number.

Before writing analysis code, it helps to look at the long-run price history of each metal so you know what "normal" looks like; the gold and silver USD price charts here are a quick visual reference. It makes it much easier to spot a bad tick or a unit error (for example a silver series accidentally in cents) when your ratio suddenly prints 7,000.

3. Sanity-check units and outliers

A quick set of assertions saves a lot of debugging later:

assert ratio.between(10, 200).all(), ratio[~ratio.between(10, 200)]

daily_change = ratio.pct_change().abs()
spikes = daily_change[daily_change > 0.08]
print(spikes)

A ratio move of more than 8% in a day does happen (March 2020 had several), but each one is worth eyeballing. If the spike reverses completely the next day, it's usually a bad print in one leg rather than a real market move.

4. Rolling z-score

"The ratio is high" only means something relative to a window. A rolling z-score expresses today's value in standard deviations from its recent mean:

def rolling_z(s: pd.Series, window: int = 252) -> pd.Series:
    mean = s.rolling(window, min_periods=window).mean()
    std = s.rolling(window, min_periods=window).std()
    return ((s - mean) / std).rename(f"z_{window}")

z1 = rolling_z(ratio, 252)     # about one trading year
z3 = rolling_z(ratio, 756)     # about three years

Two details:

1. min_periods=window keeps the first year as NaN instead of computing a z-score from ten data points. Without it, the early values look dramatic and are meaningless.

2. Consider computing on np.log(ratio) instead of the raw ratio. Ratios are multiplicative; moving from 60 to 80 and from 80 to 106.7 are the same percentage move, and log space treats them that way.

import numpy as np
zlog = rolling_z(np.log(ratio), 252)

5. A backtest sanity check (not a strategy)

The natural next question is whether "buy silver vs gold when z > 2" works. I'm not going to claim it does; the point of this section is to show how to test the idea without lying to yourself.

signal = pd.Series(0, index=ratio.index)
signal[zlog > 2] = 1      # ratio stretched high: long silver / short gold
signal[zlog < -2] = -1    # ratio stretched low: long gold / short silver

# Trade on the NEXT day's return, never the same day
rel_ret = np.log(both["silver"]).diff() - np.log(both["gold"]).diff()
pnl = signal.shift(1) * rel_ret

print("days in market:", (signal != 0).mean().round(3))
print("annualized mean:", (pnl.mean() * 252).round(4))
print("annualized vol:", (pnl.std() * np.sqrt(252)).round(4))

The single most important line is signal.shift(1). Without the shift, you're using today's closing ratio to trade today's return, which is look-ahead bias, and almost any mean-reversion rule looks brilliant that way.

Other things to check before trusting any result:

• Costs. Futures roll costs and spreads are real. Subtract a few basis points per trade at minimum.

• Regime dependence. Split the sample (e.g. before and after 2013) and see whether the result survives in both halves.

• Window sensitivity. If the rule only works with a 252-day window and falls apart at 200 or 300, it's probably fitted to noise.

6. Plot it

import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 6), sharex=True)
ratio.plot(ax=ax1, lw=1)
ax1.set_ylabel("gold / silver")
zlog.plot(ax=ax2, lw=1)
for lvl in (-2, 2):
    ax2.axhline(lvl, ls="--", lw=0.8)
ax2.set_ylabel("z (log, 252d)")
plt.tight_layout()
plt.show()

Wrap-up

The ratio itself is a division. Getting it right means aligning dates before dividing, refusing to forward-fill long gaps, checking units with a couple of assertions, using a rolling z-score in log space, and shifting signals by a day before you look at any backtest number. The same pattern works for any pair of related series: Brent vs WTI, copper vs gold, or two share classes of the same company.

This post walks through a small parser built on the standard library plus requests and pandas, and the edge cases I had to handle to get clean numbers.

Step 1: Find the filings

EDGAR's submissions API returns a company's recent filings as JSON. The SEC asks every client to send a descriptive User-Agent with a contact email, and it rate-limits clients that don't (or that send more than 10 requests a second).

import requests

HEADERS = {"User-Agent": "Your Name your-email@example.com"}

def recent_13f(cik: int):
    url = f"https://data.sec.gov/submissions/CIK{cik:010d}.json"
    data = requests.get(url, headers=HEADERS, timeout=30).json()
    recent = data["filings"]["recent"]
    for form, acc, date in zip(recent["form"],
                               recent["accessionNumber"],
                               recent["filingDate"]):
        if form in ("13F-HR", "13F-HR/A"):
            yield form, acc, date

for row in list(recent_13f(1067983))[:4]:   # Berkshire Hathaway
    print(row)

Note that amendments (13F-HR/A) show up in the same list. Some amendments restate the whole table, others only add new holdings. The amendmentType element in the primary document tells you which, so don't blindly add an amendment's rows to the original.

Step 2: Locate the information table file

The file name isn't fixed. You'll see infotable.xml, form13fInfoTable.xml, or a numeric name. The reliable way is to read the filing index and pick the XML file that isn't primary_doc.xml:

def info_table_url(cik: int, accession: str) -> str:
    acc = accession.replace("-", "")
    base = f"https://www.sec.gov/Archives/edgar/data/{cik}/{acc}/"
    idx = requests.get(base + "index.json", headers=HEADERS, timeout=30).json()
    for item in idx["directory"]["item"]:
        name = item["name"]
        if name.lower().endswith(".xml") and name != "primary_doc.xml":
            return base + name
    raise LookupError("no information table found")

Step 3: Parse with namespaces

Every element in the information table lives in the namespace http://www.sec.gov/edgar/document/thirteenf/informationtable. Forget the namespace and find() silently returns None for everything. Some filers also use a prefix (ns1:infoTable) and others use a default namespace. Matching on the namespace URI handles both.

import xml.etree.ElementTree as ET
import pandas as pd

NS = {"it": "http://www.sec.gov/edgar/document/thirteenf/informationtable"}

def text(el, path):
    node = el.find(path, NS)
    return node.text.strip() if node is not None and node.text else None

def parse_info_table(xml_bytes: bytes) -> pd.DataFrame:
    root = ET.fromstring(xml_bytes)
    rows = []
    for t in root.findall("it:infoTable", NS):
        rows.append({
            "issuer": text(t, "it:nameOfIssuer"),
            "class": text(t, "it:titleOfClass"),
            "cusip": text(t, "it:cusip"),
            "value": int(text(t, "it:value") or 0),
            "shares": int(text(t, "it:shrsOrPrnAmt/it:sshPrnamt") or 0),
            "share_type": text(t, "it:shrsOrPrnAmt/it:sshPrnamtType"),
            "put_call": text(t, "it:putCall"),
            "discretion": text(t, "it:investmentDiscretion"),
            "vote_sole": int(text(t, "it:votingAuthority/it:Sole") or 0),
            "vote_shared": int(text(t, "it:votingAuthority/it:Shared") or 0),
            "vote_none": int(text(t, "it:votingAuthority/it:None") or 0),
        })
    return pd.DataFrame(rows)

ElementTree is enough here. These files are rarely more than a few megabytes, so there's no need for lxml or a streaming parser.

Step 4: The edge cases

Value units changed in 2023. Before the SEC's amendments took effect at the start of 2023, value was reported in thousands of dollars. Filings after that report whole dollars. If you compare a 2022 table with a 2024 table without adjusting, positions look like they grew 1,000x overnight. I normalize on the period of report:

def normalize_value(df: pd.DataFrame, period_of_report: str) -> pd.DataFrame:
    if period_of_report < "2022-12-31":
        df["value"] = df["value"] * 1000
    return df

Some filers got the change wrong for a quarter or two, so it's also worth a sanity check: divide value by shares and flag any row whose implied price is wildly off from the market price that quarter.

Duplicate CUSIPs. One position can be split across several rows, for example by otherManager or investment discretion. Group before you analyze:

holdings = (df[df["put_call"].isna()]
            .groupby(["cusip", "issuer", "class"], as_index=False)
            [["value", "shares"]].sum()
            .sort_values("value", ascending=False))

Options rows. Rows with putCall set to Put or Call report the underlying shares and value, not the option premium. Mixing them into long equity positions overstates exposure, which is why the snippet above filters them out. Keep them in a separate frame if you need them.

Principal amounts. sshPrnamtType is SH for shares and PRN for principal amount (convertible notes and similar). A PRN row's "shares" is a dollar face amount, so drop or separate these before computing share changes.

Step 5: Quarter-over-quarter changes

With two clean frames, position changes are a merge:

def diff_quarters(prev: pd.DataFrame, curr: pd.DataFrame) -> pd.DataFrame:
    m = prev.merge(curr, on="cusip", how="outer",
                   suffixes=("_prev", "_curr")).fillna(0)
    m["share_change"] = m["shares_curr"] - m["shares_prev"]
    m["status"] = "changed"
    m.loc[m["shares_prev"] == 0, "status"] = "new"
    m.loc[m["shares_curr"] == 0, "status"] = "exited"
    return m.sort_values("share_change")

Join on CUSIP, not on issuer name. Names vary between filings ("APPLE INC" vs "Apple Inc."), but the CUSIP for a given security class stays the same.

When I want to check my parser's output against a finished view of the same filings, I compare a few managers on https://13f.chat, which lays out the reported holdings for each fund. If my totals disagree, it's almost always one of the edge cases above, usually units or options rows.

A few more gotchas

• Confidential treatment. Managers can ask the SEC to keep some positions confidential for a while. Those holdings show up later in an amendment, so a table can be legitimately incomplete.

• 45-day lag. Filings are due 45 days after quarter end. Treat the data as a snapshot of the past, not live positions.

• Long US positions only. Shorts, most non-US listings and cash aren't on the form. A 13F is a partial picture of a portfolio.

• Be polite to EDGAR. Cache everything you download and stay under 10 requests per second, or your IP gets blocked for a while.

Wrap-up

The parser itself is short. Most of the work is in the details: the namespace, the 2023 change in value units, splitting out options and principal rows, and grouping duplicate CUSIPs. Handle those four and you can turn any 13F-HR information table into a tidy DataFrame you can trust.