Public data has no memory
Government data gets published, then overwritten. Keep every copy and compare them, and you end up with a history nobody else has. Notes from building Tributary and Ditchrider.
A lot of valuable public data is published in a way that forgets. The SEC posts a roster of every registered investment adviser each month. Western states post water-rights filings as lists and map layers. When the next version goes up, the old one is gone, or buried somewhere nobody will ever look.
The data is public. The history usually isn’t. And the history is where the useful information is: who registered, who left, who filed to change a water right, who let one lapse.
So both Tributary and Ditchrider start with the same boring job. They take a copy on a schedule, keep all of them, and compare each one to the last.
The source will surprise you
Rebuilding the SEC roster back to January 2024 turned up almost every way a source can misbehave:
- Months that don’t exist. October and November 2025 have no roster at all. The fix is to label those as gaps rather than pretend nothing changed.
- A month that’s really another month. The September 2025 file is a byte-for-byte re-post of August. Diffing it would report zero changes, which is wrong in a quieter way. The pipeline skips identical files.
- A format change. Rosters before December 2025 are Excel files, and later ones are CSV. The Excel dates are stored as serial numbers. A small reader handles both, so the rest of the pipeline never has to know.
- A column that disappears. One month is missing a status field the others have. It’s treated as optional instead of failing the run.
- A calendar spike. Every April, annual amendments push the change count to roughly ten times a normal month. “Typical” on the site means the median, not the average.
None of these are exotic. They’re what happens when a file gets produced by hand, somewhere, every month, for years.
Heal instead of failing
The fetch step doesn’t just grab the latest file. It grabs the newest few months, plus any month since the start that doesn’t have a snapshot yet. So a fresh run on a clean machine rebuilds the entire history by itself, and a missed month fills in the next time the job runs. Nobody has to remember anything.
Keep it small enough to keep forever
Ditchrider snapshots water-rights sources for Utah, Idaho, Colorado and Montana every night, and the archive grows fast. Montana’s pending-application bundle alone runs about 3 GB, over GitHub’s 2 GB limit for a single release file. The answer was to store each document once, named by a hash of its contents. When a document doesn’t change, it costs nothing to snapshot again. When it does, only the new version is added.
Decide the kill line before you build
Before building Ditchrider’s ranker, it had a written test: if the four states didn’t produce at least 200 rankable filings in a month, stop. The first 30 days produced 1,044. Setting the line first matters more than the number. It turns “this feels promising” into a yes or no you can’t argue with later.
Why start early
Almost no state publishes daily or weekly history. An archive started today is worth a little more every day it runs, and it can’t be recreated later by someone who starts next year. That’s the whole moat, and it costs one scheduled job to begin.