Data Sources

Last reviewed: 2026-09-07

breeding is built from two independent sources. Neither is authoritative on its own, and the fact that they were assembled from different origins is what makes their agreement meaningful.

Bulk data lives under %BREEDING_DATA%, defaulting to Z:\Breedinginfill. copy_data_to_z.bat puts it there. There is deliberately no silent fallback to the project directory: if the share is not mounted the run stops, rather than quietly reading a stale local copy that nobody has updated since the move.

---

1. PedigreeQuery

The primary source, and the only one with global coverage.

Acquisition and parse chain

pedigreequery.com pages           saved HTML, one file per horse
   -> PQData\ToProcess\<xx>\<name>.html      ~2.97 M files
   -> parser + integrity checks
   -> rs.source_files                        2,815,483 rows
   -> rs.parsed_horses                       71,825,680 rows, 162 GB

Each saved page is a horse's pedigree chart, so parsing one page yields the horse and every ancestor shown on it. parsed_horses therefore holds one row per mention, not per horse.

Measure	Value
Mentions	71,825,680
Mentions per distinct horse (measured)	23.6
Implied distinct horses	~3.0 M
Storage	107 GB data + 55 GB index = 162 GB, 12 indexes

96 % of parsed_horses is the same horses said again. The current pipeline reads it once, collapses it to one row per key_name in a scratch table, and never touches payload_json or profile_json — which is the difference between a 162 GB scan and a ~20 GB one.

The 5 September load produced 2,496,498 rows from this source in 1.6 minutes.

What PedigreeQuery is good at

• Coverage. Every major racing jurisdiction, back to foundation stock. The table's oldest identities are 18th-century.
• Identity disambiguation. PedigreeQuery already separates horses sharing a registered name — GOLD DIGGER2, TIEPOLO3, ACT OF WAR4 — and does so consistently across every page. Tested by grouping mentions by key_name and counting distinct values of each identity attribute across 20,292 identities: one conflicting identity, 0.005 %.
• Depth. Pages carry five generations, so an ancestor with no page of its own still enters the table from its descendants' pages.

Where it is weak

Weakness	Detail
Year only, no date	PedigreeQuery publishes a foaling year. foal_date is NULL on every PQ-sourced row. NULL is an accepted permanent state here, not a gap to report against.
Missing sire pages	24,986 of 112,295 sires (22.2 %) have no page of their own — including NORTHERN DANCER, NATIVE DANCER, SECRETARIAT, NASRULLAH, HYPERION, PHALARIS. They enter correctly; what is lost is depth behind them. Dams are 1.7 %.
~YYY- truncation	The "approximate year, unknown final digit" notation is over-trimmed by one character: ~184- parses as 184. This is why the table's minimum year is 182. Not repairable from the database — year_text already holds the damaged value.
Markup leak	Some publish_name values carry ONMOUSEOUT fragments. Caught by rule R5 at load.
Turkish charset	Windows-1254 read as Latin-1. Parser-level; takes effect on re-import only.
"Missing horse" pages	47,445 files recorded as parse errors. A random sample of 400, reopened against the source site, found 399 were PedigreeQuery's expected "missing horse" page rather than a parser failure. Extrapolated genuine failures across all 47,445: roughly 100–120. Not worth chasing individually.

Generation 5

Generation 5 is deliberately ignored: its parent links are not reliably represented by the source page.

---

2. Australian Studbook

The depth source for Australia and New Zealand, and the only source of exact foaling dates.

Form

%BREEDING_DATA%\Studbook\newAUData\**\horses_*.jsonl.gz

134 gzipped JSONL files, ~58 MB compressed, 1,168,043 records. Each record carries the horse and, inline, its sire and dam as name + country + year. hid is a real studbook id; it is not stored — identity in breeding is the triple like everything else — but the file and hid are written to link_to_file so any row can be traced back:

horses_0042.jsonl.gz#hid=1173408

edges.jsonl.gz is a flattening of the same sire/dam objects and adds no information, so it is not read.

The 5 September load queued 998,626 records, of which 611,461 created new identities and 387,165 merged into identities PedigreeQuery had already made.

What the Studbook is good at

• Exact foaling dates. 776,711 rows in breeding carry a real foal_date, and every one came from here. That is what makes the season-year rule provable and what arbitrates duplicate identities.
• Registered sex. is_male had zero male/female conflicts across the whole dump when tested against tblhorse.
• AU/NZ completeness. AUS is the second-largest country in the table at 756,439 rows, NZ 165,548.

Where it is weak

Weakness	Detail
The name suffix	67,284 records carry the country and year inside the name string — "The Hoe (AUS) 1972 ntb" — and have their country and year columns NULL. Rule R1 would drop all of them. split_suffix() recovers 98.0 % of them before the rules are applied.
Stallion vs gelding	The dump labels roughly half of all male horses stallion regardless of gelding. Measured on 66 matched AU/NZ males: 33 said stallion where the contemporaneous run records said gelding. The label is therefore not stored at all — see Breeding Rules §2.
Null date sentinels	00/00/YYYY is caught by an ordinary range check, but 01/01/1900 is a syntactically perfect date that passes every check. 47 records, all AU/NZ. Named explicitly as a sentinel.
Season-year on parent references	An AU-sourced reference to a northern-hemisphere horse as a parent quotes the season year, one lower than the horse's own record. Verified across 17,206 northern records with a real date of birth: no exceptions in any month. Handled by rule B pass 2b.
Unknown countries	359 records on the last run named a country code the country table does not carry.

---

3. Reference data

Table	Rows	Role
rs.country	240	country_id FK target. Verified identical between rs.country and qdb.country across all 240 rows, so no remapping is needed on ship.
rs.studbook_profiles	887,566	Legacy studbook profile table, used by the pre-rebuild enrichment and locking scripts.

The loaders build a code map of iso3 -> country.id plus the racing codes the sources actually use (NZ→NZL, GB→GBR, IRE→IRL, GER→DEU, SAF→ZAF, BRZ→BRA, ITY→ITA, CHI→CHL, URU→URY, UAE→ARE, ZIM→ZWE and others), giving 244 recognised codes.

qdb.country.racing_code was corrected on ODIN on 17 August 2026: 18 rows — 12 blanks filled, 6 ISO3 values replaced with real racing codes (SAU→KSA, MUS→MRI, MAR→MOR, OMN→OMA, SRB→SER, SGP→SIN). 240 rows, 67 codes, all distinct.

On record: the NED/NED duplication on Holland (91) and Netherlands (146) is inherited from data/qdb_country_seed.json, not a local bulk-fill regression as earlier documentation claimed. rs.country.racing_code is byte-identical to its source across all 240 rows.

---

4. What each source contributed

Row provenance in the live table, by the first source to claim the identity:

Source	Rows
PedigreeQuery HTML page	2,495,266
Australian Studbook JSONL	611,460

Plus 387,165 studbook records that merged into existing PQ rows. Those rows are dual-sourced, but link_to_file records only the first claimant, so the table itself does not say so — which is precisely why reconcile_sources.py re-reads the dump rather than trying to compute agreement from the table.

5. Legacy staging, now dead weight

Table	Rows	Size	Status
rs.parsed_horses	71,825,680	162 GB	Still the input to step 1; a re-parse cache thereafter
rs.import_decisions	80,704,431	64 GB	No remaining consumer
rs.breeding_candidate	501,596	0.2 GB	Superseded — staging table of the retired pipeline

Roughly 226 GB. Keep-or-drop is an open decision; the disk is real.