FormCrawler

Last reconciled: 2026-08-10 AEST

This is the starting-point status index for FormCrawler. It reconciles the pushed repository, the separate Harness/Greyhound branch and worktree, the live-validation reports, and the QDG Knowledge Base.

Status key

• ✅: crawler code exists for this lifecycle. A tick does not hide a documented transport or archive limitation.
• ❌: the source was determined unavailable, unsuitable, wrong-discipline, or outside the permitted acquisition contract.
• ?: incomplete, partially validated, unresolved, or not yet implemented.
• Captured in Others means FormCrawler acquired that asset. Available means it was found and linked for later work but is not currently in the default crawler scope.

Git and validation baseline

Scope	Location	State at reconciliation	Verification
Unified main	main includes the unified T/H/G baseline plus the August re-examination	26 Thoroughbred, nine Harness, and two Greyhound adapters in independent discipline registries	209 tests passed; 37 unique source IDs
Historical Harness/Greyhound branch	490b0c9	Eight H/G adapters, including LeTROT and separate T/H/G registries	131 tests passed before integration; full-day artifact audits passed
Runtime	local machine	No FormCrawler crawler process was running at reconciliation	Canonical artifact and handoff audits are recorded below
Operations dashboard branch	codex/crawl-dashboard at 3319e66 (pushed)	Concurrent daily supervisor and local monitoring UI; pending integration with current unified main	139 dashboard-branch tests; first eight-worker run and 91-file artifact/handoff audit passed

main is now the single T/H/G starting point. New work should branch from the updated origin/main, not the historical discipline branch.

<!-- operations-dashboard:start -->

Concurrent operations dashboard

The pushed codex/crawl-dashboard branch adds run-day and a loopback browser dashboard while preserving independent Thoroughbred, Harness, and Greyhound source modules. A daily plan gives every source/phase its own subprocess, SQLite queue and retained log; 1-32 jobs may run concurrently, but shared host controls continue to serialize requests to the same website.

The dashboard can start runs, navigate retained run days, filter by discipline/source, and show requested-day meeting counts, files, queue progress, retries, failures and worker logs. Unsupported plans are explicit not_available outcomes, and failures remain isolated to their jobs. This development branch must be integrated with the authoritative unified main baseline before production use. See formcrawler-dashboard for the full operator guide. <!-- operations-dashboard:end -->

Thoroughbred

Country / source	Pre	Post	Others
Australia — Racing Australia	✅	✅	Captured: acceptances, gear changes, scratchings. National source covers ACT, NSW, NT, QLD, SA, TAS, VIC and WA.
New Zealand — LoveRacing	✅	✅	Combined meeting-lifecycle page; profiles and breeding links available.
United Kingdom — Racing Post	✅	✅	Profiles, breeding and form links available.
Korea — KRA	✅	✅	Horse and pedigree detail available.
Malaysia — Selangor Turf Club	✅	✅	Complete 12-race meeting card/results validated.
Qatar — QREC	✅	✅	Meeting/race documents and horse detail available.
Bahrain — BHRA	✅	✅	Meeting/race documents and horse detail available.
South Africa — Gallop	✅	✅	Meeting/day lifecycle captured; some additional targets unpublished.
South Africa — Gold Circle	?	❌	Available: pre-race PDF/calendar documents; no dependable machine post-race archive.
Turkey — TJK	✅	✅	Complete meeting lifecycle validated; profiles/breeding available.
Japan — Umanity JRA	✅	✅	Full-day 36 pre/post pairs validated; month calendar rechecked live on 2026-08-08.
Japan — Umanity NAR	✅	✅	Full-day 34 pre/post pairs validated; month calendar rechecked live on 2026-08-08.
Japan — JBIS	❌	?	Available: results and rich breeding/horse profiles; no suitable pre-race route established.
Japan — NAR Keiba	❌	❌	Source outage prevented a safe crawler; Umanity NAR is the working source.
Czechia — Dostihy JC	✅	✅	Full-day 28 pairs validated; horse/pedigree detail available.
Spain — Jockey Club Español	✅	✅	Captured: stewards content within combined results pages.
Canada — Equibase mirror workflow	✅	✅	One complete meeting pair validated; horse/profile links available.
Sweden — Swedish Horse Racing	✅	✅	Three pairs captured; three additional targets were not yet published.
Hungary — Kincsem Park gallop	✅	✅	Horse/profile and breeding fields available.
United Arab Emirates — Emirates Racing	✅	✅	Horse/profile and breeding fields available.
India — Bangalore Races	✅	✅	Available: fixtures, videos, ratings, archives and profiles.
India — Hyderabad Races	✅	✅	Available: trackwork, entries, handicaps, acceptances, declarations, ratings, swimming, veterinary remarks and incidents.
Argentina — Stud Book	✅	✅	Home-page Today/next-meeting cards plus guarded result calendar. Horse/profile links available.
Chile — Hipódromo Chile	✅	✅	Captured: official meeting-program JSON/PDF and rolling workouts JSON; horse/profile detail available.
Germany — Deutscher Galopp	✅	✅	rennkalender.php discovers s=S racecards and s=R results; both lifecycles validated.
Uruguay — Maroñas	?	✅	Captured: horse search/profile JSON with sire, dam and damsire.
Saudi Arabia — JCSA	?	✅	Post-race results validated; pre-race validator remained incomplete after three attempts.
USA — official Equibase	✅	✅	Crawler downloads card/summary HTML, full-meeting and race PDFs, and published GPS HTML. Licensed browser validation passed; unattended licensed transport remains pending.
Germany — German Racing	❌	❌	Transport failed; Deutscher Galopp is the working German source.
Italy — SNAI Thoroughbred	?	?	Signed/unstable routes remain unresolved.
Belgium — Belgian Galopp	?	?	Available: meeting/program documents; no adapter.
Chile / Uruguay / Brazil — AMR Channel	❌	❌	Authentication boundary; not bypassed.
Norway — RaceBets	❌	❌	Licensed commercial database; unsuitable crawler source.
Mauritius — MTC Jockey Club	❌	❌	Transport blocked. Available: workouts, trials, stewards, replays and search links.
Morocco — SOREC Galop	?	?	Stateful JSF unresolved. Available: programmes, VOD, bulletins and technical documents.
Netherlands — NDR Thoroughbred listing	❌	❌	Supplied route is Harness, not Thoroughbred.
Hong Kong — HKJC	?	?	Available: trackwork, vet/form/incidents, replays, sectionals and profiles; no completed adapter.
France — France Galop	?	?	Available: programme, agenda, ratings, breeders, sires and bulletins; result route unresolved.
India — IndiaRace	✅	✅	Current meeting cards/results plus trackwork; incidents, ratings, horse links and breeding fields available.
India — Racing Pulse	?	?	Stateful postback archive unresolved.
India — Delhi / DRC	?	?	Document-oriented source only in this pass.
India — RCTC	?	?	Current/document routes found; no completed adapter.
Austria	?	?	No source URL supplied.
Ireland	?	?	No Thoroughbred source URL supplied.
Peru	?	?	No source URL supplied.
Panama	?	?	No source URL supplied.
Mexico	?	?	No source URL supplied.

Harness

Country / source	Pre	Post	Others
Australia — Harness Racing Australia	✅	?	Available: trials, stewards reports, replays and horse profiles. Current cards passed 25/26; complete national result-day coverage remained partial.
New Zealand — HRNZ	✅	✅	Available: trial fields/results, workout fields/results, programmes and horse detail.
France — LeTROT	✅	✅	Available: qualifications, replays, publications and horse profiles. Code is committed and pushed on the Harness/Greyhound branch.
Denmark — Paris-Turf	❌	❌	Catalogue mismatch: French publication did not provide a reliable Denmark source.
Denmark — Swedish Horse Racing	✅	✅	First-party calendar, raceday and race JSON; Odense pre/post pair live-validated.
Belgium — Belgium Horse Racing	?	?	Strong next candidate. Available: race newspaper, rankings and qualification/test races.
Finland — Heppa	✅	✅	First-party range, meeting and start JSON; horse IDs/profile route and breeding data available.
Netherlands — NDR	✅	✅	Upcoming cards and monthly result archive resolve to stable printable meeting HTML; bulletins available.
Norway — RaceBets	❌	❌	Licensed commercial database; unsuitable crawler source.
Norway — Swedish Horse Racing	?	?	Historical official PDFs and pedigree details available; API unresolved.
Spain — Federació Balear de Trot	?	?	Strong next candidate. Available: history, bulletins, videos, programme PDFs and profiles.
Hungary — Kincsem Park trotting	?	?	Strong next candidate using the same platform pattern as Kincsem Greyhound.
Sweden — Travsport	?	?	Bot verification not bypassed; approved proxy remains a bounded future check.
Sweden — Swedish Horse Racing	?	?	Shared Angular/XHR source remains unresolved.
USA — USTA	?	?	Stateful ASP.NET session/postback workflow unresolved; Pathway profiles available separately.
Canada — Standardbred Canada	✅	✅	Available: judges reports, race dates, feature races and TrackIT profiles.
Germany — HVT	✅	✅	Approved proxy used. Available: monthly reports, videos, finish photos, official PDFs and pedigree.
Italy — SNAI Harness	✅	✅	Available: meeting PDFs, videos, qualification trials and profiles; historical pre cards are unavailable.
Austria	?	?	No Harness source URL supplied.
Malta	?	?	No Harness source URL supplied.

Greyhound

Country / source	Pre	Post	Others
Australia — The Dogs	❌	❌	Direct site returned HTTP 403; no bypass attempted.
Australia — GRV / Watchdog	?	?	Official authenticated/licensed Topaz API is preferred; FastTrack and pedigree/profile data available.
New Zealand — GRNZ	❌	❌	Cloudflare verification blocked access; no challenge bypass attempted.
United Kingdom — Racing Post Greyhounds	?	?	Hash-route SPA/API unresolved; pedigree and exact date of birth available in profiles.
Ireland — Greyhound Racing Ireland	✅	✅	Available: official trials, greyhound search/profiles and videos.
Hungary — Kincsem Park Greyhound	✅	✅	Available: participant/sire statistics, training list/calendar and full pedigree/profile.

Validation evidence

• Dashboard branch gate: 139 tests passed, browser behavior was verified, and the first eight-worker run captured 73 requested-phase meetings in 91 files (116,881,425 bytes). All 91 passed independent SHA-256, byte-length, metadata identity/URI, discipline/source and FormParser-handoff checks.

• Unified code gate: all modules compile and 208 tests pass. Registry counts are T=26, H=9 and G=2, with 37 unique combined source IDs.

• Re-examination live gate: Germany Krefeld pre-race HTML, Argentina La Plata current card, Chile meeting-program JSON/PDF, IndiaRace pre/post HTML, Denmark Odense pre/post JSON, Finland Härmä pre/post JSON, and NDR Eenrum/Emmeloord pre/post HTML all passed source readiness validation. Japan JRA and NAR calendar controls also returned valid live pages.

• Thoroughbred canonical audit: all 21 adapters registered at the 08:44 snapshot had real canonical artifacts, with zero missing/corrupt files, hash/length mismatches, or FormParser handoff gaps. Four later adapters were separately validated; the pushed suite passes 158 tests.

• Harness/Greyhound audit: 86 Harness and 43 Greyhound artifacts passed non-empty, SHA-256, byte-length, metadata and handoff checks. Full-day counts are detailed in HARNESS_GREYHOUND_SOURCE_REPORT.md.

• Strong complete-day evidence includes GB 31 pairs, JRA 36 pairs, NAR 34 pairs, Czechia 28 pairs, Korea 20 cards plus 10 results, Canada Harness 14 cards and 9 results, GRI 29 cards and 9 results, and LeTROT 16 pre-race plus 4 post-race meetings.

• Complete-day storage gate: concurrent unbounded crawls of selected published days/meetings produced 313 immutable artifacts and 313 FormParser handoffs across the re-examined jurisdictions. Recomputed lengths, SHA-256 values, artifact IDs and handoff URIs matched with zero errors. Complete lifecycle evidence includes Germany 9/9, Denmark 9/9, Finland 30/30, JRA 36/36 and NAR 24/24 pre/post pairs, plus whole-meeting evidence for Argentina, India and Netherlands and Chile's complete program/all 20 results.

Starting-point action

Use the updated origin/main as the parent for new work. Keep the historical Harness/Greyhound branch only as development history; the combined registry on main is authoritative.

Architecture and data boundary

FormCrawler discovers acquisition targets, applies the shared policy-controlled fetcher and readiness checks, then writes immutable raw artifacts and FormParser inbox handoffs. Canonical field parsing remains outside this project. Runtime data stays under the configured external t-artifacts, h-artifacts, and g-artifacts trees.

Detailed pages

• thoroughbred-source-research-2026-08: routes, supplementary assets, breeding/profile findings and validation.
• harness-greyhound-source-research-2026-08: Harness/Greyhound routes and evidence.
• form-crawler-programs: modules, launchers and operating commands.
• formcrawler-dashboard: concurrent daily-run controls, live status, storage, validation and deployment guidance.
• scraping-urls-list: active and candidate source URLs.
• country-*: source-specific status and operational notes.