FormCrawler

Last reviewed: 2026-08-08 AEST

Purpose

FormCrawler is the policy-controlled acquisition component of the racing-data ingestion workflow. It discovers in-scope public source URLs, retrieves raw pages or first-party documents, and preserves immutable artifacts plus metadata and FormParser inbox handoffs. It parses only what target discovery requires; canonical racing-data parsing belongs to FormParser.

Status key

• ✅: crawler code exists for this lifecycle. A tick does not hide a documented transport or archive limitation.
• ❌: the source was determined unavailable, unsuitable, wrong-discipline, or outside the permitted acquisition contract.
• ?: incomplete, partially validated, unresolved, or not yet implemented.
• Captured in Others means FormCrawler acquired that asset. Available means it was found and linked for later work but is not currently in the default crawler scope.

Git and validation baseline

Scope	Location	State at reconciliation	Verification
Unified main	origin/main contains merge ab0441e, combining Thoroughbred 4d68b42 and Harness/Greyhound 490b0c9	25 Thoroughbred, six Harness, and two Greyhound adapters in independent discipline registries	200 tests passed; 33 unique source IDs
Historical Harness/Greyhound branch	490b0c9	Eight H/G adapters, including LeTROT and separate T/H/G registries	131 tests passed before integration; full-day artifact audits passed
Runtime	local machine	No FormCrawler crawler process was running at reconciliation	Canonical artifact and handoff audits are recorded below

main is now the single T/H/G starting point. New work should branch from the updated origin/main, not the historical discipline branch.

Thoroughbred

Country / source	Pre	Post	Others
Australia — Racing Australia	✅	✅	Captured: acceptances, gear changes, scratchings. National source covers ACT, NSW, NT, QLD, SA, TAS, VIC and WA.
New Zealand — LoveRacing	✅	✅	Combined meeting-lifecycle page; profiles and breeding links available.
United Kingdom — Racing Post	✅	✅	Profiles, breeding and form links available.
Korea — KRA	✅	✅	Horse and pedigree detail available.
Malaysia — Selangor Turf Club	✅	✅	Complete 12-race meeting card/results validated.
Qatar — QREC	✅	✅	Meeting/race documents and horse detail available.
Bahrain — BHRA	✅	✅	Meeting/race documents and horse detail available.
South Africa — Gallop	✅	✅	Meeting/day lifecycle captured; some additional targets unpublished.
South Africa — Gold Circle	?	❌	Available: pre-race PDF/calendar documents; no dependable machine post-race archive.
Turkey — TJK	✅	✅	Complete meeting lifecycle validated; profiles/breeding available.
Japan — Umanity JRA	✅	✅	Full-day 36 pre/post pairs validated.
Japan — Umanity NAR	✅	✅	Full-day 34 pre/post pairs validated.
Japan — JBIS	❌	?	Available: results and rich breeding/horse profiles; no suitable pre-race route established.
Japan — NAR Keiba	❌	❌	Source outage prevented a safe crawler; Umanity NAR is the working source.
Czechia — Dostihy JC	✅	✅	Full-day 28 pairs validated; horse/pedigree detail available.
Spain — Jockey Club Español	✅	✅	Captured: stewards content within combined results pages.
Canada — Equibase mirror workflow	✅	✅	One complete meeting pair validated; horse/profile links available.
Sweden — Swedish Horse Racing	✅	✅	Three pairs captured; three additional targets were not yet published.
Hungary — Kincsem Park gallop	✅	✅	Horse/profile and breeding fields available.
United Arab Emirates — Emirates Racing	✅	✅	Horse/profile and breeding fields available.
India — Bangalore Races	✅	✅	Available: fixtures, videos, ratings, archives and profiles.
India — Hyderabad Races	✅	✅	Available: trackwork, entries, handicaps, acceptances, declarations, ratings, swimming, veterinary remarks and incidents.
Argentina — Stud Book	?	✅	Current-month calendar discovery is guarded; direct meeting capture works. Horse/profile links available.
Chile — Hipódromo Chile	?	✅	Captured: rolling workouts JSON; horse/profile detail available.
Germany — Deutscher Galopp	?	✅	Complete post-race meetings validated; no dependable historical pre-race contract.
Uruguay — Maroñas	?	✅	Captured: horse search/profile JSON with sire, dam and damsire.
Saudi Arabia — JCSA	?	✅	Post-race results validated; pre-race validator remained incomplete after three attempts.
USA — official Equibase	✅	✅	Crawler downloads card/summary HTML, full-meeting and race PDFs, and published GPS HTML. Licensed browser validation passed; unattended licensed transport remains pending.
Germany — German Racing	❌	❌	Transport failed; Deutscher Galopp is the working German source.
Italy — SNAI Thoroughbred	?	?	Signed/unstable routes remain unresolved.
Belgium — Belgian Galopp	?	?	Available: meeting/program documents; no adapter.
Chile / Uruguay / Brazil — AMR Channel	❌	❌	Authentication boundary; not bypassed.
Norway — RaceBets	❌	❌	Licensed commercial database; unsuitable crawler source.
Mauritius — MTC Jockey Club	❌	❌	Transport blocked. Available: workouts, trials, stewards, replays and search links.
Morocco — SOREC Galop	?	?	Stateful JSF unresolved. Available: programmes, VOD, bulletins and technical documents.
Netherlands — NDR Thoroughbred listing	❌	❌	Supplied route is Harness, not Thoroughbred.
Hong Kong — HKJC	?	?	Available: trackwork, vet/form/incidents, replays, sectionals and profiles; no completed adapter.
France — France Galop	?	?	Available: programme, agenda, ratings, breeders, sires and bulletins; result route unresolved.
India — IndiaRace	?	?	Interactive archive unresolved.
India — Racing Pulse	?	?	Stateful postback archive unresolved.
India — Delhi / DRC	?	?	Document-oriented source only in this pass.
India — RCTC	?	?	Current/document routes found; no completed adapter.
Austria	?	?	No source URL supplied.
Ireland	?	?	No Thoroughbred source URL supplied.
Peru	?	?	No source URL supplied.
Panama	?	?	No source URL supplied.
Mexico	?	?	No source URL supplied.

Harness

Country / source	Pre	Post	Others
Australia — Harness Racing Australia	✅	?	Available: trials, stewards reports, replays and horse profiles. Current cards passed 25/26; complete national result-day coverage remained partial.
New Zealand — HRNZ	✅	✅	Available: trial fields/results, workout fields/results, programmes and horse detail.
France — LeTROT	✅	✅	Available: qualifications, replays, publications and horse profiles. Code is committed and pushed on the Harness/Greyhound branch.
Denmark — Paris-Turf	❌	❌	Catalogue mismatch: French publication did not provide a reliable Denmark source.
Denmark — Swedish Horse Racing	?	?	Historical calendar works; Angular/XHR API unresolved.
Belgium — Belgium Horse Racing	?	?	Strong next candidate. Available: race newspaper, rankings and qualification/test races.
Finland — Heppa	?	?	Client/API index unresolved; horse statistics available.
Netherlands — NDR	?	?	Programmes, calendars, downloads and bulletins available; dynamic result selector unresolved.
Norway — RaceBets	❌	❌	Licensed commercial database; unsuitable crawler source.
Norway — Swedish Horse Racing	?	?	Historical official PDFs and pedigree details available; API unresolved.
Spain — Federació Balear de Trot	?	?	Strong next candidate. Available: history, bulletins, videos, programme PDFs and profiles.
Hungary — Kincsem Park trotting	?	?	Strong next candidate using the same platform pattern as Kincsem Greyhound.
Sweden — Travsport	?	?	Bot verification not bypassed; approved proxy remains a bounded future check.
Sweden — Swedish Horse Racing	?	?	Shared Angular/XHR source remains unresolved.
USA — USTA	?	?	Stateful ASP.NET session/postback workflow unresolved; Pathway profiles available separately.
Canada — Standardbred Canada	✅	✅	Available: judges reports, race dates, feature races and TrackIT profiles.
Germany — HVT	✅	✅	Approved proxy used. Available: monthly reports, videos, finish photos, official PDFs and pedigree.
Italy — SNAI Harness	✅	✅	Available: meeting PDFs, videos, qualification trials and profiles; historical pre cards are unavailable.
Austria	?	?	No Harness source URL supplied.
Malta	?	?	No Harness source URL supplied.

Greyhound

Country / source	Pre	Post	Others
Australia — The Dogs	❌	❌	Direct site returned HTTP 403; no bypass attempted.
Australia — GRV / Watchdog	?	?	Official authenticated/licensed Topaz API is preferred; FastTrack and pedigree/profile data available.
New Zealand — GRNZ	❌	❌	Cloudflare verification blocked access; no challenge bypass attempted.
United Kingdom — Racing Post Greyhounds	?	?	Hash-route SPA/API unresolved; pedigree and exact date of birth available in profiles.
Ireland — Greyhound Racing Ireland	✅	✅	Available: official trials, greyhound search/profiles and videos.
Hungary — Kincsem Park Greyhound	✅	✅	Available: participant/sire statistics, training list/calendar and full pedigree/profile.

Validation evidence

• Unified code gate: all modules compile and 200 tests pass with ResourceWarning promoted to an error. Registry counts are T=25, H=6 and G=2, with 33 unique combined source IDs.
• Thoroughbred canonical audit: all 21 adapters registered at the 08:44 snapshot had real canonical artifacts, with zero missing/corrupt files, hash/length mismatches, or FormParser handoff gaps. Four later adapters were separately validated; the pushed suite passes 158 tests.
• Harness/Greyhound audit: 86 Harness and 43 Greyhound artifacts passed non-empty, SHA-256, byte-length, metadata and handoff checks. Full-day counts are detailed in HARNESS_GREYHOUND_SOURCE_REPORT.md.
• Strong complete-day evidence includes GB 31 pairs, JRA 36 pairs, NAR 34 pairs, Czechia 28 pairs, Korea 20 cards plus 10 results, Canada Harness 14 cards and 9 results, GRI 29 cards and 9 results, and LeTROT 16 pre-race plus 4 post-race meetings.

Starting-point action

Use the updated origin/main as the parent for new work. Keep the historical Harness/Greyhound branch only as development history; the combined registry on main is authoritative.

Architecture and data boundary

calendar/index/meeting source
            |
            v
      source adapter
            |
            v
 persistent queue -> policy-controlled fetch -> readiness validation
                                                |
                                                v
                                      immutable raw artifact
                                                |
                                                v
                                       FormParser handoff

Runtime artifacts remain outside the repository under the configured FORMCRAWLER_DATA_ROOT, using sibling t-artifacts, h-artifacts, and g-artifacts trees. The repository contains source adapters, launchers, tests, and documentation only.

All configured requests use the common per-host limiter, bounded retries/backoff, response-size limit, circuit breaker, persistent queue, and external STOP kill switch. Public GET requests challenged by 403 or 429 may use the approved Racing and Sports proxy; signed, credentialed, private, POST-dependent, CAPTCHA, and licensing boundaries are not bypassed.

Detailed pages

• thoroughbred-source-research-2026-08: complete Thoroughbred route, method, supplementary, breeding/profile, and validation report.
• harness-greyhound-source-research-2026-08: complete Harness/Greyhound route, method, supplementary, breeding/profile, and validation report.
• form-crawler-programs: executable module and launcher index.
• scraping-urls-list: active source and candidate URL inventory.
• country-*: source-specific implementation, findings, limitations, examples, and changelog pages.