Form Crawler Programs

<!-- repository-baseline:start -->

Repository baseline

• origin/main is the authoritative unified T/H/G baseline and contains merge commit ab0441e plus the synchronized documentation commit 545e1fe.
• Registry counts are 26 Thoroughbred, nine Harness and two Greyhound adapters: 37 unique source IDs.
• All modules compile and the unified suite passes 208 tests with ResourceWarning promoted to an error.
• New work must branch from updated origin/main; the former Harness/Greyhound branch is development history only.
• See overview for the complete country/source Pre/Post/Others matrix. <!-- repository-baseline:end -->

Last reviewed: 2026-08-10 AEST

FormCrawler has one shared command-line program, reusable acquisition modules, and source-specific country adapters and launchers. Every command acquires and preserves source material; FormParser—not FormCrawler—maps racing values into canonical JSON.

Program entry points

• scripts\formcrawler.py is the repository launcher and adds src to Python's import path.
• python -m formcrawler runs the same CLI when the package is installed or src is on PYTHONPATH.
• The installed formcrawler console command also invokes formcrawler.cli:main.

Available commands

Command	Purpose
capture	Validate, download, and store one explicit source target.
discover	Capture a supported control page and register its discovered targets.
run-queue	Sequentially claim and capture ready persistent targets.
capture-meeting	Start with one Deutscher Galopp race and capture its complete meeting.
capture-discovered	Capture a control page and only the exact targets it discovers, used by Australia bundles and Uruguay horse profiles.
capture-date-range	Perform inclusive, resumable dated discovery and capture with an optional per-date target limit and repeatable discovered page-type filters.
run-day	Plan and supervise concurrent one-day pre/post acquisition with isolated source/phase queues and logs.
dashboard	Serve the local browser run-control and monitoring interface.
queue-status	Display persistent target counts without making a source request.

Common runtime options include --data-root, --state-database, and source selection. Date ranges can repeat --page-type to filter discovered targets before queue insertion. Direct public GET is the default; HTTP 403/429 automatically invokes one Racing and Sports proxy attempt, while --use-racingandsports-proxy or FORMCRAWLER_USE_RACINGANDSORTS_PROXY=1 forces proxy-only retrieval from the first request.

<!-- operations-dashboard:start -->

Concurrent daily operations

Commit 3319e66 on codex/crawl-dashboard adds an independent operations layer without changing FormCrawler's raw-acquisition boundary.

python scripts\formcrawler.py run-day --pre-date YYYY-MM-DD --post-date YYYY-MM-DD --max-workers 8 --data-root "C:\Users\Robert\project-data"
python scripts\formcrawler.py dashboard --host 127.0.0.1 --port 8765 --data-root "C:\Users\Robert\project-data"

run-day creates one subprocess, SQLite queue and retained log per source/phase. It accepts 1-32 workers and schedules distinct sources in the first wave while the shared host limiter continues to serialize same-site requests. The dashboard starts later runs and reports run history, source/discipline filters, requested-day meetings, saved files, queue progress, retries, failures and bounded log tails. Unsupported phase plans are stored as not_available; a valid empty discovery is no_data; productive mixed outcomes aggregate to partial.

See formcrawler-dashboard for the operator guide, source-planning exceptions, runtime layout, validation evidence and deployment boundary. <!-- operations-dashboard:end -->

Country programs

Country	Program and source ID	Main artifacts	Launcher(s)	Detailed page
Australia	Racing Australia (au_racing_australia)	Weekly meeting-index HTML plus either the five lifecycle pages or filtered results-only pages	crawl_racing_australia_range.bat; crawl_racing_australia_results.bat; crawl_racing_australia_meeting.bat; crawl_racing_australia_today.bat	country-australia-racing-australia
Germany	Deutscher Galopp (de_deutscher_galopp)	Calendar plus pre/post race HTML	crawl_deutscher_galopp_prerace.bat; crawl_deutscher_galopp_results.bat	country-germany-deutscher-galopp
Chile	Hipódromo Chile (cl_hipodromo_chile)	Meeting/index JSON, per-race result JSON, rolling workout JSON	crawl_hipodromo_chile_results.bat; crawl_hipodromo_chile_workouts.bat	country-chile-hipodromo-chile
Uruguay	Maroñas (uy_maronas)	Range calendar JSON, complete meeting XML, horse search/profile JSON	crawl_maronas_results.bat; crawl_maronas_horse_profile.bat	country-uruguay-maronas
Argentina	Stud Book Argentina (ar_stud_book)	Current/upcoming card HTML plus calendar and complete meeting results	crawl_stud_book_argentina_prerace.bat; crawl_stud_book_argentina_results.bat	country-argentina-stud-book
India	IndiaRace (in_indiarace)	Current card, results and trackwork meeting HTML	crawl_indiarace_current.bat	country-india-indiarace

<!-- harness-greyhound-modules:start -->

Harness and greyhound source modules

Country	Source ID	Discipline	Main capture forms	Detailed page
Australia	au_harness_racing	Harness	Current fields plus monthly-indexed meeting results HTML	country-australia-harness-racing
New Zealand	nz_hrnz	Harness	Static official fields and results HTML	country-new-zealand-hrnz
Canada	ca_standardbred_canada	Harness	Dated entries/results indexes and meeting .dat HTML	country-canada-standardbred-canada
Germany	de_hvt_harness	Harness	Monthly calendar plus complete starter/result day HTML	country-germany-hvt
France	fr_letrot	Harness	Dated calendar plus lifecycle-aware whole-meeting programme HTML	country-france-letrot-harness; crawl_letrot_harness.bat
Italy	it_snai_ippica_harness	Harness	Signed current-entry and dated-result meeting HTML	country-italy-snai-ippica-harness
Denmark	dk_swedish_horse_racing	Harness	Calendar/raceday/race pre/post JSON	country-denmark-swedish-horse-racing; crawl_danish_harness_range.bat
Finland	fi_heppa_harness	Harness	Range/meeting/start pre/post JSON	country-finland-heppa; crawl_finnish_heppa_range.bat
Netherlands	nl_ndr_harness	Harness	Upcoming and result meeting HTML	country-netherlands-ndr; crawl_ndr_harness.bat
Ireland	ie_gri	Greyhound	Current card list and dated meeting-result HTML	country-ireland-gri
Hungary	hu_kincsem_park_greyhound	Greyhound	Embedded year calendar plus deterministic card/result day HTML	country-hungary-kincsem-greyhound

All 11 Harness/Greyhound modules acquire raw source responses only. Supplementary links and breeding/profile availability are documented on the jurisdiction pages but are outside their crawl scope. <!-- harness-greyhound-modules:end -->

Common operating pattern

1. A country adapter validates the supplied canonical URL and creates a typed target.
2. A discovery/control capture may filter by date and page type, then register normalized child targets.
3. SQLite deduplicates targets and records provenance, attempts, retry timing, leases, and artifacts.
4. The shared policy serializes host requests and normally waits 15-25 seconds between request starts.
5. The source adapter checks identity and publication readiness.
6. The artifact store writes immutable content and metadata.
7. A FormParser inbox manifest points to that immutable artifact.

Stop with Ctrl+C and rerun the same launcher or command to resume. The external FormCrawler\STOP file prevents new requests after a hard stop.

Supporting modules

Acquisition and policy

• formcrawler.engine coordinates a validated target, fetcher, validator, and artifact writer.
• formcrawler.http performs bounded direct HTTP retrieval, anonymous same-host session-cookie reuse in memory, automatic challenge fallback, and explicit proxy-only Racing and Sports transport.
• formcrawler.policy supplies same-host serialization, 10-second base delay, 5-15-second jitter, circuit breaking, and shared local rate-limit state.

Discovery and source handling

• formcrawler.sources.base defines the adapter contract and not-ready exception.
• formcrawler.sources.thoroughbred, .harness, and .greyhound own independent discipline adapter maps.
• formcrawler.sources.registry combines them for stable CLI source-ID resolution and validates discipline/collision integrity.
• formcrawler.html_links decodes HTML and extracts links without mapping racing fields.
• Country modules live under formcrawler.sources.au, .nz, .ca, .de, .cl, .uy, .ar, .it, .ie, .hu, and .fr.

Queue, workflow, and storage

• formcrawler.workflow coordinates discovery, date and page-type filtering, date ranges, queue capture, retries, and meeting fan-out.
• formcrawler.state.base defines backend-neutral crawl state.
• formcrawler.state.sqlite is the local persistent implementation.
• formcrawler.storage writes immutable artifacts and FormParser handoffs.
• formcrawler.models defines the typed target, fetch, lease, discovery, and artifact contracts.
• formcrawler.status standardizes transient component states and permanent key=value summaries for interactive commands and logs.
• formcrawler.operations plans daily source/phase jobs, supervises concurrent subprocesses, and stores durable run/job state.
• formcrawler.dashboard serves the local JSON API and packaged browser interface.

Data structure and location

Local runtime data defaults to C:\Users\Robert\project-data; Linux containers default to /data. Override either with FORMCRAWLER_DATA_ROOT.

FormCrawler\<discipline>-artifacts\<year>\<race-code>\<source-id>\<event>\
  <readable-artifact-name>--<source-key>--<artifact-prefix>.<ext>
  <readable-artifact-name>--<source-key>--<artifact-prefix>.metadata.json

FormParser\inbox\<discipline>\<year>\<race-code>\<source-id>\<artifact-id>.json

Every metadata/handoff contract identifies the source, ISO country, racing code, discipline, page and artifact types, phase, source URL, retrieval time, content hash, and backend-neutral content/metadata URIs. Source-specific keys are described on each country page.

Proxy transport

Robert's service endpoint is https://sproxy.racingandsports.com/Gateway. FormCrawler sends POST application/json with url, direct=true, and force=true, then unwraps html_data, stores it as UTF-8 with racingandsports_proxy provenance, limits both destination and gateway hosts, and runs source validation. It is the automatic fallback for a direct public GET challenged with 403 or 429, and can be forced from the first request. Public GET targets only are eligible; the gateway cannot reproduce a source target's POST body, credentials, cookies, or signed/private access.

Verification

Dashboard branch verification: commit 3319e66 compiles and passes 139 tests. Its first eight-worker run captured 73 requested-phase meetings in 91 files (116,881,425 bytes); all 91 passed independent hash, byte-length, metadata identity/URI and FormParser-handoff checks. See formcrawler-dashboard for job outcomes and known source exceptions.

LeTROT full-day validation captured 4/4 completed meetings and 16/16 incoming meetings with zero retries/failures; all accumulated artifacts and handoffs passed integrity checks. The Harness/Greyhound integration branch passes 131 tests.

As of 2026-08-08, twelve source adapters are implemented and live-verified at representative or full-day scope. Racing Australia supports weekly-indexed lifecycle ranges, results-only ranges, today capture, and direct meeting seeds. Its results-only smoke test queued five result targets and verified the 190,581-byte Coffs Harbour result artifact, SHA-256, and FormParser handoff. The full project suite contains 158 passing tests with ResourceWarning promoted to an error.

Thoroughbred jurisdiction launchers - 2026-08-08

The expanded registry includes nz_loveracing, gb_racing_post, kr_kra, my_selangor_turf_club, qa_qrec, bh_bhra, za_gallop, tr_tjk, jp_umanity, jp_umanity_nar, cz_dostihy_jc, es_jockey_club, ca_equibase, us_equibase, se_swedish_horse_racing, sa_jcsa, hu_kincsem_park, ae_emirates_racing, in_bangalore_races, and in_hydraces. Source-specific .bat launchers provide bounded date-range acquisition. FormCrawler preserves raw responses and parses only what target discovery requires.

<!-- usa-equibase-program:start -->

USA Equibase official acquisition

us_equibase uses the official entries, summary-results, and full-chart indexes. A bounded date-range run can queue complete card HTML, all-races summary HTML, full-meeting and per-race chart PDFs, and published-only GPS HTML. Nested result indexes are filtered to the exact requested dates. Canada remains under ca_equibase with its existing mirror contract.

scripts\crawl_equibase_range.bat us_equibase START_DATE END_DATE DATA_ROOT

The URL and content contracts pass the complete 158-test suite and were validated in the licensed browser against the eight-race Del Mar meeting on 7 August 2026. Unattended direct/proxy transport still receives Imperva HTML, so production crawling requires a licensed non-interactive session/API transport; challenge HTML must never be stored as a card or PDF. <!-- usa-equibase-program:end -->