Form Crawler Programs

Last reviewed: 2026-08-08 AEST

FormCrawler has one shared command-line program, reusable acquisition modules, and source-specific country adapters and launchers. Every command acquires and preserves source material; FormParser—not FormCrawler—maps racing values into canonical JSON.

Program entry points

• scripts\formcrawler.py is the repository launcher and adds src to Python's import path.
• python -m formcrawler runs the same CLI when the package is installed or src is on PYTHONPATH.
• The installed formcrawler console command also invokes formcrawler.cli:main.

Available commands

Command	Purpose
capture	Validate, download, and store one explicit source target.
discover	Capture a supported control page and register its discovered targets.
run-queue	Sequentially claim and capture ready persistent targets.
capture-meeting	Start with one Deutscher Galopp race and capture its complete meeting.
capture-discovered	Capture a control page and only the exact targets it discovers, used by Australia bundles and Uruguay horse profiles.
capture-date-range	Perform inclusive, resumable dated discovery and capture with an optional per-date target limit and repeatable discovered page-type filters.
queue-status	Display persistent target counts without making a source request.

Common runtime options include --data-root, --state-database, and source selection. Date ranges can repeat --page-type to filter discovered targets before queue insertion. Direct public GET is the default; HTTP 403/429 automatically invokes one Racing and Sports proxy attempt, while --use-racingandsports-proxy or FORMCRAWLER_USE_RACINGANDSORTS_PROXY=1 forces proxy-only retrieval from the first request.

Country programs

Country	Program and source ID	Main artifacts	Launcher(s)	Detailed page
Australia	Racing Australia (au_racing_australia)	Weekly meeting-index HTML plus either the five lifecycle pages or filtered results-only pages	crawl_racing_australia_range.bat; crawl_racing_australia_results.bat; crawl_racing_australia_meeting.bat; crawl_racing_australia_today.bat	country-australia-racing-australia
Germany	Deutscher Galopp (de_deutscher_galopp)	One post-race HTML page per race	crawl_deutscher_galopp_results.bat	country-germany-deutscher-galopp
Chile	Hipódromo Chile (cl_hipodromo_chile)	Meeting/index JSON, per-race result JSON, rolling workout JSON	crawl_hipodromo_chile_results.bat; crawl_hipodromo_chile_workouts.bat	country-chile-hipodromo-chile
Uruguay	Maroñas (uy_maronas)	Range calendar JSON, complete meeting XML, horse search/profile JSON	crawl_maronas_results.bat; crawl_maronas_horse_profile.bat	country-uruguay-maronas
Argentina	Stud Book Argentina (ar_stud_book)	Monthly calendar HTML and complete meeting-results HTML	crawl_stud_book_argentina_results.bat	country-argentina-stud-book

<!-- harness-greyhound-modules:start -->

Harness and greyhound source modules

Country	Source ID	Discipline	Main capture forms	Detailed page
Australia	au_harness_racing	Harness	Current fields plus monthly-indexed meeting results HTML	country-australia-harness-racing
New Zealand	nz_hrnz	Harness	Static official fields and results HTML	country-new-zealand-hrnz
Canada	ca_standardbred_canada	Harness	Dated entries/results indexes and meeting .dat HTML	country-canada-standardbred-canada
Germany	de_hvt_harness	Harness	Monthly calendar plus complete starter/result day HTML	country-germany-hvt
France	fr_letrot	Harness	Dated calendar plus lifecycle-aware whole-meeting programme HTML	country-france-letrot-harness; crawl_letrot_harness.bat
Italy	it_snai_ippica_harness	Harness	Signed current-entry and dated-result meeting HTML	country-italy-snai-ippica-harness
Ireland	ie_gri	Greyhound	Current card list and dated meeting-result HTML	country-ireland-gri
Hungary	hu_kincsem_park_greyhound	Greyhound	Embedded year calendar plus deterministic card/result day HTML	country-hungary-kincsem-greyhound

All eight modules acquire raw source responses only. Supplementary links and breeding/profile availability are documented on the jurisdiction pages but are outside their crawl scope. <!-- harness-greyhound-modules:end -->

Common operating pattern

1. A country adapter validates the supplied canonical URL and creates a typed target.
2. A discovery/control capture may filter by date and page type, then register normalized child targets.
3. SQLite deduplicates targets and records provenance, attempts, retry timing, leases, and artifacts.
4. The shared policy serializes host requests and normally waits 15-25 seconds between request starts.
5. The source adapter checks identity and publication readiness.
6. The artifact store writes immutable content and metadata.
7. A FormParser inbox manifest points to that immutable artifact.

Stop with Ctrl+C and rerun the same launcher or command to resume. The external FormCrawler\STOP file prevents new requests after a hard stop.

Supporting modules

Acquisition and policy

• formcrawler.engine coordinates a validated target, fetcher, validator, and artifact writer.
• formcrawler.http performs bounded direct HTTP retrieval, automatic challenge fallback, and explicit proxy-only Racing and Sports transport.
• formcrawler.policy supplies same-host serialization, 10-second base delay, 5-15-second jitter, circuit breaking, and shared local rate-limit state.

Discovery and source handling

• formcrawler.sources.base defines the adapter contract and not-ready exception.
• formcrawler.sources.thoroughbred, .harness, and .greyhound own independent discipline adapter maps.
• formcrawler.sources.registry combines them for stable CLI source-ID resolution and validates discipline/collision integrity.
• formcrawler.html_links decodes HTML and extracts links without mapping racing fields.
• Country modules live under formcrawler.sources.au, .nz, .ca, .de, .cl, .uy, .ar, .it, .ie, .hu, and .fr.

Queue, workflow, and storage

• formcrawler.workflow coordinates discovery, date and page-type filtering, date ranges, queue capture, retries, and meeting fan-out.
• formcrawler.state.base defines backend-neutral crawl state.
• formcrawler.state.sqlite is the local persistent implementation.
• formcrawler.storage writes immutable artifacts and FormParser handoffs.
• formcrawler.models defines the typed target, fetch, lease, discovery, and artifact contracts.
• formcrawler.status standardizes transient component states and permanent key=value summaries for interactive commands and logs.

Data structure and location

Local runtime data defaults to C:\Users\Robert\project-data; Linux containers default to /data. Override either with FORMCRAWLER_DATA_ROOT.

FormCrawler\<discipline>-artifacts\<year>\<race-code>\<source-id>\<event>\
  <readable-artifact-name>--<source-key>--<artifact-prefix>.<ext>
  <readable-artifact-name>--<source-key>--<artifact-prefix>.metadata.json

FormParser\inbox\<discipline>\<year>\<race-code>\<source-id>\<artifact-id>.json

Every metadata/handoff contract identifies the source, ISO country, racing code, discipline, page and artifact types, phase, source URL, retrieval time, content hash, and backend-neutral content/metadata URIs. Source-specific keys are described on each country page.

Proxy transport

Robert's service endpoint is https://sproxy.racingandsports.com/Gateway. FormCrawler sends POST application/json with url, direct=true, and force=true, then unwraps html_data, stores it as UTF-8 with racingandsports_proxy provenance, limits both destination and gateway hosts, and runs source validation. It is the automatic fallback for a direct public GET challenged with 403 or 429, and can be forced from the first request. Public GET targets only are eligible; the gateway cannot reproduce a source target's POST body, credentials, cookies, or signed/private access.

Verification

LeTROT full-day validation captured 4/4 completed meetings and 16/16 incoming meetings with zero retries/failures; all accumulated artifacts and handoffs passed integrity checks. The Harness/Greyhound integration branch passes 131 tests.

As of 2026-08-08, twelve source adapters are implemented and live-verified at representative or full-day scope. Racing Australia supports weekly-indexed lifecycle ranges, results-only ranges, today capture, and direct meeting seeds. Its results-only smoke test queued five result targets and verified the 190,581-byte Coffs Harbour result artifact, SHA-256, and FormParser handoff. The full project suite contains 158 passing tests with ResourceWarning promoted to an error.

Thoroughbred jurisdiction launchers - 2026-08-08

The expanded registry includes nz_loveracing, gb_racing_post, kr_kra, my_selangor_turf_club, qa_qrec, bh_bhra, za_gallop, tr_tjk, jp_umanity, jp_umanity_nar, cz_dostihy_jc, es_jockey_club, ca_equibase, us_equibase, se_swedish_horse_racing, sa_jcsa, hu_kincsem_park, ae_emirates_racing, in_bangalore_races, and in_hydraces. Source-specific .bat launchers provide bounded date-range acquisition. FormCrawler preserves raw responses and parses only what target discovery requires.

<!-- usa-equibase-program:start -->

USA Equibase official acquisition

us_equibase uses the official entries, summary-results, and full-chart indexes. A bounded date-range run can queue complete card HTML, all-races summary HTML, full-meeting and per-race chart PDFs, and published-only GPS HTML. Nested result indexes are filtered to the exact requested dates. Canada remains under ca_equibase with its existing mirror contract.

scripts\crawl_equibase_range.bat us_equibase START_DATE END_DATE DATA_ROOT

The URL and content contracts pass the complete 158-test suite and were validated in the licensed browser against the eight-race Del Mar meeting on 7 August 2026. Unattended direct/proxy transport still receives Imperva HTML, so production crawling requires a licensed non-interactive session/API transport; challenge HTML must never be stored as a card or PDF. <!-- usa-equibase-program:end -->