Form Crawler Programs

Last reviewed: 2026-08-08 AEST

FormCrawler has one shared command-line program, reusable acquisition modules, and source-specific country adapters and launchers. Every command acquires and preserves source material; FormParser?not FormCrawler?maps racing values into canonical JSON.

Program entry points

• scripts\formcrawler.py is the repository launcher and adds src to Python's import path.
• python -m formcrawler runs the same CLI when the package is installed or src is on PYTHONPATH.
• The installed formcrawler console command also invokes formcrawler.cli:main.

Available commands

Command	Purpose
capture	Validate, download, and store one explicit source target.
discover	Capture a supported control page and register its discovered targets.
run-queue	Sequentially claim and capture ready persistent targets.
capture-meeting	Start with one Deutscher Galopp race and capture its complete meeting.
capture-discovered	Capture a control page and only the exact targets it discovers, used by Australia bundles and Uruguay horse profiles.
capture-date-range	Perform inclusive, resumable dated discovery and capture with an optional per-date target limit.
queue-status	Display persistent target counts without making a source request.

Common runtime options include --data-root, --state-database, and source selection. Public GET workflows can explicitly use Robert's gateway with --use-racingandsports-proxy or FORMCRAWLER_USE_RACINGANDSORTS_PROXY=1.

Country programs

Country	Program and source ID	Main artifacts	Launcher(s)	Detailed page
Australia	Racing Australia (au_racing_australia)	Weekly meeting-index HTML plus five meeting HTML pages: program, acceptances, gear, scratchings, results	crawl_racing_australia_range.bat; crawl_racing_australia_meeting.bat	country-australia-racing-australia
Germany	Deutscher Galopp (de_deutscher_galopp)	One post-race HTML page per race	crawl_deutscher_galopp_results.bat	country-germany-deutscher-galopp
Chile	Hipódromo Chile (cl_hipodromo_chile)	Meeting/index JSON, per-race result JSON, rolling workout JSON	crawl_hipodromo_chile_results.bat; crawl_hipodromo_chile_workouts.bat	country-chile-hipodromo-chile
Uruguay	Maroñas (uy_maronas)	Range calendar JSON, complete meeting XML, horse search/profile JSON	crawl_maronas_results.bat; crawl_maronas_horse_profile.bat	country-uruguay-maronas
Argentina	Stud Book Argentina (ar_stud_book)	Monthly calendar HTML and complete meeting-results HTML	crawl_stud_book_argentina_results.bat	country-argentina-stud-book

Common operating pattern

1. A country adapter validates the supplied canonical URL and creates a typed target.
2. A discovery/control capture may register normalized child targets.
3. SQLite deduplicates targets and records provenance, attempts, retry timing, leases, and artifacts.
4. The shared policy serializes host requests and normally waits 15-25 seconds between request starts.
5. The source adapter checks identity and publication readiness.
6. The artifact store writes immutable content and metadata.
7. A FormParser inbox manifest points to that immutable artifact.

Stop with Ctrl+C and rerun the same launcher or command to resume. The external FormCrawler\STOP file prevents new requests after a hard stop.

Supporting modules

Acquisition and policy

• formcrawler.engine coordinates a validated target, fetcher, validator, and artifact writer.
• formcrawler.http performs bounded direct HTTP retrieval and the explicit Racing and Sports proxy transport.
• formcrawler.policy supplies same-host serialization, 10-second base delay, 5-15-second jitter, circuit breaking, and shared local rate-limit state.

Discovery and source handling

• formcrawler.sources.base defines the adapter contract and not-ready exception.
• formcrawler.sources.registry resolves stable source IDs.
• formcrawler.html_links decodes HTML and extracts links without mapping racing fields.
• Country modules live under formcrawler.sources.au, .de, .cl, .uy, and .ar.

Queue, workflow, and storage

• formcrawler.workflow coordinates discovery, date ranges, queue capture, retries, and meeting fan-out.
• formcrawler.state.base defines backend-neutral crawl state.
• formcrawler.state.sqlite is the local persistent implementation.
• formcrawler.storage writes immutable artifacts and FormParser handoffs.
• formcrawler.models defines the typed target, fetch, lease, discovery, and artifact contracts.

Data structure and location

Local runtime data defaults to C:\Users\Robert\project-data; Linux containers default to /data. Override either with FORMCRAWLER_DATA_ROOT.

FormCrawler\<discipline>-artifacts\<year>\<race-code>\<source-id>\<event>\
  <readable-artifact-name>--<source-key>--<artifact-prefix>.<ext>
  <readable-artifact-name>--<source-key>--<artifact-prefix>.metadata.json

FormParser\inbox\<discipline>\<year>\<race-code>\<source-id>\<artifact-id>.json

Every metadata/handoff contract identifies the source, ISO country, racing code, discipline, page and artifact types, phase, source URL, retrieval time, content hash, and backend-neutral content/metadata URIs. Source-specific keys are described on each country page.

Proxy transport

Robert's service endpoint is https://sproxy.racingandsports.com/Gateway. FormCrawler sends POST application/json with url, direct=true, and force=true, then unwraps html_data, stores it as UTF-8 with racingandsports_proxy provenance, limits both destination and gateway hosts, and runs source validation. The transport is opt-in and supports public GET targets only; it cannot reproduce a source target's POST body.

Verification

As of 2026-08-08, five country adapters are implemented and live-verified. Racing Australia supports both weekly-indexed date ranges and direct meeting seeds. The full project suite contains 75 passing tests with ResourceWarning promoted to an error.