Form Crawler Programs

Last reviewed: 2026-08-08 AEST

FormCrawler has one shared command-line program, reusable acquisition modules, and source-specific country adapters and launchers. Every command acquires and preserves source material; FormParser?not FormCrawler?maps racing values into canonical JSON.

Program entry points

• scripts\formcrawler.py is the repository launcher and adds src to Python's import path.
• python -m formcrawler runs the same CLI when the package is installed or src is on PYTHONPATH.
• The installed formcrawler console command also invokes formcrawler.cli:main.

Available commands

Command	Purpose
capture	Validate, download, and store one explicit source target.
discover	Capture a supported control page and register its discovered targets.
run-queue	Sequentially claim and capture ready persistent targets.
capture-meeting	Start with one Deutscher Galopp race and capture its complete meeting.
capture-discovered	Capture a control page and only the exact targets it discovers, used by Australia bundles and Uruguay horse profiles.
capture-date-range	Perform inclusive, resumable dated discovery and capture with an optional per-date target limit and repeatable discovered page-type filters.
queue-status	Display persistent target counts without making a source request.

Common runtime options include --data-root, --state-database, and source selection. Date ranges can repeat --page-type to filter discovered targets before queue insertion. Direct public GET is the default; HTTP 403/429 automatically invokes one Racing and Sports proxy attempt, while --use-racingandsports-proxy or FORMCRAWLER_USE_RACINGANDSORTS_PROXY=1 forces proxy-only retrieval from the first request.

Country programs

Country	Program and source ID	Main artifacts	Launcher(s)	Detailed page
Australia	Racing Australia (au_racing_australia)	Weekly meeting-index HTML plus either the five lifecycle pages or filtered results-only pages	crawl_racing_australia_range.bat; crawl_racing_australia_results.bat; crawl_racing_australia_meeting.bat; crawl_racing_australia_today.bat	country-australia-racing-australia
Germany	Deutscher Galopp (de_deutscher_galopp)	One post-race HTML page per race	crawl_deutscher_galopp_results.bat	country-germany-deutscher-galopp
Chile	Hipódromo Chile (cl_hipodromo_chile)	Meeting/index JSON, per-race result JSON, rolling workout JSON	crawl_hipodromo_chile_results.bat; crawl_hipodromo_chile_workouts.bat	country-chile-hipodromo-chile
Uruguay	Maroñas (uy_maronas)	Range calendar JSON, complete meeting XML, horse search/profile JSON	crawl_maronas_results.bat; crawl_maronas_horse_profile.bat	country-uruguay-maronas
Argentina	Stud Book Argentina (ar_stud_book)	Monthly calendar HTML and complete meeting-results HTML	crawl_stud_book_argentina_results.bat	country-argentina-stud-book

Common operating pattern

1. A country adapter validates the supplied canonical URL and creates a typed target.
2. A discovery/control capture may filter by date and page type, then register normalized child targets.
3. SQLite deduplicates targets and records provenance, attempts, retry timing, leases, and artifacts.
4. The shared policy serializes host requests and normally waits 15-25 seconds between request starts.
5. The source adapter checks identity and publication readiness.
6. The artifact store writes immutable content and metadata.
7. A FormParser inbox manifest points to that immutable artifact.

Stop with Ctrl+C and rerun the same launcher or command to resume. The external FormCrawler\STOP file prevents new requests after a hard stop.

Supporting modules

Acquisition and policy

• formcrawler.engine coordinates a validated target, fetcher, validator, and artifact writer.
• formcrawler.http performs bounded direct HTTP retrieval, automatic challenge fallback, and explicit proxy-only Racing and Sports transport.
• formcrawler.policy supplies same-host serialization, 10-second base delay, 5-15-second jitter, circuit breaking, and shared local rate-limit state.

Discovery and source handling

• formcrawler.sources.base defines the adapter contract and not-ready exception.
• formcrawler.sources.registry resolves stable source IDs.
• formcrawler.html_links decodes HTML and extracts links without mapping racing fields.
• Country modules live under formcrawler.sources.au, .de, .cl, .uy, and .ar.

Queue, workflow, and storage

• formcrawler.workflow coordinates discovery, date and page-type filtering, date ranges, queue capture, retries, and meeting fan-out.
• formcrawler.state.base defines backend-neutral crawl state.
• formcrawler.state.sqlite is the local persistent implementation.
• formcrawler.storage writes immutable artifacts and FormParser handoffs.
• formcrawler.models defines the typed target, fetch, lease, discovery, and artifact contracts.
• formcrawler.status standardizes transient component states and permanent key=value summaries for interactive commands and logs.

Data structure and location

Local runtime data defaults to C:\Users\Robert\project-data; Linux containers default to /data. Override either with FORMCRAWLER_DATA_ROOT.

FormCrawler\<discipline>-artifacts\<year>\<race-code>\<source-id>\<event>\
  <readable-artifact-name>--<source-key>--<artifact-prefix>.<ext>
  <readable-artifact-name>--<source-key>--<artifact-prefix>.metadata.json

FormParser\inbox\<discipline>\<year>\<race-code>\<source-id>\<artifact-id>.json

Every metadata/handoff contract identifies the source, ISO country, racing code, discipline, page and artifact types, phase, source URL, retrieval time, content hash, and backend-neutral content/metadata URIs. Source-specific keys are described on each country page.

Proxy transport

Robert's service endpoint is https://sproxy.racingandsports.com/Gateway. FormCrawler sends POST application/json with url, direct=true, and force=true, then unwraps html_data, stores it as UTF-8 with racingandsports_proxy provenance, limits both destination and gateway hosts, and runs source validation. It is the automatic fallback for a direct public GET challenged with 403 or 429, and can be forced from the first request. Public GET targets only are eligible; the gateway cannot reproduce a source target's POST body, credentials, cookies, or signed/private access.

Verification

As of 2026-08-08, five country adapters are implemented and live-verified. Racing Australia supports weekly-indexed lifecycle ranges, results-only ranges, today capture, and direct meeting seeds. Its results-only smoke test queued five result targets and verified the 190,581-byte Coffs Harbour result artifact, SHA-256, and FormParser handoff. The full project suite contains 89 passing tests with ResourceWarning promoted to an error.