FormCrawler

Last reviewed: 2026-08-08 AEST

Purpose

FormCrawler is the policy-controlled acquisition component of the racing-data ingestion workflow. It discovers in-scope source URLs, retrieves public source pages or first-party documents, and preserves immutable artifacts with metadata for FormParser.

FormCrawler owns acquisition, request scheduling, retry state, rate limiting, diagnostics, immutable storage, and FormParser inbox handoffs. It does not map source fields into canonical racing JSON or update the racing database; those responsibilities belong to FormParser and downstream systems.

Implemented country modules

Country	Source ID	Main capture forms	Country page
Australia	au_racing_australia	Weekly meeting-index HTML plus five lifecycle pages or filtered results-only HTML	country-australia-racing-australia
Germany	de_deutscher_galopp	Post-race race HTML	country-germany-deutscher-galopp
Chile	cl_hipodromo_chile	Meeting/race/workout JSON	country-chile-hipodromo-chile
Uruguay	uy_maronas	Calendar JSON, meeting XML, horse JSON	country-uruguay-maronas
Argentina	ar_stud_book	Monthly calendar and complete meeting HTML	country-argentina-stud-book

The form-crawler-programs page is the command and module index. Each country page documents its source routes, source data structure, launcher and generic commands, readiness rules, verification, limitations, and module-specific changelog.

Architecture

Calendar, meeting, race, horse, or supplementary source
                         |
                         v
                  Country adapter
                         |
                         v
 Persistent target queue -> policy-controlled fetch -> source validation
                                                        |
                                                        v
                                               immutable artifact
                                                        |
                                                        v
                                                FormParser handoff

Reusable layers remain independent of country rules:

• policy.py and http.py enforce request safeguards, direct retrieval, automatic challenge proxy fallback, and explicit proxy-only transport.
• storage.py writes immutable artifacts and URI-based handoffs.
• state/base.py defines backend-neutral crawl state.
• state/sqlite.py supplies the local development queue and capture ledger.
• workflow.py coordinates discovery, date and page-type filtering, date ranges, worker leases, retries, and fan-out.
• sources/<country>/... contains strict source URL, discovery, and readiness rules.
• cli.py exposes direct capture, discovery, queue, meeting, exact-discovery, filtered date-range, and status commands.
• status.py standardizes transient component states and permanent command summaries.

Data contract and storage

Program source remains in C:\Users\Robert\Projects\FormCrawler. Runtime data remains outside the repository under C:\Users\Robert\project-data locally or the configured FORMCRAWLER_DATA_ROOT; Linux containers default to /data.

FormCrawler\t-artifacts\<year>\<race-code>\<source-id>\<event>\
FormParser\inbox\T\<year>\<race-code>\<source-id>\<artifact-id>.json

Implemented racing-code/ISO pairs are AUS/AU, GER/DE, CHI/CL, URU/UY, and ARG/AR. Metadata and handoffs identify source URL, retrieval time, content type/charset, phase, page and artifact type, source keys, content hash, and backend-neutral content/metadata URIs.

Direct responses remain byte-for-byte immutable. Proxied html_data is stored as UTF-8 with racingandsports_proxy provenance whether proxy use is automatic after a challenge or explicitly selected.

Friendly scraping policy

All source requests share central safeguards:

• one concurrent request per host;
• 10-second base delay plus 5-15 seconds of jitter, giving a normal 15-25-second interval;
• bounded retries and backoff;
• 25 MB response limit;
• circuit breaking;
• persistent shared host timing;
• one approved proxy attempt after a direct public GET receives 403/429, followed by the first-block pause and repeated-block hard stop if proxy retrieval fails;
• external FormCrawler\STOP kill switch.

Robert's Racing and Sports gateway is the approved automatic fallback for a direct public GET challenged with 403 or 429; it can also be selected from the first request. Both gateway and destination hosts are rate-limited, proxy provenance is retained, and the selected country adapter still validates returned content. POST, credentialed, signed, and private targets remain ineligible.

Operation

Country launchers:

.\scripts\crawl_racing_australia_range.bat <start> <end> [limit]
.\scripts\crawl_racing_australia_results.bat <start> <end> [limit]
.\scripts\crawl_racing_australia_today.bat
.\scripts\crawl_racing_australia_meeting.bat "<meeting-url>"
.\scripts\crawl_deutscher_galopp_results.bat
.\scripts\crawl_hipodromo_chile_results.bat <start> <end> [limit]
.\scripts\crawl_hipodromo_chile_workouts.bat
.\scripts\crawl_maronas_results.bat <start> <end> [limit]
.\scripts\crawl_maronas_horse_profile.bat "<horse-name>"
.\scripts\crawl_stud_book_argentina_results.bat <start> <end> [limit]

Generic queue check:

python scripts\formcrawler.py queue-status

Run tests:

$env:PYTHONPATH="C:\Users\Robert\Projects\FormCrawler\src"
python -W error::ResourceWarning -m unittest discover -s tests -v

Deployment

FormCrawler is required to run in Docker on Amazon EKS. Core workflows and source adapters depend on interfaces rather than Windows paths or SQLite behavior. The filesystem store, SQLite state, and local rate-limit coordinator are development adapters. Production still requires shared object storage, distributed queue/state, and distributed rate-limit implementations.

Verified status

As of 2026-08-08:

• all five country adapters have passed live acquisition checks;
• source-specific artifact hashes and FormParser handoffs have been verified for representative captures;
• Racing Australia results-only range capture queued one result per meeting and verified the Coffs Harbour result artifact, content hash, and FormParser handoff;
• the current project suite passes 89 tests with ResourceWarning treated as an error;
• the initial local commit remains 816f590; later working-tree changes have not yet been committed or pushed.

Current limitations

• FormParser does not yet formally validate and consume the artifact/handoff contract.
• Retention and repeated-snapshot content-deduplication policy remains undefined.
• Conditional HTTP requests using stored ETag and Last-Modified values are not implemented.
• Scheduled discovery has not been assigned an operating timetable.
• EKS production adapters remain to be implemented.
• Stud Book Argentina historical month selection requires interactive verification; FormCrawler rejects the site's silent current-month fallback.

Thoroughbred source expansion - 2026-08-08

The bounded jurisdiction pass classified every supplied Thoroughbred source and every no-URL jurisdiction. Twenty source adapters were added or completed across the wider run. The final suite passes 152 tests. Saudi JCSA is partial after the requested three-attempt ceiling; inaccessible, authentication-only, document-only, research-only, and non-Thoroughbred sources are explicitly recorded. See thoroughbred-source-research-2026-08 and the source-specific country pages.