changelog

FormCrawler Changelog

Version 15 · Added French LeTROT Harness implementation, full-day evidence, corrected acquisition method, and independent discipline-module architecture.

Historical version

FormCrawler Changelog

2026-08-08 - French LeTROT Harness and discipline modules

Added and changed

Implemented French Harness LeTROT as fr_letrot using public server-rendered HTML, one dated calendar per day, and one whole-meeting programme per discovered meeting.
Added explicit individual race-detail capture for optional richer tracking, sectional, steward, replay, origin and qualification retention.
Separated source registration into independent Thoroughbred (T), Harness (H), and Greyhound (G) modules. The combined registry remains compatible and enforces declared discipline plus cross-module source-ID uniqueness.
Added scripts\crawl_letrot_harness.bat and the dedicated country-france-letrot-harness page.

Verification

Direct public GET worked; no proxy or access bypass was required.
The complete 7-August result day captured 4/4 meetings and the complete 9-August pre-race day captured 16/16 meetings, both with zero retries/failures.
All 26 accumulated LeTROT artifacts passed byte-length, SHA-256, metadata identity/URI and FormParser handoff checks.
The Harness/Greyhound integration branch passes 131 tests.

Corrected finding

The earlier API-only/Vue-shell conclusion was incorrect. LeTROT embeds complete lifecycle data in server-rendered HTML, including deterministic meeting/race routes and first-party INCOMING/DEFINITIVE status.

2026-08-08 - USA Equibase official lifecycle

Implemented

Replaced the USA TVG-mirror assumption with official www.equibase.com entries, summary-results, and full-chart indexes.
us_equibase now downloads and stores complete card HTML, complete all-races summary HTML, the full-meeting PDF, every published per-race chart PDF, and every published GPS HTML page. PDFs are immutable binary post-race artifacts with FormParser handoffs, not link-only records. Canada remains on its separate existing mirror workflow.
Added two-level nested discovery with exact requested-date filtering, removing the broad-index overqueue defect while allowing date-less result control indexes to be traversed.
Added strict binary PDF checks and official-origin, lifecycle, published-only GPS, and nested-date regression tests. The full suite passes 158 tests.
Documented the observed acquisition payload: card conditions/runners/participants/odds and pedigrees; summary placings, payouts and winning connections; detailed chart timing/running-line/claims/scratches/wagering information; and publication-dependent GPS HTML.

Licensed validation and remaining limitation

The licensed browser session validated Del Mar on 7 August 2026: 8-race complete card, 8-race summary, 8 race PDFs, 7 published GPS pages, and a 10-page full-meeting PDF.
Unattended direct and approved-proxy requests returned Imperva HTML, including for PDF URLs. No replacement canonical artifact or successful unattended full-day crawl is claimed; production execution needs a licensed non-interactive session/API transport.

2026-08-08 - Harness and greyhound jurisdictions and full-day validation

Added

Implemented seven raw-acquisition adapters: Harness Racing Australia, HRNZ, Standardbred Canada, HVT harness, SNAI harness, Greyhound Racing Ireland, and Kincsem Park greyhound.
Added one source-specific jurisdiction page for each adapter, including routes, commands, raw artifact and handoff structure, supplementary data, breeding/profile availability, limitations, and module history.
Corrected the Kincsem catalogue entry from trotting to the official greyhound calendar, card, and result routes.
Renamed the Greyhound source to hu_kincsem_park_greyhound so it can coexist with Hungary Thoroughbred's hu_kincsem_park registry and storage identity; reran the complete calendar/pre/post day under the new artifact and handoff paths.

Full validation

The complete suite passes 123 tests with ResourceWarning treated as an error.
Canada captured 14/14 current entry cards and 9/9 meetings from the selected results day; HVT captured a complete starter/result day through the approved proxy.
SNAI captured 11/11 current cards and 3/3 selected-day results; Kincsem captured its year control and complete pre/post day; HRNZ captured 2/2 current fields and 8/8 published results.
GRI captured all 29 published current cards and 9/9 selected-day results. One future 15-August card remained not-ready after a bounded recheck.
HRA captured 25/26 current cards, but remained partial for a complete national results day: two bounded dates yielded 2/7 and 1/3 ready result pages.
Every stored full-test artifact passed recomputed byte-length and SHA-256 checks plus metadata and FormParser handoff identity/URI validation.

Safety and limitations

All requests used the shared 15-25-second host limiter, bounded retries, kill switch, and normal source readiness validation.
HVT used the approved Racing and Sports public-GET proxy after the known direct challenge. No authentication, CAPTCHA, licensing boundary, or access control was bypassed.
Listed but unpublished future pages remain retryable and are not classified as successful captures or transport failures.

2026-08-08 - Thoroughbred canonical artifact audit at 08:44 AEST

Audit scope and integrity

Audited real canonical downloads for all 21 Thoroughbred adapters registered at the 08:44 snapshot under C:\Users\Robert\project-data\FormCrawler\t-artifacts.
Found zero corrupt or missing artifacts, zero SHA-256 or byte-length mismatches, and zero missing FormParser handoffs.
Every crawl process belonging to the audit had finished.
The registry subsequently reached 25 Thoroughbred adapters; Hungary, UAE, Bangalore, and Hyderabad were added later and have separate successful artifact/handoff evidence in thoroughbred-source-research-2026-08.

Strong complete-day evidence

Racing Post GB: 31 pre/post pairs.
Umanity JRA: 36 pairs; Umanity NAR: 34 pairs.
Czechia: 28 pairs; Bahrain: 9 pairs; Qatar: 7 pairs.
Korea: 20 racecards plus 10 results.
Turkey and Selangor: complete meetings.

Recorded limitations

Argentina, Chile, Germany, and Uruguay were post-race only.
New Zealand used a combined meeting-lifecycle page rather than split pre/post files.
Saudi provided 12 results; racecards were unavailable.
Sweden captured three pairs while three additional pairs remained unpublished.
Spain and South Africa had valid meeting/day evidence with additional unavailable targets.
Canada and USA Equibase each had one complete meeting pair; the broad-index overqueue defect remains open.
Candidate countries without registered adapters were outside this audit.

Australia documentation consolidation

Racing Australia's national weekly index covers Australian state meetings, including NSW, Queensland, and Western Australia.
Retired the three redundant state no-source pages; country-australia-racing-australia is the single Thoroughbred Australia page.

2026-08-08 - Selangor completion review

Reviewed

Re-reviewed my_selangor_turf_club before a complete-day live run.
Confirmed the dated whole-meeting racecard is the pre-race control and discovers exactly one same-date whole-meeting results target.
Confirmed strict HTTPS host/path/query/date validation and separate readiness markers for racecards and results.

Verification gate

Four focused Selangor tests passed.
The complete 152-test suite passed with ResourceWarning promoted to an error.
Python compilation and git diff --check passed.
No outstanding implementation issue blocks the complete pre-race/results day test.

Complete-day live result

The unlimited 26 July 2026 Selangor run captured both the whole-meeting racecard and results on attempt 1 with no retry-wait or failed targets.
The 805,712-byte pre-race artifact and 545,777-byte post-race artifact each contain all 12 race sections.
Recomputed SHA-256 values match the queue and metadata, and both FormParser handoffs match their artifact, content, and metadata references.

2026-08-08 - Results-only acquisition and shared runtime behavior

Added

Added repeatable capture-date-range --page-type filtering before discovered targets enter the queue.
Added crawl_racing_australia_results.bat <start> <end> [limit] for full, bounded, discovery-only, and resumable results ranges.
Added crawl_racing_australia_today.bat for all indexed Australian meeting lifecycle pages on the local date.
Added shared command-line status routines with [HTTP], [PROXY], [DISCOVERY], [RANGE], and [QUEUE] transient states plus stable permanent summaries.
Added automatic Racing and Sports proxy fallback after a direct public GET receives 403 or 429; proxy-only mode remains available.

Safety and behavior

Results-only filtering happens before queue insertion and creates no program, acceptance, gear, or scratching targets.
Proxy fallback supports public GET only, limits both destination and gateway hosts, retains racingandsports_proxy provenance, and still applies source validation.
Failed or ineligible proxy fallback returns to the existing first-block pause, repeated-block hard stop, retry limits, and external kill switch.

Verification

A bounded 2026-08-07 Australian results run captured one weekly index, discovered five meetings, and queued five result targets only.
The 190,581-byte Coffs Harbour result returned HTTP 200 with zero retries and failures. SHA-256 b7062597bc8e307b20cd34ce9c987eba26c5bbe0b3a547939fa405e88726cb72 and the FormParser handoff matched.
Automatic challenge-to-proxy behavior and default/forced transport selection have regression coverage.
All 89 tests pass with ResourceWarning promoted to an error.

2026-08-08 - Country crawler documentation

Added

Added country-australia-racing-australia and synchronized version 2 with the weekly homepage index, inclusive date-range workflow, range launcher, direct meeting seeds, five-page bundle, 75-test status, and live range verification.
Added country-germany-deutscher-galopp and corrected its Unicode labels in version 2.
Added country-chile-hipodromo-chile and corrected its accented source title in version 2.
Added country-uruguay-maronas version 1 for meeting XML, range calendar JSON, and exact horse-profile JSON workflows.
Added country-argentina-stud-book version 1 for monthly calendar and complete meeting HTML capture.

Country-page standard

Every country page now records:

program and adapter names, source ID, racing code, ISO code, and source module;
key public pages and first-party endpoints;
capture workflow and readiness rules;
source response and artifact data structure;
batch launcher and generic CLI examples;
storage and FormParser handoff paths;
limitations, recovery, and live verification;
a reverse-chronological changelog for that specific country module.

Updated

Updated form-crawler-programs through version 3 as the index for all five country modules, current CLI commands, both Racing Australia launchers, artifact forms, storage, and Robert's proxy transport.
Updated overview through version 3 with the current five-country architecture, Racing Australia range command, 75-test status, and country-page references.
Documented Robert's Racing and Sports proxy contract and its public-GET boundary while retaining source validation and dual-host request limiting.

2026-08-07 - Candidate racing-site catalogue

Added

Added documents\SITE_LINKS.md as the local planning catalogue for Thoroughbred, Harness, and Greyhound candidate sources derived from the supplied workbook.
Added the supplied Denmark (DEN) and Morocco URLs that were missing from the workbook inventory.
Proposed stable source IDs for planning and documented source_id as the website, feed, or source-definition segment in <discipline-root>\<year>\<race_code>\<source_id>.

Documentation and verification

Reformatted the catalogue as Country: Site name (source_id) followed by an individual URL line.
Verified 67 named entries, 66 URL occurrences, and 57 unique URLs with no embedded <br> tags or Markdown tables.
Updated formcrawler/scraping-urls-list through version 4 while preserving the active Deutscher Galopp configuration and common request controls.

Scope boundary

Candidate URLs are reference inventory, not approved crawl scope.
Proposed source IDs remain subject to confirmation when adapter boundaries, allowed hosts and paths, capture behaviour, and tests are designed.

2026-08-07 - Initial German post-race acquisition workflow

Source commit: 816f590 - Add German post-race meeting acquisition workflow

Added

Reusable Python acquisition package with shared models, HTTP retrieval, request policy, immutable storage, orchestration, and source-adapter registration.
Deutscher Galopp adapter for individual race URLs and race-target discovery from supported calendar, meeting, and race pages.
Complete-meeting capture starting from one race URL.
Backend-neutral crawl-state interface.
SQLite development adapter with target deduplication, discovery provenance, capture attempts, artifact history, retry scheduling, expiring leases, stale-worker protection, and queue summaries.
CLI commands: capture, discover, run-queue, capture-meeting, and queue-status.
FormParser inbox handoffs using content and metadata URIs.
Mandatory scraping safeguards: serialized same-host access, 10-second base delay, 5-15 seconds of jitter, retry/backoff limits, response-size enforcement, circuit breaking, shared local rate-limit state, and an external kill switch.
External runtime-data ownership under C:\Users\Robert\project-data.
Docker/EKS portability decision with replaceable storage, queue/state, and rate-limit adapters.

Data classification

German Thoroughbred artifacts use racing jurisdiction code GER.
ISO country metadata remains DE.
Discipline is T and artifacts are stored beneath t-artifacts.
Runtime data and downloaded content are excluded from Git; only program source, tests, scripts, and documentation are versioned.

Verification

All source modules compiled successfully.
14 unit tests passed.
Live seed race 1364507 discovered 9 meeting targets.
Queue result: 9 captured, 0 failed.
All stored content hashes and FormParser URI references were verified.
Eight live same-host request gaps fell between 15.949 and 24.660 seconds.

Known next work

Add formal FormParser validation for the 1.0.0 handoff and artifact contracts.
Define retention and repeated-snapshot content-deduplication behavior.
Add conditional HTTP requests.
Agree and implement scheduled calendar discovery.
Implement shared cloud storage, queue/state, and rate-limit adapters before EKS deployment.

2026-08-08 - Thoroughbred source expansion

Classified every supplied Thoroughbred source in documents/SITE_LINKS.md, including supplementary data and breeding/horse-profile availability.
Added or completed 20 source adapters and repeatable date-range launchers across the wider source-research run.
Live-validated Hungary, UAE, Bangalore and Hyderabad end to end, including immutable hashes, metadata and FormParser handoffs.
Stopped Saudi JCSA after the requested three validation attempts and recorded the precise validator limitation.
Published per-source status pages and thoroughbred-source-research-2026-08; the complete suite passes 152 tests with ResourceWarning promoted to an error.