FormCrawler Changelog

2026-08-08 - Country crawler documentation

Added

• Added country-australia-racing-australia version 1 for the Racing Australia five-page meeting lifecycle bundle.
• Added country-germany-deutscher-galopp and corrected its Unicode labels in version 2.
• Added country-chile-hipodromo-chile and corrected its accented source title in version 2.
• Added country-uruguay-maronas version 1 for meeting XML, range calendar JSON, and exact horse-profile JSON workflows.
• Added country-argentina-stud-book version 1 for monthly calendar and complete meeting HTML capture.

Country-page standard

Every country page now records:

• program and adapter names, source ID, racing code, ISO code, and source module;
• key public pages and first-party endpoints;
• capture workflow and readiness rules;
• source response and artifact data structure;
• batch launcher and generic CLI examples;
• storage and FormParser handoff paths;
• limitations, recovery, and live verification;
• a reverse-chronological changelog for that specific country module.

Updated

• Updated form-crawler-programs to version 2 as the index for all five country modules, current CLI commands, launchers, artifact forms, storage, and Robert's proxy transport.
• Updated overview to version 2 with the current five-country architecture, operating commands, 71-test status, and country-page references.
• Documented Robert's Racing and Sports proxy contract and its public-GET boundary while retaining source validation and dual-host request limiting.

<!-- site-catalogue-changelog:start -->

2026-08-07 - Candidate racing-site catalogue

Added

• Added documents\SITE_LINKS.md as the local planning catalogue for Thoroughbred, Harness, and Greyhound candidate sources derived from the supplied workbook.
• Added the supplied Denmark (DEN) and Morocco URLs that were missing from the workbook inventory.
• Proposed stable source IDs for planning and documented source_id as the website, feed, or source-definition segment in <discipline-root>\<year>\<race_code>\<source_id>.

Documentation and verification

• Reformatted the catalogue as Country: Site name (source_id) followed by an individual URL line.
• Verified 67 named entries, 66 URL occurrences, and 57 unique URLs with no embedded <br> tags or Markdown tables.
• Updated formcrawler/scraping-urls-list through version 4 while preserving the active Deutscher Galopp configuration and common request controls.

Scope boundary

• Candidate URLs are reference inventory, not approved crawl scope.
• Proposed source IDs remain subject to confirmation when adapter boundaries, allowed hosts and paths, capture behaviour, and tests are designed. <!-- site-catalogue-changelog:end -->

2026-08-07 - Initial German post-race acquisition workflow

Source commit: 816f590 - Add German post-race meeting acquisition workflow

Added

• Reusable Python acquisition package with shared models, HTTP retrieval, request policy, immutable storage, orchestration, and source-adapter registration.
• Deutscher Galopp adapter for individual race URLs and race-target discovery from supported calendar, meeting, and race pages.
• Complete-meeting capture starting from one race URL.
• Backend-neutral crawl-state interface.
• SQLite development adapter with target deduplication, discovery provenance, capture attempts, artifact history, retry scheduling, expiring leases, stale-worker protection, and queue summaries.
• CLI commands: capture, discover, run-queue, capture-meeting, and queue-status.
• FormParser inbox handoffs using content and metadata URIs.
• Mandatory scraping safeguards: serialized same-host access, 10-second base delay, 5-15 seconds of jitter, retry/backoff limits, response-size enforcement, circuit breaking, shared local rate-limit state, and an external kill switch.
• External runtime-data ownership under C:\Users\Robert\project-data.
• Docker/EKS portability decision with replaceable storage, queue/state, and rate-limit adapters.

Data classification

• German Thoroughbred artifacts use racing jurisdiction code GER.
• ISO country metadata remains DE.
• Discipline is T and artifacts are stored beneath t-artifacts.
• Runtime data and downloaded content are excluded from Git; only program source, tests, scripts, and documentation are versioned.

Verification

• All source modules compiled successfully.
• 14 unit tests passed.
• Live seed race 1364507 discovered 9 meeting targets.
• Queue result: 9 captured, 0 failed.
• All stored content hashes and FormParser URI references were verified.
• Eight live same-host request gaps fell between 15.949 and 24.660 seconds.

Known next work

• Add formal FormParser validation for the 1.0.0 handoff and artifact contracts.
• Define retention and repeated-snapshot content-deduplication behavior.
• Add conditional HTTP requests.
• Agree and implement scheduled calendar discovery.
• Implement shared cloud storage, queue/state, and rate-limit adapters before EKS deployment.