changelog

FormCrawler Changelog

Version 11 · Record seven harness/greyhound adapters, jurisdiction pages, uncapped full-day validation, integrity checks, and bounded limitations.

Historical version

FormCrawler Changelog

2026-08-08 - Harness and greyhound jurisdictions and full-day validation

Added

Implemented seven raw-acquisition adapters: Harness Racing Australia, HRNZ, Standardbred Canada, HVT harness, SNAI harness, Greyhound Racing Ireland, and Kincsem Park greyhound.
Added one source-specific jurisdiction page for each adapter, including routes, commands, raw artifact and handoff structure, supplementary data, breeding/profile availability, limitations, and module history.
Corrected the Kincsem catalogue entry from trotting to the official greyhound calendar, card, and result routes.
Renamed the Greyhound source to hu_kincsem_park_greyhound so it can coexist with Hungary Thoroughbred's hu_kincsem_park registry and storage identity; reran the complete calendar/pre/post day under the new artifact and handoff paths.

Full validation

The complete suite passes 123 tests with ResourceWarning treated as an error.
Canada captured 14/14 current entry cards and 9/9 meetings from the selected results day; HVT captured a complete starter/result day through the approved proxy.
SNAI captured 11/11 current cards and 3/3 selected-day results; Kincsem captured its year control and complete pre/post day; HRNZ captured 2/2 current fields and 8/8 published results.
GRI captured all 29 published current cards and 9/9 selected-day results. One future 15-August card remained not-ready after a bounded recheck.
HRA captured 25/26 current cards, but remained partial for a complete national results day: two bounded dates yielded 2/7 and 1/3 ready result pages.
Every stored full-test artifact passed recomputed byte-length and SHA-256 checks plus metadata and FormParser handoff identity/URI validation.

Safety and limitations

All requests used the shared 15-25-second host limiter, bounded retries, kill switch, and normal source readiness validation.
HVT used the approved Racing and Sports public-GET proxy after the known direct challenge. No authentication, CAPTCHA, licensing boundary, or access control was bypassed.
Listed but unpublished future pages remain retryable and are not classified as successful captures or transport failures.

2026-08-08 - Selangor completion review

Reviewed

Re-reviewed my_selangor_turf_club before a complete-day live run.
Confirmed the dated whole-meeting racecard is the pre-race control and discovers exactly one same-date whole-meeting results target.
Confirmed strict HTTPS host/path/query/date validation and separate readiness markers for racecards and results.

Verification gate

Four focused Selangor tests passed.
The complete 152-test suite passed with ResourceWarning promoted to an error.
Python compilation and git diff --check passed.
No outstanding implementation issue blocks the complete pre-race/results day test.

Complete-day live result

The unlimited 26 July 2026 Selangor run captured both the whole-meeting racecard and results on attempt 1 with no retry-wait or failed targets.
The 805,712-byte pre-race artifact and 545,777-byte post-race artifact each contain all 12 race sections.
Recomputed SHA-256 values match the queue and metadata, and both FormParser handoffs match their artifact, content, and metadata references.

2026-08-08 - Results-only acquisition and shared runtime behavior

Added

Added repeatable capture-date-range --page-type filtering before discovered targets enter the queue.
Added crawl_racing_australia_results.bat <start> <end> [limit] for full, bounded, discovery-only, and resumable results ranges.
Added crawl_racing_australia_today.bat for all indexed Australian meeting lifecycle pages on the local date.
Added shared command-line status routines with [HTTP], [PROXY], [DISCOVERY], [RANGE], and [QUEUE] transient states plus stable permanent summaries.
Added automatic Racing and Sports proxy fallback after a direct public GET receives 403 or 429; proxy-only mode remains available.

Safety and behavior

Results-only filtering happens before queue insertion and creates no program, acceptance, gear, or scratching targets.
Proxy fallback supports public GET only, limits both destination and gateway hosts, retains racingandsports_proxy provenance, and still applies source validation.
Failed or ineligible proxy fallback returns to the existing first-block pause, repeated-block hard stop, retry limits, and external kill switch.

Verification

A bounded 2026-08-07 Australian results run captured one weekly index, discovered five meetings, and queued five result targets only.
The 190,581-byte Coffs Harbour result returned HTTP 200 with zero retries and failures. SHA-256 b7062597bc8e307b20cd34ce9c987eba26c5bbe0b3a547939fa405e88726cb72 and the FormParser handoff matched.
Automatic challenge-to-proxy behavior and default/forced transport selection have regression coverage.
All 89 tests pass with ResourceWarning promoted to an error.

2026-08-08 - Country crawler documentation

Added

Added country-australia-racing-australia and synchronized version 2 with the weekly homepage index, inclusive date-range workflow, range launcher, direct meeting seeds, five-page bundle, 75-test status, and live range verification.
Added country-germany-deutscher-galopp and corrected its Unicode labels in version 2.
Added country-chile-hipodromo-chile and corrected its accented source title in version 2.
Added country-uruguay-maronas version 1 for meeting XML, range calendar JSON, and exact horse-profile JSON workflows.
Added country-argentina-stud-book version 1 for monthly calendar and complete meeting HTML capture.

Country-page standard

Every country page now records:

program and adapter names, source ID, racing code, ISO code, and source module;
key public pages and first-party endpoints;
capture workflow and readiness rules;
source response and artifact data structure;
batch launcher and generic CLI examples;
storage and FormParser handoff paths;
limitations, recovery, and live verification;
a reverse-chronological changelog for that specific country module.

Updated

Updated form-crawler-programs through version 3 as the index for all five country modules, current CLI commands, both Racing Australia launchers, artifact forms, storage, and Robert's proxy transport.
Updated overview through version 3 with the current five-country architecture, Racing Australia range command, 75-test status, and country-page references.
Documented Robert's Racing and Sports proxy contract and its public-GET boundary while retaining source validation and dual-host request limiting.

2026-08-07 - Candidate racing-site catalogue

Added

Added documents\SITE_LINKS.md as the local planning catalogue for Thoroughbred, Harness, and Greyhound candidate sources derived from the supplied workbook.
Added the supplied Denmark (DEN) and Morocco URLs that were missing from the workbook inventory.
Proposed stable source IDs for planning and documented source_id as the website, feed, or source-definition segment in <discipline-root>\<year>\<race_code>\<source_id>.

Documentation and verification

Reformatted the catalogue as Country: Site name (source_id) followed by an individual URL line.
Verified 67 named entries, 66 URL occurrences, and 57 unique URLs with no embedded <br> tags or Markdown tables.
Updated formcrawler/scraping-urls-list through version 4 while preserving the active Deutscher Galopp configuration and common request controls.

Scope boundary

Candidate URLs are reference inventory, not approved crawl scope.
Proposed source IDs remain subject to confirmation when adapter boundaries, allowed hosts and paths, capture behaviour, and tests are designed.

2026-08-07 - Initial German post-race acquisition workflow

Source commit: 816f590 - Add German post-race meeting acquisition workflow

Added

Reusable Python acquisition package with shared models, HTTP retrieval, request policy, immutable storage, orchestration, and source-adapter registration.
Deutscher Galopp adapter for individual race URLs and race-target discovery from supported calendar, meeting, and race pages.
Complete-meeting capture starting from one race URL.
Backend-neutral crawl-state interface.
SQLite development adapter with target deduplication, discovery provenance, capture attempts, artifact history, retry scheduling, expiring leases, stale-worker protection, and queue summaries.
CLI commands: capture, discover, run-queue, capture-meeting, and queue-status.
FormParser inbox handoffs using content and metadata URIs.
Mandatory scraping safeguards: serialized same-host access, 10-second base delay, 5-15 seconds of jitter, retry/backoff limits, response-size enforcement, circuit breaking, shared local rate-limit state, and an external kill switch.
External runtime-data ownership under C:\Users\Robert\project-data.
Docker/EKS portability decision with replaceable storage, queue/state, and rate-limit adapters.

Data classification

German Thoroughbred artifacts use racing jurisdiction code GER.
ISO country metadata remains DE.
Discipline is T and artifacts are stored beneath t-artifacts.
Runtime data and downloaded content are excluded from Git; only program source, tests, scripts, and documentation are versioned.

Verification

All source modules compiled successfully.
14 unit tests passed.
Live seed race 1364507 discovered 9 meeting targets.
Queue result: 9 captured, 0 failed.
All stored content hashes and FormParser URI references were verified.
Eight live same-host request gaps fell between 15.949 and 24.660 seconds.

Known next work

Add formal FormParser validation for the 1.0.0 handoff and artifact contracts.
Define retention and repeated-snapshot content-deduplication behavior.
Add conditional HTTP requests.
Agree and implement scheduled calendar discovery.
Implement shared cloud storage, queue/state, and rate-limit adapters before EKS deployment.

2026-08-08 - Thoroughbred source expansion

Classified every supplied Thoroughbred source in documents/SITE_LINKS.md, including supplementary data and breeding/horse-profile availability.
Added or completed 20 source adapters and repeatable date-range launchers across the wider source-research run.
Live-validated Hungary, UAE, Bangalore and Hyderabad end to end, including immutable hashes, metadata and FormParser handoffs.
Stopped Saudi JCSA after the requested three validation attempts and recorded the precise validator limitation.
Published per-source status pages and thoroughbred-source-research-2026-08; the complete suite passes 152 tests with ResourceWarning promoted to an error.