FormCrawler Changelog

<!-- site-catalogue-changelog:start -->

2026-08-07 - Candidate racing-site catalogue

Added

• Added documents\SITE_LINKS.md as the local planning catalogue for Thoroughbred, Harness, and Greyhound candidate sources derived from the supplied workbook.
• Added the supplied Denmark (DEN) and Morocco URLs that were missing from the workbook inventory.
• Proposed stable source IDs for planning and documented source_id as the website, feed, or source-definition segment in <discipline-root>\<year>\<race_code>\<source_id>.

Documentation and verification

• Reformatted the catalogue as Country: Site name (source_id) followed by an individual URL line.
• Verified 67 named entries, 66 URL occurrences, and 57 unique URLs with no embedded <br> tags or Markdown tables.
• Updated formcrawler/scraping-urls-list through version 4 while preserving the active Deutscher Galopp configuration and common request controls.

Scope boundary

• Candidate URLs are reference inventory, not approved crawl scope.
• Proposed source IDs remain subject to confirmation when adapter boundaries, allowed hosts and paths, capture behaviour, and tests are designed. <!-- site-catalogue-changelog:end -->

2026-08-07 - Initial German post-race acquisition workflow

Source commit: 816f590 - Add German post-race meeting acquisition workflow

Added

• Reusable Python acquisition package with shared models, HTTP retrieval, request policy, immutable storage, orchestration, and source-adapter registration.
• Deutscher Galopp adapter for individual race URLs and race-target discovery from supported calendar, meeting, and race pages.
• Complete-meeting capture starting from one race URL.
• Backend-neutral crawl-state interface.
• SQLite development adapter with target deduplication, discovery provenance, capture attempts, artifact history, retry scheduling, expiring leases, stale-worker protection, and queue summaries.
• CLI commands: capture, discover, run-queue, capture-meeting, and queue-status.
• FormParser inbox handoffs using content and metadata URIs.
• Mandatory scraping safeguards: serialized same-host access, 10-second base delay, 5-15 seconds of jitter, retry/backoff limits, response-size enforcement, circuit breaking, shared local rate-limit state, and an external kill switch.
• External runtime-data ownership under C:\Users\Robert\project-data.
• Docker/EKS portability decision with replaceable storage, queue/state, and rate-limit adapters.

Data classification

• German Thoroughbred artifacts use racing jurisdiction code GER.
• ISO country metadata remains DE.
• Discipline is T and artifacts are stored beneath t-artifacts.
• Runtime data and downloaded content are excluded from Git; only program source, tests, scripts, and documentation are versioned.

Verification

• All source modules compiled successfully.
• 14 unit tests passed.
• Live seed race 1364507 discovered 9 meeting targets.
• Queue result: 9 captured, 0 failed.
• All stored content hashes and FormParser URI references were verified.
• Eight live same-host request gaps fell between 15.949 and 24.660 seconds.

Known next work

• Add formal FormParser validation for the 1.0.0 handoff and artifact contracts.
• Define retention and repeated-snapshot content-deduplication behavior.
• Add conditional HTTP requests.
• Agree and implement scheduled calendar discovery.
• Implement shared cloud storage, queue/state, and rate-limit adapters before EKS deployment.