Germany Crawler - Deutscher Galopp

Last reviewed: 2026-08-08 AEST

Module identity

• Country: Germany
• Program/source name: Deutscher Galopp
• Adapter source ID: de_deutscher_galopp
• Python module: src\formcrawler\sources\de\deutscher_galopp.py
• Launcher: scripts\crawl_deutscher_galopp_results.bat
• Discipline: Thoroughbred (T)
• Racing jurisdiction code: GER
• ISO country code: DE
• Allowed hosts: deutscher-galopp.de, www.deutscher-galopp.de

Scope and key pages

Page	Route	Purpose
Calendar	/gr/renntage/	Discover in-scope race links from the meeting calendar.
Results index	/gr/renntage/ergebnisse/	Discover currently published result race links.
Meeting	/gr/renntage/<meeting-id>/?d=YYYYMMDD	Discover every race at one dated meeting.
Individual race	/gr/renntage/rennen.php?id=<race-id>&d=YYYYMMDD&s=<view>	Capture one post-race HTML artifact.

Race URLs require one numeric id, one d value in YYYYMMDD, and a non-empty s view value. Canonical URLs use the www.deutscher-galopp.de host.

Capture workflow

The standard batch workflow captures the results index, discovers every current race link, registers normalized targets in SQLite, and drains the queue sequentially. A complete-meeting workflow can instead start with one known race and discover its sibling race links.

Every request uses the shared queue, worker leases, retry state, 15-25-second same-host spacing, response-size limit, circuit breaker, and external kill switch. A race is stored successfully only after the page contains the post-race marker Status: Ergebnis.

Source and artifact data structure

Control pages and race results are stored as immutable HTML. Race target metadata contains:

• race_id from id;
• date from d as YYYYMMDD;
• view from s;
• page_type=race;
• capture_phase=post_race;
• artifact_type=race_page.

Operational file labels are copied from source-native HTML classes where available:

• racetrack for venue;
• racetitle for race title;
• pageNaviCurrent for race number.

These labels improve storage navigation only; FormParser remains responsible for racing-field extraction.

C:\Users\Robert\project-data\FormCrawler\t-artifacts\<year>\GER\de_deutscher_galopp\<event>\
C:\Users\Robert\project-data\FormParser\inbox\T\<year>\GER\de_deutscher_galopp\<artifact-id>.json

A typical readable artifact resembles:

düsseldorf-2026-08-02\01-persil-rennen-i-hkjc-world-pool-cup--1364507--<artifact-prefix>.html

How to run

Discover and capture the currently listed results:

.\scripts\crawl_deutscher_galopp_results.bat

Capture a complete meeting from one race:

python scripts\formcrawler.py capture-meeting `
  --source de_deutscher_galopp `
  --seed-race-url "https://www.deutscher-galopp.de/gr/renntage/rennen.php?id=1364507&d=20260802&s=R"

Run discovery and queue processing separately:

python scripts\formcrawler.py discover `
  --source de_deutscher_galopp `
  --url "https://www.deutscher-galopp.de/gr/renntage/ergebnisse/"
python scripts\formcrawler.py run-queue --source de_deutscher_galopp --limit 10000
python scripts\formcrawler.py queue-status

To route public GET retrieval through Robert's proxy, add --use-racingandsports-proxy to the generic command or set FORMCRAWLER_USE_RACINGANDSORTS_PROXY=1 before running the batch launcher.

Known limitations and recovery

• The results-index launcher reflects what the source currently lists; it is not a historical date-range interface.
• A race without Status: Ergebnis remains not ready and must be retried later.
• FormCrawler preserves HTML and metadata but does not map German fields into canonical racing JSON.
• Stop with Ctrl+C and rerun the same launcher to resume the persistent queue.

Verification

On 2026-08-07, seed race 1364507 for Düsseldorf on 2026-08-02 discovered and captured all nine meeting races. The queue completed with nine captured and zero failed, every stored SHA-256 and FormParser URI matched, and eight observed request gaps ranged from 15.949 to 24.660 seconds. All 17 German artifacts were later migrated and hash-verified in the readable Unicode storage layout.

Module changelog

2026-08-07 - Results-index and operational improvements

• Added results-index discovery and crawl_deutscher_galopp_results.bat.
• Added non-scrolling progress, block-response hard-stop handling, and source-native artifact labels.
• Migrated and verified all 17 existing German artifacts and handoffs into the readable Unicode layout.
• Verified complete-meeting capture and 15-25-second request spacing.

2026-08-06 - Initial implementation

• Added the first FormCrawler source adapter for Deutscher Galopp race pages.
• Added calendar, meeting, and sibling-race discovery.
• Added persistent queue/state, immutable storage, and FormParser handoffs.
• Created local commit 816f590, Add German post-race meeting acquisition workflow.