Scraping URL's List
Version 3 · Tidied the candidate catalogue by placing each URL on its own Markdown table row and documented source_id as the stable directory source name.
Historical versionScraping URL's List
Purpose
This page distinguishes sources that FormCrawler is configured to access from candidate racing websites supplied for future discovery and capture work. Active sources include their executable identifiers, allowed hosts, supported entry points, and implementation status. Candidate sources are reference inventory only and are not approved crawl scope.
The source-adapter registry remains the executable authority. Update this page whenever an active source adapter, allowed host, control URL, supported page pattern, or candidate source catalogue entry is added, changed, disabled, or removed.
Active sites
| Status | Source | Source ID | Country | Discipline | Racing code | Canonical site |
|---|---|---|---|---|---|---|
| Active | Deutscher Galopp | de_deutscher_galopp |
Germany (DE) |
Thoroughbred (T) |
GER |
https://www.deutscher-galopp.de/ |
Deutscher Galopp
Allowed hosts
www.deutscher-galopp.dedeutscher-galopp.de
Only HTTPS is accepted. Valid URLs are canonicalized to www.deutscher-galopp.de; allowing a host does not permit unrestricted crawling of every path on that host.
Supported URL patterns
| Page type | URL or pattern | What FormCrawler uses it for |
|---|---|---|
| Calendar/control page | https://www.deutscher-galopp.de/gr/renntage/ |
Captures the racing calendar page and discovers in-scope race links when they are present. |
| Meeting page | https://www.deutscher-galopp.de/gr/renntage/{meeting_id}/?d={YYYYMMDD} |
Captures one dated meeting page and discovers its individual race links. {meeting_id} must be numeric. |
| Individual race page | https://www.deutscher-galopp.de/gr/renntage/rennen.php?id={race_id}&d={YYYYMMDD}&s=R |
Captures a pre-race or post-race race page. For the current post-race workflow, s=R is used and the page must show that a result is available before it is accepted. |
Verified example
https://www.deutscher-galopp.de/gr/renntage/rennen.php?id=1364507&d=20260802&s=R
This seed URL was used on 2026-08-07 to discover all nine races at the 2026-08-02 meeting. The queue completed with nine captured targets and no failures.
Implementation reference
The registered adapter is src\formcrawler\sources\de\deutscher_galopp.py. It validates the HTTPS scheme, allowed hosts, supported paths, numeric identifiers, date format, and required query values before any capture is attempted.
<!-- candidate-site-inventory:start -->
Candidate source inventory
Last updated: 2026-08-07
Purpose
This page records candidate racing source websites for FormCrawler discovery and capture work. It is an inventory only: inclusion does not mean that a source adapter exists or that the URL has been technically validated.
Sources and conventions
- Base inventory:
C:\Users\Robert\Downloads\Site Links.xlsx, supplied on 2026-08-07. - Denmark (
DEN) and Morocco links: supplied directly by Robert on 2026-08-07. - Disciplines use the FormCrawler codes
T(Thoroughbred),H(Harness), andG(Greyhound). - Country labels and URLs are preserved from the workbook unless a clearer country name is shown with the workbook label in a note.
- Multiple URLs for one country are retained because they may expose different calendars, race cards, results, or regional coverage.
Source identity and directories
Every implemented input source needs a stable machine-readable source_id. It is the source-name segment in the artifact path:
<discipline-root>\<year>\<race_code>\<source_id>\<event_slug>-<event_date>\
For example:
t-artifacts\2026\GER\de_deutscher_galopp\
In that example, GER is the controlled racing jurisdiction code and de_deutscher_galopp is the source_id. The identifier names the website, feed, or source definition—not an individual downloaded file. Multiple control, calendar, meeting, race, or result URLs can share one source_id when one adapter owns them. Use separate source IDs when sites require independent adapters, host rules, or capture behaviour.
Candidate links below do not receive a source_id until their source boundary and adapter design are agreed.
Thoroughbred (T)
Workbook jurisdictions without a supplied Thoroughbred URL: Australia (NSW), Australia (QLD), Australia (WA), Austria, Ireland, Peru, Panama, and Mexico.
Harness (H)
| Country or jurisdiction | Candidate source URL | Notes |
|---|---|---|
| Australia | https://www.harness.org.au/ | |
| New Zealand | https://www.hrnz.co.nz/ | |
| France | https://www.letrot.com/ | |
Denmark (DEN) |
https://www.paris-turf.com/ | Appears in the workbook. |
Denmark (DEN) |
https://www.swedishhorseracing.com/races#/calendar/2022-06-29 | Added from Robert's supplied links on 2026-08-07. |
| Belgium | https://belgiumhorseracing.be/ | |
| Finland | https://heppa.hippos.fi/mobiili/races/results | |
| Netherlands | https://ndr.nl/selectieproeven/ | |
| Norway | https://www.racebets.com/en/horse-racing/ | Workbook association preserved. |
| Norway | http://swedishhorseracing.com/ | Workbook association preserved. |
| Spain | https://www.federaciobaleardetrot.com/ | |
| Hungary | https://mla.kincsempark.hu/results/trotting/2025-02-08 | |
| Sweden | https://sportapp.travsport.se/start | |
| Sweden | http://swedishhorseracing.com/ | |
| USA | https://racing.ustrotting.com/ | |
| Canada | https://standardbredcanada.ca/ | |
| Germany | https://www.hvtonline.de/ | Workbook label: German. |
| Italy | https://ippica.snai.it/partenti |
Workbook countries without a supplied Harness URL: Austria and Malta.
Greyhound (G)
| Country or jurisdiction | Candidate source URL | Notes |
|---|---|---|
| Australia | https://www.thedogs.com.au/racing/ | |
| Australia | https://www.grv.org.au/ | |
| New Zealand | https://www.grnz.co.nz/catch-the-action/fields.aspx | |
| United Kingdom | https://greyhoundbet.racingpost.com/ | |
| Ireland | https://www.grireland.ie/racing/upcoming-race-cards/ | |
| Hungary | https://mla.kincsempark.hu/results/trotting/2025-02-08 | Workbook URL references trotting; retained without reclassification. |
Maintenance
When a source is implemented, keep this inventory as the human-readable catalogue and record operational source identifiers, allowed hosts, discovery URLs, capture phases, and page types in versioned non-secret FormCrawler configuration. Do not place credentials or session material in this page. <!-- candidate-site-inventory:end -->
Request controls applying to every configured site
Every discovery, pre-race, post-race, and supplementary request must pass through FormCrawler's central policy layer. Current defaults serialize requests per host and apply a 10-second base delay plus 5-15 seconds of jitter, giving a 15-25-second interval between same-host request starts.
The common policy also applies bounded retries and backoff, response-size limits, circuit breaking, duplicate URL removal, shared local rate-limit state, and the external STOP kill switch. Source-specific settings may be stricter; relaxing a default requires Robert's explicit approval.
Candidate sources are not configured
The candidate inventory above is not a list of permitted crawl targets. Promoting a candidate requires a separate source adapter or explicit source definition, a stable source ID, restricted hosts and paths, capture phase and artifact type, tests, and an update to the active-site section before live requests are enabled.
Maintenance checklist
When adding or changing a source:
- Record the organization, source ID, country, discipline, and racing code.
- List exact allowed hosts and require HTTPS unless an explicit exception is approved.
- List control URLs and supported page patterns; do not describe an entire domain as crawlable when only selected paths are supported.
- Record whether each endpoint is discovery, pre-race, post-race, or supplementary capture.
- Add adapter validation and discovery tests before making live requests.
- Confirm that all requests use the common queue and scraping policy.
- Update this page and the FormCrawler changelog with the delivered source capability.