Form Crawler Programs

FormCrawler has one command-line program with several commands, backed by reusable modules. All commands acquire and preserve source material; none of them parses racing values or updates the racing database.

Program entry points

scripts\formcrawler.py

This is the local launcher used from the repository. It adds the project src directory to Python's import path and starts the shared FormCrawler command-line interface, making it convenient for development without first installing the package.

python -m formcrawler

This starts the same command-line interface through src\formcrawler\__main__.py. Use it when the package is installed or when src is already on PYTHONPATH.

Installed formcrawler command

The pyproject.toml console-script entry also exposes the same interface as formcrawler. This is the intended packaged command; all three entry points ultimately run formcrawler.cli:main and therefore have identical behavior.

Available commands

capture

Downloads and stores one explicitly supplied source URL. The source adapter validates the URL, the common HTTP policy controls the request, and the artifact store writes immutable content, metadata, and a FormParser inbox handoff.

discover

Captures a supported calendar, meeting, or race page and extracts in-scope race links from that page. Discovered targets are normalized, deduplicated, and saved in the persistent crawl-state database for later capture.

run-queue

Claims ready targets from the persistent queue and captures them sequentially. It records attempts, successful artifacts, retry waits, failures, and worker leases while applying the same central request safeguards to every page.

capture-meeting

Provides the complete German post-race meeting workflow from a single seed race URL. It captures the seed page, discovers all linked races at the meeting, and processes the additional queued races without parsing their racing data.

queue-status

Displays target counts by queue status and the location of the local state database. It is a read-only operational check and makes no source-site requests.

Supporting program modules

Acquisition and policy

• formcrawler.engine coordinates a validated target, a fetcher, and an artifact writer. Its protocol-based dependencies allow the local implementations to be replaced for cloud deployment.
• formcrawler.http performs bounded HTTP retrieval, observes the external kill switch, enforces response-size limits, and works with retry and rate-limit controls.
• formcrawler.policy provides same-host serialization, the 10-second base delay plus 5-15 seconds of jitter, shared local timing state, and circuit-breaking behavior.

Discovery and source handling

• formcrawler.html_links decodes captured HTML and extracts links without interpreting racing fields.
• formcrawler.sources.base defines the common interface that every country or source adapter must implement.
• formcrawler.sources.registry maps a configured source ID to its adapter, keeping the CLI and workflow independent of country modules.
• formcrawler.sources.de.deutscher_galopp validates Deutscher Galopp URLs, builds German targets, discovers race links, and checks only whether a post-race result is available. It classifies the source as discipline T, racing code GER, and ISO country DE.

Queue, workflow, and storage

• formcrawler.workflow coordinates discovery runs, target queue processing, retry scheduling, and complete-meeting fan-out.
• formcrawler.state.base defines the backend-neutral crawl-state contract for target registration, worker claims, and result recording.
• formcrawler.state.sqlite is the local development implementation. It stores targets, discovery provenance, attempts, artifacts, retry state, and expiring worker leases in SQLite.
• formcrawler.storage writes immutable source artifacts outside the repository and creates URI-based FormParser handoff records.
• formcrawler.models contains the shared typed records passed between adapters, acquisition, state, workflow, and storage modules.

Data location

Runtime data is deliberately outside the program repository. Local commands default to C:\Users\Robert\project-data; Linux containers default to /data, and either environment can use FORMCRAWLER_DATA_ROOT to select another external root.