TD Architecture (Infrastructure & Deployment)

This is the infrastructure view — environments, deployment targets, network topology, external dependencies, ports/domains. For the business-logic architecture (what the services actually do), see Overview, Scraper & File Generator Process, Database Reference, and the API references (TroyenAPI, TroyenDataHelpers).

Two independent deployment tiers, two different CI stories

IIS tier (TroyenAPI, TroyenDataHelpers, TroyenDataWeb) — has real CI: .github/workflows/TD-DEV.yml (branch dev) and TD-PROD.yml (branch main), both running on self-hosted Windows runners ([self-hosted, windows, x64, td-dev] / td-prod). Structurally identical: dotnet restore/build/publish → force-kill w3wp.exe/dotnet.exe for the target app pool → back up/restore web.config around a recursive file copy → restart the site. No GitHub Actions secrets are used anywhere — all site names/paths are hardcoded literals in the YAML; credential management for this tier lives entirely in the deployed appsettings.json on the target box.

Docker tier (everything else — PunterScraper, TroyenDataFilesGenerator, TroyenService, TroyenRaceIngestor, NedsScraper, TroyenUploader, the Broker) — has no CI automation at all. Root-level .bat scripts (front-b&p.bat, back-b&p.bat, punter-scraper-b&p.bat, race-ingestor-b&p.bat) run docker compose build && docker compose push manually, pushing to ghcr.io/tpl-tech-titans/....

Notable overlap: front-b&p.bat also builds Docker images for TroyenAPI/TroyenDataHelpers/TroyenDataWeb — the same three services the GitHub Actions IIS pipeline deploys. There's no evidence in this repo that those Docker images are ever actually run anywhere; this repo can't tell you whether IIS or Docker is the real production path for these three, or whether they serve genuinely different environments. Confirm with whoever runs ops before assuming either one is authoritative.

Deployment matrix

Service	DEV IIS site/pool	PROD IIS site/pool	Docker image (ghcr.io/tpl-tech-titans/…)	Container port	Compose host port	Dev (launchSettings) port
TroyenAPI	TroyonAPI	TD-External	troyen-external-api	8080	7502	6101 (IIS Express 53537)
TroyenDataHelpers	TroyenDataHelpersAPI	TD-Internal	troyen-internal-api	8080	7503	5235/7140 (IIS Express 48938)
TroyenDataWeb	TroyenDataWeb	TD-Web	troyen-web	8080	7501	5035/7255 (IIS Express 8042)
TroyenDataFilesGenerator	—	—	troyen-data-gen	5100 (default)	5100:5100	56730/56731
TroyenRaceIngestor	—	—	troyen-race-ingestor	5101 (default)	${INGESTOR_API_PORT:-5101}:5101	56726/56727
NedsScraper	—	—	ned-scraper	8080	7510	56728/56729
PunterScraper	—	—	punter-scraper	none (CLI/cron, no listener)	none published	none (CLI args)
TroyenService	—	—	troyen-services (main) / troyen-service (jobs variant)	none (cron-only)	none published	none (console)
TroyenUploader	—	—	tpl/troyen-uploader (different registry namespace — not ghcr.io/tpl-tech-titans)	7890	7890:7890	—
Broker	—	—	broker	3000 (compose default)	${PORT:-3000}:3000	—

Note the DEV vs PROD IIS site-name mismatch: TroyonAPI→TD-External, TroyenDataHelpersAPI→TD-Internal, TroyenDataWeb→TD-Web — same service, different site name per branch. Don't assume a site name generalizes across environments.

Broker port inconsistency: the Broker's compose file defaults to host port 3000, but every actual consumer (.env.sample's BROKER_URLS, Common's appsettings Brokers array, PunterScraper's launchSettings.json) points at ports 5900/5901 on two separate hosts (e.g. http://138.226.222.210:5900,...:5901) — the real deployment must override PORT per instance; the compose file's own default is misleading read in isolation.

Docker network topology

No root-level compose file, no networks: block anywhere — every project's docker-compose.yml is fully standalone, each getting Compose's own implicit default network. Containers cannot reach each other by service name even if co-located on the same host — confirmed by TroyenDataWeb's own config pointing at sibling services via raw host IP + published port (http://138.226.222.210:5100, :5101), not a Compose service name. Everything talks over the host's public network stack, not an internal Docker network.

TroyenRaceIngestor's compose file is the only one with a host bind-mount, and it hardcodes an Ubuntu path (/home/ubuntu/TD/TroyenRaceIngestor/Log:/app/logs) — confirms the Docker host is Linux (Ubuntu), not Windows (the IIS tier is the separate Windows side).

External dependencies

Domain / host	Purpose
s3.troyendata.com	Self-hosted S3-compatible storage endpoint (all buckets, below).
internal.troyendata.com	Public URL for TroyenDataHelpers.
external.troyendata.com	Public URL for TroyenAPI.
silk.troyendata.com	Public URL serving silk images — likely a CNAME/alias in front of the troyen-silk S3 bucket, not independently confirmed.
rundeck.troyendata.com	Rundeck instance #1 — app code POSTs to two webhooks here: Clear_web_cache and Clear_CF_cache (Common/CacheHelper.cs). The actual Cloudflare zone purge happens inside that Rundeck job — no Cloudflare API call or zone ID exists anywhere in this repo, so the target zone/domain can't be confirmed from source.
rundeckthor.troyendata.com	Rundeck instance #2, a different host — linked from TroyenDataWeb's in-app Neds-scraper fallback guide for a human to manually trigger a scrape job. Don't conflate the two Rundeck hosts; they serve different purposes.
api.punters.com.au / www.punters.com.au	Primary scrape target (PunterScraper, PunterWebScraper).
api.racenet.com.au / www.racenet.com.au	Form-history scrape target (PunterScraper, RacenetScraper).
puntapi.com	The GraphQL/REST backend actually shared by both punters.com.au and racenet.com.au front-ends.
nedsform.com.au	NedsScraper's target.
tabapi.racingandsports.com	RS meetings API (Common appsettings ApiSettings.MeetingApiBaseUrl).
57.181.204.168:8085	External scratchings/stats/meetings API host (labeled "Sanka" in the in-app Neds guide) — getScratchingsGrey, getStatsGrey, api/v1/neds/meetings.
api.openai.com	LLM provider #1 — model chatgpt-4o-latest for config-driven calls (the pipeline's hard-coded generation calls use GPT-4o-mini specifically, see Scraper & File Generator Process).
generativelanguage.googleapis.com	LLM provider #2 — Google Gemini accessed via its OpenAI-compatibility endpoint (.../v1beta/openai), model gemini-2.0-flash. No Azure OpenAI endpoint exists anywhere in the repo. DefaultLLM = OpenAi.
smtp.zoho.com	Outbound email ([email protected]).
api.ipify.org	Public-IP lookup used by the cookie/proxy-refresh logic — not business data, just anti-bot plumbing.

S3 buckets

Bucket	Reader/writer
troyon-watchdog	Flag-file signal bucket, PunterScraper → TroyenDataFilesGenerator (see pipeline doc). Also watched by TroyenRaceIngestor.
troyen-data	Generated RaceCard/DataDump JSON output.
punter-cache	Default raw-scrape cache bucket ("skip if already scraped" logic).
troyen-resources	Shared resource files — proxy lists, transformation assets.
troyen-silk	Jockey/driver silk images; served publicly via silk.troyendata.com.
punter-web-scraper	Intake bucket TroyenRaceIngestor polls for pre-supplied meeting JSON.
troyen-watcher	Manual test-upload target for the standalone TroyenUploader dev tool (pending/ prefix).
troyen-gen	TroyenRaceIngestor's own completion-flag bucket, separate from troyon-watchdog — only found in design docs, not in .env.sample or any compose env list; unconfirmed whether it's actually configured in production.
rob-rp-videos	Race replay video storage, read by TroyenDataHelpers.RacesController.

Monitoring

There's no centralized monitoring — no Sentry/Application Insights/Datadog/New Relic anywhere in the repo. What exists:

• Per-service heartbeat endpoints (TroyenAPI's heart-beat, TroyenDataHelpers' heartbeat — naming differs, see TroyenDataHelpers Reference).
• PunterScraper's Docker HEALTHCHECK is self-contained inside the container (--health-check CLI flag) — not exposed over the network or wired to any external monitor.
• TroyenDataWeb ships a browser-side dashboard (wwwroot/js/dashboard/main.js) that polls the internal API's heartbeat/cache-clear endpoints and the file-generator/ingestor queue-inspection endpoints on a timer — this is the closest thing to a status page, but it only runs while someone has the admin portal open, not a server-side alerting system.
• Rundeck is used the opposite way you might expect: application code/UI calls into Rundeck (cache-clear webhooks, a manual "run this job" link), Rundeck does not poll the apps' health.

Security posture (infra-relevant, not covered elsewhere)

• Plaintext secrets are committed to git: Properties/launchSettings.json for TroyenAPI, TroyenDataHelpers, TroyenDataWeb, TroyenService, PunterScraper, plus Common/appsettings/common.appsettings.{Development,Local,Production}.json all contain live-looking Mongo connection strings, S3 keys, an OpenAI key, a Google Gemini key, JWT signing secrets, and an internal API pass-key, in plaintext, in source control. This is in addition to the hardcoded RS-MySQL/S3-uploader fallback credentials already flagged in Overview — the scope is broader than just those two spots.
• Multiple, possibly-stale Mongo hosts appear across configs: current launchSettings.json/.env.sample point at 62.171.228.224:20007 and 138.226.222.210:20007; older-looking entries in Common/appsettings/common.appsettings.*.json reference 102.130.127.112:20007 and 62.171.228.166:5900/5901. This repo alone can't tell you which are live vs. stale — worth confirming with whoever manages the Mongo hosts before treating any of them as current.
• No reverse proxy/gateway artifact exists in-repo (no nginx/Traefik/HAProxy config, no web.config checked in). Whatever SSL termination and domain routing fronts internal.troyendata.com/external.troyendata.com/silk.troyendata.com (and whether Cloudflare actually sits in front of them, given the Clear_CF_cache webhook) lives entirely outside this repo — confirm with DNS/network ops rather than assuming a topology.