SmartMail Architecture

Evidence-based trace of how SmartMail actually behaves at runtime (not just intended design). See also Overview, Database, API Reference, Workflow.

1. App boot sequence (app/main.py, module import time)

Triggered by run.py, service_wrapper.py, or wsgi.py — all three import app.main.app, so all three inherit identical boot-time side effects:

1. Flask(__name__) created once; app.secret_key is hardcoded ("change-this-key") — should be an env-sourced secret.
2. ensure_default_categories() seeds default Mongo categories (wrapped in try/except).
3. Two background threads start unconditionally:
   • start_background_summary_poller(app) (app/poller_mongo.py) — the real, active mail-to-summary poller (see §3).
   • start_background_email_sync(app) (defined in main.py) — loops every 60s calling sync_inbox_to_mongo(), which is a literal pass. No-op, safe to remove.
4. Blueprints registered: bp_category_prompt_api, bp_gpt_prompt_api, bp_people_api, bp_compose_api, automation_bp. bp_inbox is imported and even extended with a route later in the file, but never passed to register_blueprint() — so /api/inbox and /api/mongodb/recent-inbox are unreachable dead code.
5. app/config.py (imported transitively by nearly everything) does hard, import-time validation — missing OPENAI_API_KEY/CLIENT_ID/TENANT_ID/SCOPES raises and crashes the process at startup, in every deployment mode.

2. Request lifecycle

• GET /outlook renders a static SPA shell (outlook.html) with no server-side Graph/Mongo calls; the page then drives everything through JSON APIs (/api/folders, /api/emails/<folder>, etc.), each independently calling get_access_token_silent() per request.
• A legacy parallel route, GET /inbox, calls a guard (_require_token()) that raises SystemExit(1) if unauthenticated — hitting it unauthenticated kills the whole waitress worker process, not just returns 401. It also writes to Mongo synchronously on every hit via upsert_graph_message().
• Two independent, overlapping in-process caches exist for similar Graph data: app/graph.py's _CACHE (TTL keyed by token hash) and main.py's own smaller _FOLDER_LIST_CACHE.

3. Background sync — five modules claim to do this, only one really does

Module	Started at boot?	Verdict
app/poller_mongo.py start_background_summary_poller	Yes	The one real, active sync. Every 15–20s, lists messages via Graph, and for anything not yet in Mongo email_summaries, cleans the body, calls OpenAI (app/llm.py), and writes both the raw cache and the summary. Respects DISABLE_BACKGROUND_SUMMARY=true.
main.py's start_background_email_sync	Yes	Dead/no-op — calls a pass-only function every 60s.
app/poller.py	No	Fully commented out.
app/background_inbox_sync.py	No (never called)	Would ImportError if ever wired up — imports a non-existent app.graph_client module.
app/services/inbox_sync.py	No (never called)	Dead helper functions duplicating logic already inline in main.py/app/email_store.py.
app/automation1.py run_polling_loop/poll_inbox	No	Dormant SQL-based ingestion path (different cursor mechanism, IngestCursor SQL table); GPT calls inside it are commented out.
automation1.py's /graph/webhook route	Registered, but inert	Handles Graph's validation handshake, but create_subscription() (needed to register the webhook with Graph) is never called — Graph never sends this endpoint anything.

Net: there are effectively three unrelated "have we processed this message" mechanisms (Mongo existence-check, a JSON cursor file app/ingest_cursor.json, and a SQL IngestCursor table) — only the Mongo one, inside poller_mongo.py, is live.

4. Auth / token flow

Two divergent MSAL implementations exist:

• app/auth.py — the one actually imported everywhere (main.py, routes). Uses a relative cache.bin (relative to process CWD, not app/config.TOKEN_CACHE_PATH), scopes limited to ["Mail.Read"], and shadows its own import of auth_public's function with a locally redefined one of the same name.
• app/auth_public.py — implements the real device-code flow with the broader scope set from app/config.py and the absolute app/cache.bin path, but is effectively unused (nothing calls it); its own get_token() has a tuple-unpacking bug that would raise if ever called.

Because of this split, the token cache actually read/written at runtime is CWD-relative cache.bin via app/auth.py — app/config.TOKEN_CACHE_PATH (app/cache.bin) is a second, unrelated file on disk that only the unused module touches. Runtime flow: get_access_token_silent() loads cache.bin, tries acquire_token_silent; if that fails, routes just return 401 (no in-request interactive fallback). Interactive/device-code login only happens via the CLI (login subcommand in run.py/app/main.py), which writes the cache the web server later reads silently.

5. AI pipeline

Two independent AI paths exist, targeting different stores:

A. Live Mongo path (used by the real poller and by "open a message"): app/graph.py fetch → HTML-to-text cleanup (duplicated near-verbatim in main.py and poller_mongo.py) → prompt built inline → app/llm.py:_summarise_llm() (OpenAI or Azure OpenAI, default model gpt-4o-mini, env OPENAI_MODEL) → save_email_summary() into Mongo email_summaries.

• Category classification is not an LLM call. infer_category_dynamic() does keyword/sender substring matching against a category's prompt field (parsed into rules) — despite the UI calling it a "prompt", there's no GPT call in classification.
• Draft generation (/api/draft/generate) is where an admin-configured prompt genuinely reaches GPT: it pulls a category-specific reply prompt from the Mongo gpt_prompts collection (key category_reply::<category>, falling back to reply_default), merges it with the rolling thread summary and any operator free-text instructions, and calls app/gpt.py:gpt_reply_draft() (model from OPENAI_MODEL_REPLY), then creates/updates the Graph draft.
• Two independent rolling thread-summary implementations are both live, on different collections/stores: app/thread_summarizer.py (SQLAlchemy ThreadSummary, likely a pre-Mongo leftover) vs. app/summariser.py:roll_thread_summary() (Mongo thread_summaries, used by the live "update thread summary" endpoint). Also note: app/database_mongo.py defines save_thread_summary/get_thread_summary twice with incompatible signatures — the second definition silently wins at import time; only the thread_id-keyed version is ever callable.

B. Dormant SQL/CLI path — app/gpt.py's gpt_summarise/gpt_kv_extract/this file's gpt_classify are reachable only via CLI subcommands (not the web app). app/graph.py also has its own third gpt_classify implementation referencing an undefined client — would NameError if called; looks like copy-pasted dead code.

Prompts are configured in two Mongo collections with different roles: categories.prompt (keyword-matching rules for auto-tagging) and gpt_prompts (actual GPT reply-drafting templates, editable from the admin GPT Prompt tab).

6. Deployment topologies

Both modes import the exact same app.main.app object and inherit identical boot side effects — the difference is purely in the launcher:

• Desktop exe (run.py + SmartMail.spec) — PyInstaller one-file build, console app. Checks for a cached token and exits with instructions (SmartMail.exe login) if missing, starts waitress on 0.0.0.0:5001 in a background thread, opens the browser at /outlook. Interactive, foreground, single-user.
• Windows Service (service_wrapper.py + SmartMail_Service.spec) — pywin32 ServiceFramework, service name SmartMailService. No token check and no browser launch; starts the same Flask app under waitress on the same 0.0.0.0:5001, headless. Logs to service_debug.log in overwrite mode (truncated on every restart — no rotation/append). If the token cache is stale, the service still starts and just serves 401s until login is run separately.
• wsgi.py — a bare application = app WSGI entrypoint, not referenced by either .spec or launcher; see Workflow for the third deployment path that actually uses it (run_api.cmd + waitress-serve).
• Celery (app/celery_app.py) defines a task but nothing ever calls .delay()/enqueues it, and no worker is started anywhere — vestigial, not part of the live system.

Known dead code / duplication (flagged for cleanup, not treated as load-bearing)

• app/poller.py — fully commented out.
• app/background_inbox_sync.py — would ImportError (imports non-existent app.graph_client).
• app/services/inbox_sync.py — zero callers.
• bp_inbox blueprint — imported/extended but never registered.
• app/routes/category_prompt_api.py — three Blueprint objects share one variable name; only the least useful survives to registration.
• app/database_mongo.py — save_thread_summary/get_thread_summary defined twice; second wins.
• main.py's start_background_email_sync — 60s loop calling a pass-only function.
• app/automation1.py webhook — registered but never receives real Graph traffic (create_subscription() never called).
• app/celery_app.py — no task ever enqueued.
• app/graph.py:gpt_classify — duplicate logic, references undefined client.
• app/auth.py vs app/auth_public.py — two divergent MSAL implementations, only one live.

Security note

Hardcoded fallback credentials (a live-looking MongoDB URI with embedded username/password, plus default OpenAI/Graph secrets) appear as os.getenv(..., "<hardcoded value>") defaults in several files (app/database.py, app/database_mongo.py, app/mongo_client.py, app/mongo_emails.py, app/main.py), and real secrets are committed in plaintext in .env/env/.env.development. This should be remediated (move to a secrets manager or at minimum stop hardcoding fallback defaults) — flagged here rather than reproduced.