# Ticket: main core + SMTP worker fetch EDM templates from console2 Copy the title and description below into ClickUp. Assign to the team that owns `karma-subito/karma` (main core and the SMTP worker). No work in this repo. --- ## Title **EDM: fetch email templates from console2 instead of SendGrid's template store** --- ## Description ### Why this exists Marketing EDMs are currently stored in SendGrid as dynamic templates and merged on SendGrid's servers. SendGrid caps how many stored templates an account can hold, and we are at that cap — no new EDM can be created until something is deleted. The templates have therefore been moved into our own storage, managed in console2 (the admin panel). SendGrid stays exactly as it is and keeps sending every email. The only change is where the HTML comes from: instead of asking SendGrid to fill in a stored template, we fetch the template from console2, fill it in ourselves, and hand SendGrid the finished HTML to deliver. Nothing about this ticket changes anything in SendGrid. No template is deleted, no setting is touched, and mail keeps going out through the same account. ### What already exists (nothing to build) console2 already exposes the API. It is live and testable today. This ticket is only about calling it from the main core / SMTP worker. An id-based lookup, returning the template with its placeholders still in place: ``` GET {CONSOLE2_CORE_URL}/v1/internal/edm/templates/{templateId}?fallback=none ``` ### How to call it Two credentials are accepted. Both are the same shared secret, `INTERNAL_SERVER_API_KEY`, which must match the value set on the console2 server. **1. Plain key — use this to try it by hand** ```bash curl -i "http://127.0.0.1:9005/v1/internal/edm/templates/k-3f9a2c1b?fallback=none" \ -H "Authorization: Bearer $INTERNAL_SERVER_API_KEY" ``` `x-api-key` works identically if that is easier: ```bash curl -i "http://127.0.0.1:9005/v1/internal/edm/templates/k-3f9a2c1b?fallback=none" \ -H "x-api-key: $INTERNAL_SERVER_API_KEY" ``` **2. Signed envelope — preferred for the worker** The key is encrypted into a header instead of being sent as-is, and a captured request stops working after 60 seconds. Same endpoint, one different header: ```bash curl -i "http://127.0.0.1:9005/v1/internal/edm/templates/k-3f9a2c1b?fallback=none" \ -H "x-api-payload: :" ``` The envelope is AES-256-GCM over `{"key":"","ts":}`, with the key itself used as the 32-byte encryption key, the 12-byte IV sent as the first half of the header, and the GCM auth tag appended to the ciphertext. A working implementation to copy is in this repo at `tools/edm-vscode/src/api.ts` (`buildApiPayload`). Start with the plain key to confirm connectivity, then switch to the envelope. ### What comes back **200 — the template is ours and has a published version** ```json { "success": true, "data": { "templateId": "k-3f9a2c1b", "versionId": "8f0c…", "subject": "Your {{first_name}} summer offer", "html": "…{{first_name}}…", "plain": "…", "generatePlain": true, "source": "local" } } ``` `html`, `subject` and `plain` still contain the `{{placeholders}}`. Filling them in per recipient is the worker's job — that is the part SendGrid used to do. `source` says which store answered, so a caller can tell without comparing ids. **404 — not ours** ```json { "success": false, "message": "Template not found" } ``` This is a normal, expected answer, not a failure. It means the template has not been moved yet, and the worker should send it the way it does today: pass SendGrid the `template_id` and let SendGrid merge it. Every template starts out in this state, which is what makes the rollout gradual and reversible. **Keep `?fallback=none` on every worker call.** Without it, console2 will fetch the template from SendGrid on our behalf and return it as if it were ours. That is useful for the console and the editor, and wrong for the worker: it turns "send this via SendGrid, unchanged" into "render SendGrid's HTML with our own engine", which is exactly the behaviour change the staged rollout exists to avoid. ### What the worker needs to do 1. **Look up the template** by the id it already has, with `?fallback=none`. 2. **On 200** — fill in the placeholders with the same data it currently sends to SendGrid, then send the finished HTML through SendGrid as an ordinary email. 3. **On 404, or if console2 cannot be reached** — send exactly as it does today, via SendGrid's `template_id`. This path must keep working untouched; console2 being down must never stop mail going out. 4. **Keep the existing per-send tag** (`customArgs` carrying the template id) so reporting can still attribute a send to a template once SendGrid's own per-template stats no longer apply. 5. **Cache** a fetched template briefly in memory. A campaign sends the same template thousands of times and should not re-fetch it for each recipient. The placeholder syntax is Handlebars, the same as SendGrid's dynamic templates. Reference implementations of the fetch-and-render path and the merge helpers already exist and can be copied rather than written from scratch — ask the console2 side for `packages/edm-client`. One thing to watch when rendering: an unknown helper used **with arguments** throws rather than rendering as blank. A template that referenced something SendGrid provided and we do not will fail loudly at send time, so it is worth sending a handful of real templates through before switching any campaign over. ### Configuration needed Two values on the worker: | Name | Meaning | |---|---| | `CONSOLE2_CORE_URL` | Base URL of console2 core, no trailing slash | | `INTERNAL_SERVER_API_KEY` | Must match the value set on console2 | Note for whoever deploys this: the worker currently has no `.env` wired up, and the `env_file` line in `compose.dev.yml` is commented out. That needs sorting before either value can be read. ### Roll it out gradually Nothing has to switch over at once, and nothing should: 1. Point the worker at console2 and confirm a lookup returns 200 for one template that has been moved, and 404 for one that has not. 2. Send that one template to internal addresses and compare it against the same template sent the old way — subject line, images, links, and how it looks in Gmail, Outlook and Apple Mail. 3. Move one low-risk campaign. 4. Widen only after that campaign has gone out cleanly. ### Done when - [ ] The worker fetches from console2 and sends the rendered result via SendGrid - [ ] A 404, a timeout, and an unreachable console2 all fall back to sending via SendGrid's `template_id`, verified deliberately and not just assumed - [ ] Templates are cached rather than re-fetched per recipient - [ ] Per-send template tagging still present - [ ] One real template compared side by side against a SendGrid-merged send in Gmail, Outlook and Apple Mail, and signed off - [ ] `CONSOLE2_CORE_URL` and `INTERNAL_SERVER_API_KEY` deployed to the worker ### Out of scope - Any change in SendGrid. No template deleted, no setting changed. - Deleting SendGrid's copies to reclaim quota — separate decision, irreversible, and only after local sending has been stable for a long while. - Transactional mail (the ~70 `MAIL_TEMPLATES` ids). Those stay on SendGrid permanently, by decision. ### Who to ask The console2 side owns the API and the template store. Anything about the response shape, a template that 404s unexpectedly, or the rendering helpers goes there.