← Projects

HCC CRM
Email Module

A bulk-email system for Hill Country Coders’ internal CRM: queued sends with attachments, database-driven templates, and a dual-provider fallback so a broken Gmail OAuth token doesn’t take down outbound email. Built on top of a CRM backend I also shipped general features and bug fixes for.

Employer Hill Country Coders
Role Backend / Full-stack Engineer
Stack Node.js · Express · MongoDB · Bull / Redis · Next.js
Repos emailControllerAuth2 · HCC-adam-backend · hcc-admin-v2

The Problem

One email at a time
doesn’t scale.

Hill Country Coders’ internal CRM needed to send bulk email to contact lists, outreach, notifications, campaign-style sends, with attachments, reusable templates, and personalization per recipient. The existing send path was synchronous, single-provider (Gmail OAuth only), and had no mechanism for tracking a job of 500+ recipients as anything other than one big pass/fail.

None of that survives contact with reality: attachments need to outlive a single HTTP request, templates change constantly and shouldn’t require a deploy, and a single provider’s OAuth token expiring shouldn’t mean outbound email stops entirely. The system needed to be queued, provider-agnostic, and data-driven, while staying simple enough to run on a modest VPS alongside the rest of the CRM.

Requirements

What the queue
had to survive.

Four constraints that shaped the send pipeline before any code:

  1. How do attachments survive a job queue?

    Bull jobs are serialized to Redis as JSON, and a raw file buffer can’t travel in the payload. Files needed to live somewhere addressable by reference, not by value.

  2. What happens when Gmail OAuth fails?

    The existing sendEmail() silently swallowed errors. A failed send, for any reason, including an expired refresh token, needed to surface, and ideally have a fallback path rather than just failing louder.

  3. Can templates change without a deploy?

    Templates started as three hardcoded IDs mapped to static .html files. Business users needed to add and edit templates themselves: templates had to become data.

  4. How does the frontend know a job’s status?

    An admin needed to see queued → processing → completed for a bulk send without the system needing a push-based transport for what is, in practice, an internal admin tool at modest scale.

Architecture Diagram

One send request,
five workers, two gaps.

Admin UI hcc-admin-v2 API emailControllerAuth2 POST bulk-send uploads/ Multer diskStorage writes file, keeps path only Bull queue Redis, job payload = JSON enqueues job path ref only 5 concurrent workers re-reads uploads/ by path once per recipient Worker disk re-read, per send attempts: 3, retries entire batch not just the failed recipient Gmail OAuth createGmailClient() SendGrid 50-batch throttle route: gmail route: sendgrid Recipient inbox job.status queued · processing · completed · failed, no partial 500 sent, 3 failed still reads "completed"
One admin action enqueues one Bull job. Five workers pull from the same queue, and each re-opens the attachment from disk per recipient rather than carrying it in the job payload, since a raw buffer can’t survive Redis’s JSON serialization. Two paths are drawn dashed: the retry loop, which re-sends the whole batch on any failure, and uploads/, which nothing ever cleans up.

Gap · disk leak

No unlink call exists anywhere. Every attachment written to uploads/ stays there after the job completes or fails, so disk usage grows without bound.

Gap · batch-level retry

attempts: 3 is scoped to the whole job, not the recipient. A transient failure late in a 2,000-person send risks re-sending to everyone who already got the email.

Working as intended

File-path references keep Redis job payloads small regardless of attachment size, and per-recipient failures are still caught and logged individually, even though the top-level status field can’t show it.

Architecture Decisions

Queue, disk,
and a fallback provider.

arch-01 disk-based-attachment-references-not-buffers-in-job-payload
Decision Multer configured with diskStorage, not memoryStorage. Files land in uploads/ on the initial request; only .path and .originalname go into the Bull job payload. gmailService.js and sendgridService.js re-read from disk by path at send time, once per recipient.
Why Bull jobs are serialized to Redis as JSON, and a raw buffer can’t survive that. A file reference can.
Rejected memoryStorage + base64-encoding into the job payload, which bloats Redis job size for any real attachment and still requires re-encoding per recipient anyway.
arch-02 templates-as-mongo-data-instead-of-hardcoded-html-files
Decision A Template Mongo model, controller, routes, and seeding: templates became queryable, mutable data.
Why Any new template previously required a code change and a deploy. This is also the foundational refactor that later enabled merge-tag whitelist validation: whitelist validation needs templates to be data first.
Rejected Continuing to add hardcoded template IDs and static .html files as the template count grew.
arch-03 sendgrid-fallback-path-instead-of-hardening-gmail-oauth
Decision Changed sendEmail() to re-throw instead of swallowing errors, then added a service parameter letting callers route through SendGrid instead of Gmail OAuth. The Gmail path stayed as-is, as one option among two.
Why createGmailClient() and its token-refresh logic were fragile (no try/catch around getAccessToken(), no locking around the token-update read-then-save). Rather than debug fragile auth code under time pressure, a second, more reliable provider became the practical fix.
Rejected Hardening the Gmail OAuth refresh logic itself: correct in principle, but not the pragmatic path given ownership and deadline constraints.
arch-04 60s-polling-instead-of-websocket-sse-for-job-status
Decision setInterval + axios GET every 60 seconds, plus a manual refresh button. A pollingActive ref stops the interval on fetch error instead of continuing to hit a broken endpoint.
Why This is an internal admin tool at modest scale, not a consumer real-time product. Polling is simple, low-risk, and has no perf or stale-status bug history.
Rejected WebSocket/SSE push: more infrastructure than the actual need justified at this scale.

Problems Hit

Two bugs
worth writing down.

selection-drawer-lag-useState-to-useRef-plus-virtualization

Symptom Bulk-selection UI (contact list drawer) got sluggish as selection size grew, and every checkbox toggle felt laggy.
Root cause useState({}) for the selection map meant every toggle triggered a full object-spread copy (O(n) per toggle) plus a full re-render of the component tree. The real cost was React re-render overhead, not the underlying computation: a web worker would not have fixed this.
Fix Replaced useState with a mutable useRef for the selection map (no copy, no re-render on mutation), added a cheap version counter to trigger re-render only when needed, and added react-window’s FixedSizeList with ResizeObserver-driven sizing to virtualize the selected-members panel.

local-dev-workers-sharing-a-single-cloud-redis-with-prod

Symptom Inherited docker-compose wired only to a single cloud Redis endpoint (Redis Labs) with hardcoded credentials, with no isolated local Redis for development.
Root cause An unnamespaced send-bulk-email queue name and a single shared credential-bearing compose file meant local dev workers and production shared the same queue.
Fix Flagged this during onboarding and added docker-compose.redis-local.yml, giving local dev its own isolated Redis on a distinct port before it could cause a real collision with production jobs.

What Shipped

Queued, provider-agnostic,
and data-driven.

Queue & Attachments: Bull v4 / Redis

  • Serialized job queue

    Bull v4 processes bulk sends across 5 concurrent workers, with attempts: 3 job-level retry.

  • Disk-based attachments

    Multer diskStorage, file references (not buffers) in the job payload, re-read per recipient at send time.

Templates

  • Database-driven templates

    Template Mongo model, controller, routes, seeding: add or edit without a deploy.

  • Merge-tag whitelist

    Fixed 7-tag vocabulary (firstName, lastName, company, email, senderName, senderTitle, bookingLink) enforced backend and mirrored client-side for immediate feedback.

Sending: Gmail / SendGrid Dual Path

  • Provider fallback

    Service parameter routes a send through SendGrid or Gmail OAuth; errors re-thrown instead of swallowed, so callers record per-recipient failures.

  • Per-recipient personalization

    Delivery logging and per-recipient template substitution layered around SendGrid’s existing batch throttle.

Frontend: hcc-admin-v2

  • Bulk-selection drawer

    useRef-backed selection map + react-window virtualization for large contact lists.

  • Job status polling

    60s interval + manual refresh; stops polling on fetch error rather than spamming a broken endpoint.

Infrastructure

  • Local / prod Redis separation

    Dedicated docker-compose.redis-local.yml isolates local dev queues from production.

Open Gaps

What’s honestly
still unresolved.

Things I’d fix next, in priority order.

Highest priority IDOR: session-based ownership replaced with a client-supplied param

requireAuth middleware was deleted entirely. Contact-list and bulk-job endpoints switched their ownership check from req.user.id (session-derived) to req.params.userId (caller-supplied), so any caller who knows or guesses a valid Mongo ObjectId can read or delete another user’s contact lists or bulk-job status. This was an explicit, documented stopgap to unblock frontend integration before Google OAuth session wiring was finished, a defensible call under deadline pressure at the time, but it was never closed out once that blocking condition was resolved, and no later commit re-adds the session-based check. This is the single highest-priority unresolved item on this page: re-introduce req.user.id-derived ownership checks, and if userId is still needed as a param, validate it against the authenticated session rather than trusting it.

Unmitigated XSS in template rendering

processTemplate() is a plain regex .replace() with no HTML-escaping of recipient-controlled values (name, company, email) and no double-substitution guard. A contact whose name field contains <script> flows directly into outbound email HTML. A full-history grep for xss/sanitize/escape returns zero hits in either repo; the frontend template builder also accepts raw pasted/uploaded HTML with no sanitization. Fix: HTML-escape recipient-controlled values before substitution, and add DOMPurify or equivalent on the frontend builder’s raw HTML input.

No partial-job-state tracking

Job status is exactly queued / processing / completed / failed: no partial state. A job with 500 successful sends and 3 failures still reports completed. Per-recipient failures are caught and logged individually, so the underlying data exists, but it just isn’t surfaced in the top-level status field.

No per-recipient retry or idempotency

Bull’s attempts: 3 retries the entire recipient batch, not individual failed recipients. A transient failure late in a 2000-recipient job risks re-sending to recipients who already succeeded, since there’s no dedupe check on retry. The more standard Bull pattern here would be per-recipient sub-jobs with independent retry.

Unaddressed SendGrid rate-limit bursts under concurrency

With 5 concurrent Bull workers each processing 50-recipient batches across simultaneous jobs, large campaigns can burst past SendGrid’s API rate limits with zero automated recovery with no 429-specific handling, no backoff-and-resume. The existing throttle constants weren’t designed with multi-job concurrency in mind. Fix: explicit 429 detection with exponential backoff and job-level pause/resume.

Attachment disk leak and filename-collision risk

No unlink/cleanup call exists anywhere: temp files in uploads/ are never deleted after a job completes or fails, so disk usage grows unbounded. Separately, Multer’s filename generator is Date.now() + ext with no UUID or per-request namespacing, which with 5 concurrent workers has a theoretical collision risk under simultaneous multi-file uploads in the same millisecond (inferred from the code, no confirmed incident).

Curious about the
identity / OAuth layer?

Keycloak/Cognito dual-IdP work, JWT audience-validation, and role/group sync for this same platform are documented separately since it’s a distinct, shared piece of infrastructure rather than part of the email module or CRM backend covered here.

Get in touch See all projects Identity / OAuth infra →