← Projects

Real-Time
Chat

WhatNextPlease went from zero chat capability to a fully embedded, SSO-authenticated real-time messaging system their users never have to log into separately — and the same infrastructure is ready to onboard the next tenant with a credential form and a database row.

Client Hill Country Coders
Role Full Stack Developer
Date February 2025
Live chat-app-frontend-one-coral.vercel.app ↗
Chat interface showing direct messages, channel list, and file attachments
Rich text editor with message thread and user presence indicators
Channel view showing real-time message delivery and unread counts

The Problem

Embed chat
without building auth twice.

The client needed a real-time chat application that could be embedded inside WhatNextPlease and eventually offered to other third-party clients. There was no existing chat infrastructure — the starting point was a greenfield design discussion.

The core constraints: keep cloud costs low for an initial ~100-user scale, avoid complex distributed infrastructure, and ensure the chat could be embedded into external apps without requiring those apps to build their own auth system. A user already logged into WhatNextPlease should drop into chat with no second login prompt.

Requirements I Gathered

Six questions
that shaped the infra.

Every infra decision traces back to an answer from this list:

  1. What’s the realistic concurrent user ceiling?

    Answered: ~100 users. This single number drove the entire infrastructure simplification — single Node.js process, no message queue, no separate presence server.

    → Drove: single-server architecture decision

  2. Single-tenant or multi-tenant from day one?

    Multi-tenant from day one. This shaped every MongoDB index and middleware decision — all 9 models needed tenantId scoping.

    → Drove: AsyncLocalStorage plugin + compound index design

  3. How will external apps authenticate their users into the chat?

    HMAC-signed SSO tokens — replacing an earlier shared-login idea that would have required external apps to know internal credentials.

    → Drove: verifySSOToken middleware + 3-step tenant registration

  4. Where do tokens live in an iframe context?

    localStorage fallback, not cookies — discovered during integration. Third-party cookies are blocked by default in most browsers in cross-site iframe contexts.

    → Drove: token-in-URL-query-param SSO approach

  5. Does the embedding app need notifications when the chat iframe isn’t visible?

    Yes. The parent app needed unread badge counts even when the chat panel was collapsed or hidden.

    → Drove: ChatAppMessenger postMessage bridge architecture

  6. Should “delete conversation” be permanent or reversible?

    Per-user soft delete with restore-on-new-message — not discussed initially, emerged from a product review. If User B sends a new message to User A who deleted the conversation, only messages after the deletion should be restored.

    → Drove: deletedBy[] + deletedAt map + visibleAfter filter design

Architecture Decisions

Five calls that
shaped the system.

arch-01 single-nodejs-server-instead-of-microservices
Decision One Express + Socket.IO process handles messaging, presence, auth, and file uploads. No service mesh, no message queue.
Why At 100 users, the operational cost of service discovery, message queues, and separate presence servers exceeds the benefit by an order of magnitude.
Rejected RabbitMQ (adds ~$30–50/month, zero benefit at this scale), Zookeeper (overkill for single-node discovery), Cassandra (MongoDB sufficient for expected write volume).
arch-02 fargate-standard-not-spot-for-production
Decision Standard ECS Fargate for production. Fargate Spot retained only for dev/staging environments.
Why A Spot interruption in staging dropped all active socket connections mid-session. For a user-facing real-time backend, connection drops are unacceptable — not a graceful degradation.
Rejected Fargate Spot for production — initially chosen for ~70% cost savings, abandoned after the staging interruption incident.
arch-03 sso-via-hmac-signed-tokens-in-iframe-url-query-params
Decision HMAC-signed token passed as a URL query param to the iframe. Auth starts the moment the socket connects — no timing dependency on postMessage sequencing.
Why The original postMessage-based auth flow had race conditions — the iframe fired EMBED_READY before the parent had attached its message listener, or hasSentAuth.current persisted across remounts.
Rejected postMessage-first auth (implemented, debugged, and replaced). Cookie-based SSO (blocked by SameSite/Partitioned restrictions in cross-site iframe context).
arch-04 mongodb-tenant-isolation-via-asynclocalstorage-plugin-and-compound-indexes
Decision A Mongoose plugin reads from AsyncLocalStorage and injects tenantId automatically at the query level. Compound indexes replace global unique indexes to prevent cross-tenant key conflicts.
Why All 9 models needed tenantId scoping without requiring every query to manually include it. Manual filtering is error-prone and easily missed.
Rejected Separate database per tenant (too expensive at this stage). Query-level filtering without a plugin (error-prone — any missed query leaks cross-tenant data).
arch-05 sdk-architecture-planned-iframe-integration-shipped
Decision Iframe integration shipped for the immediate need. SDK (@yourorg/chat-sdk-core + @yourorg/chat-sdk-react in a chat-sdk/ npm workspace) designed for the long-term replacement.
Why The iframe solved the immediate integration need. The SDK eliminates cookie/localStorage restrictions, postMessage race conditions, and CORS complexity — but that’s future scope.
Rejected Mixing the SDK workspace into existing frontend/backend repos now — would have caused lockfile interference (both use npm, not pnpm).

Problems Hit

Five bugs worth
writing down.

fargate-spot-interruption-breaking-all-socket-connections

Symptom Task stopped with “Your Spot Task was interrupted.” All connected users lost their socket sessions simultaneously.
Root cause Fargate Spot capacity reclaimed by AWS. No graceful connection migration was in place for long-lived WebSocket connections.
Fix Switched production to standard Fargate. Spot retained only for dev/staging where connection drops are acceptable.

cross-tenant-duplicate-key-errors-on-fresh-local-database

Symptom Creating a second account locally threw duplicate key errors on ChannelMember.
Root cause The ChannelMember model was missing tenantId in its schema field definition. The tenantIsolationPlugin tried to inject it pre-validate, but Mongoose strict mode stripped unknown fields — so docs saved without tenantId. The global unique index on channelId + userId then correctly rejected the second doc.
Fix Added tenantId explicitly to the ChannelMember schema. Added migration_001b_drop_old_tenant_indexes to prevent recurrence after local database prunes.

hasSentAuth.current-flag-surviving-iframe-remounts

Symptom After an auth timeout and page remount, EMBED_READY fired but WNP skipped sending credentials because hasSentAuth.current was still true.
Root cause The flag was set on first auth attempt. The cleanup function didn’t reset it on unmount, so it persisted across React remounts of the component.
Fix Reset hasSentAuth.current = false both in handleLoad (on iframe reload) and in the useEffect cleanup function.

deletedAt-$unset-prematurely-removing-message-filter-boundary

Symptom When User B sent a new message to User A (who had deleted the conversation), all previous messages reappeared — not just messages after the deletion.
Root cause restoreForParticipants was incorrectly calling $unset deletedAt, which is only supposed to happen in restoreForUser. Without the timestamp, the visibleAfter filter had no boundary to apply.
Fix $unset deletedAt moved exclusively to restoreForUser. restoreForParticipants only removes the user from deletedBy array. Restore call moved to after message creation so the new message triggers the restore.

cors-failure-on-sso-init-from-iframe

Symptom POST /tenants/sso/init from the iframe returned a CORS error.
Root cause The backend’s CORS_ORIGIN was a static environment variable. The iframe’s request origin didn’t match the hardcoded value.
Fix Corrected allowed origin in env (immediate fix). Identified architectural gap: dynamic per-tenant CORS validation via database-driven allowedOrigins lookup is the permanent fix — still pending.

What Shipped

The full scope
of what went live.

Core Messaging

  • Real-time DMs + channels

    Node.js/Express + Socket.IO backend. Full direct message and channel message delivery with instant broadcast.

  • Presence system

    Redis-backed online/offline status. Socket.IO-broadcast heartbeat at 30s interval. Visible across all connected users.

  • Unread message counts

    Redis-keyed per user/conversation. Real-time Socket.IO updates whenever a new message arrives in any conversation.

  • Rich text editor

    Plate.js with full toolbar for message composition. Separate RendererKit (toolbar-free) for message display.

Authentication & Multi-tenancy

  • SSO authentication

    HMAC-signed token flow. verifySSOToken middleware. Auto-create/sync users on first login from any tenant.

  • Tenant registration system

    3-step flow: admin credential generation → client registration (registrationToken + sharedSecret) → domain verification.

  • MongoDB tenant isolation

    Plugin on all 9 models injects tenantId via AsyncLocalStorage. Compound indexes prevent cross-tenant key conflicts. Migration scripts 001–004.

Embedding & Integration

  • Two deployment routes

    Next.js frontend with /embed route for iframe deployment and /chat for standalone. Same codebase, different entry points.

  • ChatAppMessenger postMessage bridge

    CHAT_AUTH_ERROR / CHAT_REAUTH protocol for session recovery without page reload. Unread badge sync to parent app.

Files & Storage

  • File attachments

    S3 presigned URL upload flow. Thumbnail generation. CDN delivery via CloudFront. Scoped per tenant, per conversation.

  • “Delete Chat for Me”

    deletedBy[] + deletedAt map on DirectMessage. visibleAfter filter in message queries. Restore-on-new-message (messages after deletion only).

Infrastructure

  • AWS CDK stack

    ECS Fargate (standard), Network Load Balancer, ECR, S3 + CloudFront for media, Secrets Manager. All infrastructure as code.

  • Redis layer

    Presence state and unread counts. Decoupled from MongoDB so socket-heavy operations don’t compete with message persistence writes.

What I’d Do Differently

CORS policy
should have been data.

Start with dynamic CORS validation — database-driven allowedOrigins per tenant — rather than a static environment variable. The env var was a reasonable shortcut early on, but it became a blocker the first time a real tenant tried to connect from their own domain. Fixing it requires an ECS redeploy rather than a database update. For a system explicitly designed for multi-tenant embedding, the CORS policy should have been data-driven from the first deploy.

Need infrastructure
that scales with you?

Get in touch See all projects WhatsNextPlease →