← Back to InsightsLocal LLM

Building OfflineMail AI: A Fully Offline AI Email Assistant with Local LLMs

· Islam Nabi · OfflineMail AI · Ollama + Qwen3 + FastAPI

OfflineMail AI architecture diagram, Next.js frontend, FastAPI backend, local Ollama inference, and SQLite persistence with localhost-only transport enforcement
End-to-end OfflineMail AI architecture: five feature routes share one SSE streaming pipeline from FastAPI through Ollama to the browser, with all data persisted locally in SQLite.

How I built a privacy-first email writing assistant that runs entirely on-device, with no OpenAI, no Claude, no Gemini, and no data ever leaving the machine.

Most AI writing tools (Grammarly, ChatGPT plugins, email co-pilots) share one architectural assumption: your text has to leave your machine to get "smart." It gets sent to someone else's API, processed on someone else's infrastructure, and in many cases retained for model improvement. For privacy-conscious individuals, regulated industries, or anyone who simply doesn't want a draft email routed through a third party, that assumption is a dealbreaker.

OfflineMail AI is my answer to that: a complete AI email-writing assistant (generate, rewrite, grammar-correct, change tone, and generate subject lines) where every single inference call happens locally, through Ollama running a Qwen3 model on my own hardware. No API keys. No cloud calls. No telemetry. If you unplug the internet, it still works.

Why build this instead of just calling an API

Calling GPT-4o or Claude for this would have taken an afternoon. Building it to run fully offline took real engineering, for reasons that map directly to production concerns:

  • Latency and cost at scale: a local 4B–8B parameter model has zero per-token cost and no network round-trip, which matters if this were embedded in a high-volume internal tool.
  • Data residency and compliance: legal, healthcare, and enterprise clients frequently cannot send drafts containing PII to a third-party API, full stop.
  • No vendor lock-in: swapping the model is an environment variable change, not a rewrite.
  • It forces you to solve the hard parts yourself: structured output, streaming UX, and prompt reliability aren't hidden behind a polished hosted API; you own the whole stack.

System architecture

The diagram above shows the full stack: a Next.js frontend with five feature routes sharing one SSE consumer hook, a FastAPI backend with decoupled API, service, prompt, and persistence layers, and local inference via Ollama with structured JSON output enforced before results are saved to SQLite history.

Backend: FastAPI + Python 3.12, structured with clean architecture boundaries, API routes, service layer, prompt layer, and persistence layer are fully decoupled, so the LLM client is swappable behind an interface and testable in isolation.

Frontend: Next.js 16 + React 19 + TypeScript, with a shared SSE-consuming hook (useSSEStream) reused across all five features, and TanStack Query managing server state for history.

Database: SQLite + async SQLAlchemy + Alembic migrations: no external database service, consistent with the "everything runs locally" constraint.

The part I'm most proud of: privacy enforced at the transport layer

It's easy to say an app doesn't call external APIs. It's harder to make that a guarantee rather than a convention that one future code change could silently break. So instead of just "not adding" external API clients, I built a custom httpx transport that validates every single outbound HTTP request against an allowlist of localhost, 127.0.0.1, and ::1, and raises an error if anything else is targeted.

That means the offline guarantee isn't a design intention documented in a README. It's enforced in code, at the network layer, regardless of what any individual service or future contributor writes. This is the kind of constraint I think about often in production AI systems: make the safety property structural, not conventional.

Making a local model feel instant

Running inference locally means no GPT-4o-class hardware backing your request, so perceived speed had to be engineered, not assumed:

  • Server-Sent Events (SSE) streaming from FastAPI to the browser, so tokens render as they're generated instead of the UI blocking on a full response.
  • think: false + a /no_think prompt directive: Qwen3 supports an extended reasoning mode that's great for hard problems but adds latency and clutters output for a task like this; disabling it means tokens start streaming immediately.
  • VRAM-aware model selection: qwen3:4b runs 100% on GPU on modest hardware (6GB VRAM), while qwen3:8b is available for machines with more headroom, swappable via a single environment variable.

Getting structured output from a local LLM

Five different features (generate, rewrite, grammar, tone, subject) all need to return the same shape: {subject, body, closing}. Getting a local model to reliably emit valid JSON (not prose, not markdown fences, not a "here's your email:" preamble) needed two layers of enforcement: Ollama's native JSON-schema-constrained output mode, plus a Pydantic validation pass on the backend that catches anything that slips through. Between the two, output is now reliable enough to persist directly to history without manual review.

What's next

A natural extension is local RAG over previously sent emails, so the assistant can match your actual writing style rather than a generic tone preset, fully offline, using the same architecture. I'd also like to benchmark output quality across Qwen3, Llama 3, and Mistral variants to give users a real accuracy/speed tradeoff instead of a single default.

For cloud-based AI Deployment and MLOps patterns at production scale, see my vehicle insurance risk assessment case study, a useful contrast to the fully on-device approach taken here.

If you're exploring what's realistic to build fully offline with today's local models, or need an AI system that can't send data to a third party by design, this is exactly the kind of problem I like working on. Get in touch →

Related Services

Building a privacy-first local LLM application or need a Generative AI Engineer for on-device inference? See my Generative AI Engineering and AI Product Engineering services.