AISSISTANT — Local Voice AI
Personal project · Apr 2025 – Jul 2025
A privacy-first voice assistant that runs LLM inference and custom text-to-speech entirely locally — built to prove that a useful multimodal assistant doesn't need to ship your data to a cloud API.
100%
on-device, zero external APIs
OCR→LLM→TTS
multimodal pipeline
Fine-tuned
custom neural TTS voice
Problem
Voice assistants are surveillance devices by default: audio, screen content, and queries get shipped to someone else's servers. For sensitive contexts — reading private documents aloud, asking questions about personal data — that's disqualifying.
AISSISTANT runs the entire loop locally: it can see (OCR over screen content or documents), reason (local LLM inference), and speak (a fine-tuned neural TTS model), with nothing leaving the machine. The engineering challenge is doing all three at usable latency on consumer hardware.
Architecture
Voice input
local speech capture
Screen / document
OCR extraction
Local LLM inference
on-device · no API calls
Fine-tuned neural TTS
custom voice · streaming audio synthesis
Playback
low-latency audio pipeline
Technical Decisions
Fully local, even where cloud would be easier
Every component — speech, OCR, LLM, TTS — was chosen or built to run on-device. This forces real engineering trade-offs (model size vs. quality vs. latency) that calling an API hides from you, and it makes the privacy guarantee absolute rather than a policy promise.
Fine-tuning TTS instead of using a stock voice
Stock open-source TTS voices are serviceable but generic, and they fall apart on unusual phonetic input. Fine-tuning a neural TTS model gave control over voice identity and let me directly target the failure cases I was seeing — at the cost of owning the training loop, dataset preparation, and evaluation myself.
OCR as the second modality, not a camera feed
The most useful 'vision' for a desktop assistant is reading what's on screen or in a document — OCR text is compact, feeds directly into LLM context, and is dramatically cheaper than running a vision model locally. It was the highest-value multimodal capability per unit of compute.
Challenges
TTS inference latency
Inference latency became the bottleneck as I pushed for natural-sounding output — a multi-second wait before the assistant starts speaking destroys the conversational feel. I cut perceived latency by chunking synthesis at phrase boundaries and streaming audio as it was generated, so playback starts while the rest of the response is still being synthesized, and by profiling the synthesis path to keep the model warm between turns.
Robustness at phonetic edge cases
Fine-tuned TTS models behave well on text resembling their training distribution and badly elsewhere: abbreviations, code identifiers, mixed-language fragments, numbers. I built an evaluation set specifically from these edge cases and iterated on text normalization ahead of the model — fixing input representation turned out to be more effective per hour invested than more fine-tuning epochs.
Noisy, ambiguous multimodal input
OCR output from real screens is messy — broken line ordering, UI chrome mixed into content, partial words. Feeding it raw into the LLM produced confidently wrong answers. The fix was a cleanup-and-structure pass between OCR and the LLM, plus prompting that makes the model acknowledge unreadable regions instead of hallucinating over them.
Thinking about misuse
A voice-cloning-capable TTS pipeline has obvious spoofing risks. I evaluated output consistency across speakers and inputs partly to understand how easily the system could be misused, which shaped what I built — voice identity stays a local artifact, not a hosted service anyone can hit.