Every time you say “hey” to a smart speaker, a recording of your voice leaves your house. It travels to a data center, gets transcribed, gets analyzed, and — depending on the vendor and the year — sometimes gets reviewed by a human contractor to “improve accuracy.” You didn’t consent to that specific recording being heard by a stranger. You consented to a terms-of-service document you didn’t read, for a product that was, functionally, a always-listening microphone you paid to install in your kitchen.
The good news: the technology to run a fully capable voice assistant entirely offline, with zero cloud dependency, has matured to the point where a hobbyist can build one in a weekend. No wake-word audio leaving your network. No transcript sitting in a vendor’s database. No third party deciding what “improve accuracy” means. Here’s exactly how to build it.
Why Cloud Smart Speakers Are a Worse Deal Than They Look
The pitch for Alexa, Google Home, and Siri devices is convenience. The hidden cost is that convenience requires continuous audio processing infrastructure you don’t control, and the business model behind that infrastructure isn’t just “sell you a speaker.”
The core issue is that wake-word detection creates a false sense of privacy boundary. Most people believe the device only starts listening after the wake word. In practice, several documented incidents have shown wake-word detection triggering falsely on similar-sounding phrases, sending genuine private conversations to cloud servers for processing. The device isn’t malicious — it’s just imprecise, and the fix for imprecision has historically been “send more audio to the cloud for review,” not “process more locally.”
Beyond the false-trigger problem, there’s the retention question:
- Voice recordings often persist longer than users expect, tied to account history unless manually deleted, and manual deletion doesn’t always mean the transcript is gone too.
- Third-party contractor review programs have existed at multiple major vendors, meaning a human — not just an algorithm — may have heard clips of your household.
- Voice data increasingly trains future models, meaning your speech patterns and requests contribute to a system you have no ownership stake in and no way to opt fully out of retroactively.
- Smart speakers are a single point of correlation — combined with purchase history, calendar data, and smart home device states, your voice assistant vendor builds a picture of your household routine that’s far more detailed than any single data point suggests.
None of this requires you to be paranoid to care about it. It just requires recognizing that “convenient” and “private” aren’t the same axis, and right now you don’t have to choose.
The Local Voice Stack: Three Components, Zero Cloud Calls
A fully local voice assistant needs three things working in sequence: something to detect the wake word, something to transcribe your speech, and something to generate a response and speak it back. Every piece of this can run on hardware you already own or a modest single-board computer.
1. Wake Word Detection: openWakeWord
openWakeWord is an open-source wake-word engine that runs efficiently on low-power hardware, including a Raspberry Pi. It listens continuously but does all processing locally — no audio buffer ever leaves the device until the wake word actually triggers, and even then, “leaving the device” means moving to your own local server, not the internet.
2. Speech-to-Text: Whisper (via whisper.cpp)
OpenAI’s Whisper model, running through the community-optimized whisper.cpp implementation, delivers transcription accuracy that rivals cloud services while running entirely on local CPU or GPU hardware. The smaller model variants (base, small) run comfortably on a Raspberry Pi 4 or 5; the larger variants want a dedicated GPU or a mini PC with decent compute.
3. The Assistant Brain and Text-to-Speech: Home Assistant + Piper
Home Assistant — specifically its Assist pipeline — ties the whole thing together, routing transcribed text to either a local LLM or rule-based intent matching for smart home commands, then passing the response to Piper, a fast, natural-sounding local text-to-speech engine.
Step-by-Step: Building Your Local Voice Assistant
This build assumes a Raspberry Pi 4 (4GB+) or 5 for the satellite device (the microphone/speaker unit), and either that same Pi or a separate always-on server for the processing pipeline if you want faster response times.
- Flash Raspberry Pi OS Lite to your SD card and get SSH access working — you won’t need a desktop environment for this.
- Install Home Assistant on your central server (a separate Pi, a mini PC, or a VM works fine):
curl -sL https://raw.githubusercontent.com/home-assistant/supervised-installer/main/installer.sh | bash - Set up the Wyoming protocol services, which is how openWakeWord, Whisper, and Piper communicate with Home Assistant’s Assist pipeline. Docker Compose makes this manageable:
services: openwakeword: image: rhasspy/wyoming-openwakeword ports: - "10400:10400" command: --preload-model "hey_jarvis" whisper: image: rhasspy/wyoming-whisper ports: - "10300:10300" command: --model base-int8 --language en piper: image: rhasspy/wyoming-piper ports: - "10200:10200" command: --voice en_US-lessac-medium - In Home Assistant, navigate to Settings → Voice Assistants and create a new Assist pipeline. Point the wake word, speech-to-text, and text-to-speech fields at your three Wyoming services using their respective ports.
- Configure your satellite device (the Pi with the microphone and speaker) to run the Wyoming satellite service, which streams audio to your central Home Assistant instance only after local wake-word detection fires:
wyoming-satellite --uri tcp://0.0.0.0:10700 \ --mic-command 'arecord -r 16000 -c 1 -f S16_LE -t raw' \ --snd-command 'aplay -r 22050 -c 1 -f S16_LE -t raw' - Test the pipeline end to end. Say your wake word, issue a simple command (“turn on the kitchen lights”), and confirm the response comes back through Piper’s synthesized voice.
- Layer in a local LLM for open-ended queries. Connect Ollama running a small instruction-tuned model (Llama 3.2 or similar) to your Assist pipeline’s conversation agent settings, so questions beyond simple device commands get a genuine conversational response — still fully local.
- Disable or remove any default cloud conversation agent in Home Assistant’s settings to confirm nothing falls back to an external API when local processing fails. This step gets skipped constantly, and it’s the one that actually determines whether you’ve built a private assistant or a mostly-private one.
Pro-Tip: The Silent Fallback That Undoes Everything
Here’s the mistake almost every first-time builder makes: they get the local pipeline working, feel satisfied, and never check what happens when it fails. Home Assistant’s default conversation agent configuration, in some setups, includes a cloud fallback option that’s enabled by default for reliability — meaning if your local Whisper instance times out or your LLM produces a malformed response, the query can silently route to an external service instead of just failing gracefully.
Go into Settings → Voice Assistants → your pipeline → Conversation Agent, and explicitly confirm only your local Ollama or intent-matching engine is selected — no cloud agent listed as a fallback. Then test it by deliberately stopping your Whisper container and issuing a voice command. If you get a response instead of an error, something is routing outside your network, and you need to find it.
A second, related check: review your router’s outbound traffic logs for your voice assistant devices’ IP addresses over a 24-hour period. A fully local setup should show essentially zero external traffic from those devices — if you see regular calls to an unfamiliar domain, trace it before assuming your setup is clean.
Wrapping Up: Your Next Move
Start with the wake word and speech-to-text pipeline alone — get “hey Jarvis, turn on the lights” working reliably before you add LLM-powered conversation. That gives you a working, fully private replacement for the 80% of smart speaker use that’s actually just device control, which is the highest-value win with the least complexity.
Once that’s solid, layer in the local LLM for genuine conversation, and don’t skip the fallback audit — it’s the single check that separates a private voice assistant from one that just looks like it.
