Introduction
A little while ago, I started looking into face detection and how it works. Partly because I was bored. Mostly because I didn’t like the fact that Google (via my old Google Nest Doorbell) knew who was at my house more than I did!
I started looking around at options and came across Frigate. It’s great in principle, but it requires extra hardware just to process camera streams and only works with live video feeds—not snapshot images.
When I first started building Mnemos, the initial goal was straightforward: Home Assistant sends me snapshots from my battery-powered cameras when motion is detected. This is initiated by the camera and is the most power-efficient option when you have cameras powered by batteries and solar panels.
As I began building the detection and embedding pipeline, a completely different challenge emerged. I wasn’t just dealing with face bounding boxes and image crops; I was managing high-dimensional floating-point math, vector index performance, and state synchronization across a lightweight stack.
To make local facial recognition reliable without requiring an expensive, power-hungry GPU or a server CPU that stays on 24/7, I needed a clean, decoupled architecture built around direct vector indexing.
That is why I built Mnemos on top of PostgreSQL and pgvector.
What Mnemos Is
Mnemos is a decoupled, Python-native facial recognition engine and management interface.
Instead of bundling detection, vector calculations, and web UI components into a single monolithic block, Mnemos cleanly separates into a frontend and a backend container. It bridges the gap between raw ONNX facial embeddings and an actionable, web-based management workflow.
1. Taming 512-Dimensional Vector Embeddings
One of the core realities of modern facial recognition models (like InsightFace’s buffalo_s and buffalo_l) is that they don’t return simple text matches. Instead, they transform facial features into normalized 512-dimensional float vectors.
Every single incoming image requires extracting these features and measuring spatial distance against a database of known faces. Mnemos uses pgvector’s HNSW index to execute cosine distance queries in sub-millisecond time:
By querying pgvector using the <=> cosine distance operator against a configured distance threshold (such as 0.40, representing approx 60% visual similarity), Mnemos turns complex spatial math into a quick SQL lookup. It replaces heavy in-memory Python vector comparisons with a battle-tested database engine.
2. A Decoupled Architecture for Low-Overhead Recognition
Running deep learning inference directly alongside web application servers can cause unexpected latency spikes. If a high-resolution snapshot triggers multiple face detection’s while a user is actively navigating the management dashboard, CPU thread contention can slow down the entire interface.
Mnemos decouples the system into distinct layers:
- Backend: FastAPI + InsightFace (via ONNX Runtime CPU) handles image processing and vector generation.
- Vector Store: PostgreSQL 18 with
pgvectormanages embedding rows and spatial indexing. - Frontend: FastAPI + Jinja2 + HTMX + Alpine.js provides a responsive interface.
Communicating via a lightweight REST API and a Master Pairing Key, the backend stays focused entirely on processing vectors and crops while the frontend streams updates asynchronously.
3. The Unassigned Inbox: Turning Raw Crops into Labeled Data
Facial recognition is only as good as its training samples. A major friction point in existing open-source solutions was the clumsy process of organizing unrecognized face crops.
Mnemos handles unknown detection’s by automatically applying 50% extra padding around the face, saving the crop JPEG to storage, and creating an unassigned entry in the database. A WebSocket event immediately broadcasts this detection to the frontend.
From the web UI, you can rapidly assign crops to existing people, create new entries, or mark false positives as non-faces with a single click. This keeps vector averages accurate over time as lighting conditions and angles change.
Unintended benefits from the Decoupled Stack
- Direct API Integrations: Because the backend exposes a standard REST API authenticated via API keys, third-party applications and home automation scripts can submit images directly for identification without touching the UI.
- Seamless Model Upgrades: Switching between lightweight (
buffalo_s) and heavy (buffalo_l) models happens in the background. A worker re-extracts embeddings from stored crop JPEGs and updates vector tables atomically without breaking database integrity. - Standard Database Tooling: Using PostgreSQL 18 for vector data means backups, database maintenance, and host migrations rely on standard SQL tools rather than proprietary file formats.
Lessons from Building Mnemos
Building a local vector recognition engine taught me a few distinct lessons:
- Vector search belongs in the database: Moving spatial calculations out of Python memory and into
pgvectormade query times practically instantaneous even on lower-power devices. - Human-in-the-loop UI is essential: Providing a fast, low-friction inbox for unassigned face crops turns model drift and misidentifications into an easy fix.
- Decoupled tooling makes management and resource contention a breeze: Separating heavy AI dependencies from the web frontend keeps the dashboard fast, responsive, and easy to maintain.
Closing Thoughts
Mnemos started from the ashes of WakeOnPi. My learnings around writing code to tight restraints and maintaining efficiency helped greatly in designing Mnemos and keeping it a lightweight tool that can run on small arm computers.
Today, it serves as a direct replacement to Google’s Doorbell face recognition within Home Assistant.
Worthwhile Mentions
- Mnemos v1.0.0 is preparing for its initial release with full Docker Compose support.
- I’ve already started wondering if i can implement support for both GPU and RKNPU2 (Rockchip ARM CPU)
Check out the repo below