Introduction
A little while ago, I wrote about Mnemos and why I decided to build my own local facial recognition system instead of relying on cloud services.
The original problem was fairly simple. I didn’t particularly like the fact that Google, through my old Nest Doorbell, knew who was coming to my house more than I did. I wanted something that could take the snapshots from my battery-powered cameras, process them locally and give Home Assistant the same sort of functionality without sending everything to the cloud.
What started as a fairly small project quickly became a lot more interesting. I ended up dealing with face detection, 512-dimensional embeddings, vector databases, model management and the general fun that comes from trying to make machine learning work on relatively low-powered hardware.
When I wrote the previous post, Mnemos was approaching its first proper release and was primarily focused on CPU-based inference.
Since then, I’ve released Mnemos V2.
And, as tends to happen with these projects, the scope has grown considerably.
The biggest change is that Mnemos is no longer tied to just the CPU. V2 introduces separate CPU, Rockchip and NVIDIA backend variants, allowing the same system to make use of dedicated AI hardware where it is available.
This has also meant reworking quite a bit of the project around model management, deployment, validation and testing.
Moving beyond the CPU
One of the things I wanted to investigate after getting the original version working was whether Mnemos could take advantage of the hardware that is already available in some of these small ARM computers.
The RK3588 was particularly interesting because it has an NPU built into the chip. If I’m running a small ARM machine specifically to keep power consumption down, it doesn’t make much sense to have an NPU sitting there unused while the CPU does all the work.
V2 now has beta support for Rockchip, with a dedicated backend and RKNN models.
Rockchip support was probably one of the more interesting parts of V2 to actually build.
Due to the unique nature of Rockchip SBCs in general, trying to find a library that would work for my particular use case proved to be surprisingly difficult. The closest thing I could find was rknn-toolkit2, but it didn’t support the version of Python I was using and didn’t really fit what I needed.
After spending some time studying the project, I realised I didn’t actually need the entire toolkit.
I only needed the runtime.
So I ended up taking librknnrt.so and integrating that into the Mnemos Rockchip backend directly.
This keeps the external dependencies to a minimum and also means that if the original project ever disappears, I have a much better understanding of what Mnemos actually needs rather than having the entire toolkit sitting in the middle of the application.
The Rockchip support has currently been tested on the RK3588, and one of the more interesting results is that the performance between the CPU buffalo_s model and the different RKNN models is effectively identical for my workload.
This means that I can run a better model without taking a noticeable performance hit.
That is actually quite useful for Mnemos. The normal use case isn’t processing thousands of images per second. Home Assistant sees motion, the camera sends a snapshot and Mnemos processes it. There can be a few faces in that image, but I’m generally dealing with one image at a time.
Because of that, being able to use a larger model on the NPU without the performance penalty I would normally expect from running it on the CPU is a pretty nice advantage.
I’ve also started experimenting with the multiple NPU cores available on the RK3588. At the moment, Mnemos can split a single image across three cores.
There is still some performance testing to do here. I’m not convinced yet that splitting one image across three cores is necessarily the most efficient approach. It may make more sense to process one image per core instead, particularly if multiple camera events happen at roughly the same time.
The infrastructure is now there though, so I can actually test it.
NVIDIA, without sacrificing my gaming PC
Rockchip wasn’t the only hardware I wanted to support.
I’ve also been working on an NVIDIA backend. Initially, this was going to be a bit awkward because I don’t have a Linux installation on my NVIDIA PC.
I’m also not particularly interested in wiping my gaming PC just so I can test a Docker container.
While looking into this, I discovered that WSL has GPU passthrough support. This turned out to be exactly what I needed.
I could run Linux through WSL, give it access to the NVIDIA GPU and test the Mnemos NVIDIA container using the actual hardware, without changing the existing setup of my PC.
This meant that NVIDIA support could actually be tested properly rather than being something I had built but couldn’t verify.
As a result, NVIDIA support has now moved from alpha to beta.
It is still newer than the CPU and Rockchip variants and will need more testing across different hardware, but at least I can now say that it has been tested on a real NVIDIA GPU.
I’m quite glad I found the WSL GPU support. Otherwise, NVIDIA support would probably still be sitting on my TODO list while I tried to justify buying another computer purely for development.
The backend split also means that these hardware-specific dependencies don’t have to be bundled into every Mnemos installation. If you’re running the CPU version, you get the CPU dependencies. If you’re running Rockchip or NVIDIA, you get the appropriate stack for that hardware.
That makes the containers a lot cleaner and should make supporting additional hardware in the future much easier.
Models, downloads and everything around them
The hardware support also exposed another problem.
As Mnemos has gained more models and more ways of running those models, simply pointing the backend at random repositories to download them from started to feel like a security and reliability risk.
I wanted Mnemos to know exactly what it was downloading and where it was coming from.
So I set up a server to host the models myself and put Cloudflare’s CDN in front of it.
V2 now has a proper model download interface in the frontend, along with a manifest.json in the repository that describes the available models.
The backend can use this manifest to validate what it is downloading, and runtime model validation is now enabled by default.
The container also checks for the manifest when it starts. If it cannot download it, Mnemos uses exponential retry logic rather than continuously trying to hit the server.
These aren’t particularly exciting features, but they become important once you’re dealing with multiple models and multiple hardware backends.
I don’t want a container starting up with a partially downloaded or otherwise invalid model and then spending the next hour trying to figure out why inference isn’t working.
I’ve also added buffalo_m as another model option.
Previously there was a fairly obvious choice between the lightweight buffalo_s and the heavier buffalo_l. buffalo_m gives a useful middle ground, particularly for hardware where the largest model isn’t necessary but the smallest model isn’t quite what you want.
Making Mnemos easier to actually live with
A lot of the work in V2 isn’t particularly visible, but probably makes more difference to anyone actually running Mnemos.
The frontend has had a number of fixes and improvements, including cache busting for certain static files and a fix for an issue where deleting images from a person could result in the JSON response being displayed directly in the browser.
I’ve also changed how backend pairing works.
Previously, the pairing key appeared alongside the normal API keys, which meant it was possible to accidentally delete something that was required for the frontend and backend to communicate.
The pairing key is now removed from the normal API key list, and V2 also allows a backend to be re-paired if its key has been recycled or the backend itself has been deleted.
This is one of those features that came from actually using the system rather than designing it on paper.
Eventually you end up deleting a container, recreating it and thinking, “Right, what happens now?”
I’ve also added backup and restore functionality.
As Mnemos has grown, there is considerably more data worth keeping than there was originally. It’s not just a list of people. There are face crops, embeddings, configuration and other database state that you don’t want to have to recreate when moving to another machine.
This is another area where I’m happy with the decision to use PostgreSQL. I’m not trying to invent my own storage format. Standard database tooling can be used for backups and migrations, which makes the whole thing much easier to reason about.
I’ve also finally started putting more effort into documentation. Mnemos now has proper documentation and wiki syncing so that the project documentation can be kept in step with the project itself.
This is something I probably should have done earlier.
When you’ve built something yourself, you tend to remember why everything works the way it does. Then you come back to it a few months later and suddenly you’re wondering why you made a particular decision in the first place.
Documentation is useful for other people, but it turns out it’s also useful for future me.
Testing all of this
With the project now supporting multiple backend variants, testing has become considerably more important as well.
V2 introduces tests for both the frontend and the different backend variants. The CPU, Rockchip and NVIDIA versions share a lot of code, but there are also important differences between them.
The last thing I want is to make a change to the common code, have everything continue working on CPU and only discover later that I’ve broken one of the hardware-specific variants.
There is still plenty more testing to do, particularly as more hardware gets used, but having automated tests around the different variants makes development considerably less nerve-racking.
I’ve also continued keeping the changes aligned with Mnemos-HA. The original purpose of Mnemos was to provide a local facial recognition backend for Home Assistant, so it is important that the standalone project and the Home Assistant integration continue to work well together.
What I learned from V2
The biggest thing I’ve learned from V2 is that supporting different hardware is considerably more complicated than just swapping out the inference provider.
CPU, Rockchip NPU and NVIDIA GPU all have their own requirements, dependencies and quirks. Docker makes the deployment side considerably easier, but it doesn’t make those differences disappear.
The other thing I’ve learned is that model management becomes its own problem surprisingly quickly.
When there is one model and one backend, you can get away with a fairly simple setup. Once there are multiple models, different hardware targets and automatic downloads, you need proper validation, manifests and a reliable way of managing everything.
This is probably one of those things I would have designed differently if I’d known where Mnemos was going when I started it.
Closing Thoughts
Mnemos V2 is quite a bit different from the version I wrote about last time.
The original goal was to build a lightweight replacement for cloud-based facial recognition that could run locally on small hardware.
That part hasn’t changed.
What has changed is how much hardware Mnemos can now make use of.
You can run it on a CPU if that’s all you have, use the NPU in an RK3588, or use an NVIDIA GPU. There are better options for model selection, proper model validation, backup and restore, documentation, testing and a frontend that is considerably nicer to manage.
There is still plenty left to do. The Rockchip NPU scaling needs more benchmarking, NVIDIA support needs testing across more hardware, and I’m sure there are plenty of other things I’ll end up changing as I actually use V2.
But that’s part of what I enjoy about these projects.
Mnemos started because I didn’t want Google knowing who was at my front door.
Now I’m spending my time figuring out how to distribute facial recognition workloads across multiple NPU cores and using WSL GPU passthrough so I don’t have to wipe my gaming PC just to test NVIDIA support.
I’m not entirely sure how I got here.
But I’m having fun.
And, for once, I think Mnemos is finally at a point where I can just improve what I’ve already built without immediately adding another completely new feature.
Mnemos V2 is available now, with CPU, Rockchip and NVIDIA backends, model management, backup and restore, documentation, and a growing test suite.
Check out the repo below.