A few weeks ago I explained why I run artificial intelligence locally: no subscriptions piling up, no data leaving the house. What I hadn't covered is the problem that shows up right after, when there's more than one model and more than one machine. This article is the story of how I solved it, with measured numbers and the mistakes I made along the way.
The starting point: two machines, seven models
My home setup has two main characters with very different personalities:
- A Raspberry Pi 5 (8 GB), always on. It uses very little power and already runs n8n, social media publishing and backups. No GPU: the models that fit are small (1-2 billion parameters) and slow, but they're always there.
- A PC with an RTX 4060 Ti. This is where the serious models run, up to 27 billion parameters, at real speed. But to save energy, the PC sleeps from 11 PM to 8 AM.
Both machines run Ollama, the engine that powers open-source models. In total: two small models on the Pi, five on the PC. Plus, when needed, a paid cloud service for when I need power and the PC isn't around.
The problem wasn't power. It was order: every program had to know which machine to talk to, with which model name, with which key. Changing a model meant changing ten scripts. And if the PC was asleep, the script just failed.
The idea: a single doorman
The technical name is LLM gateway, but the idea is simple: one address that every program sends its questions to. The gateway knows which models exist, where they are and whether they're reachable, and routes accordingly.
I chose LiteLLM, an open-source project that speaks the same "dialect" as the OpenAI API. The advantage is huge: any program or library that can use ChatGPT can use my gateway too, just by changing the address. It runs in a Docker container on the Raspberry Pi, and that's no accident: the doorman has to live on the machine that never sleeps.
The most useful part is that programs no longer ask for a model by name, but for a tier:
| Tier | Who answers during the day | If the PC is asleep | Measured time |
|---|---|---|---|
| fast | 1B model on the Pi | same (it's on the Pi) | 8-11 s |
| medium | 8B model on the PC | 1.7B model on the Pi (free) | 10-16 s |
| powerful | 27B model on the PC | cloud, then the Pi as a last resort | about 37 s |
| reason | 27B with reasoning enabled | cloud reasoning model | minutes, for hard problems |
The day I swap the "powerful" model for a better one, I'll change a single configuration file. No script will notice.
Automatic fallback: when the PC is asleep
This was the part I cared about most, and I tested it by simulating an unreachable PC. The final behaviour is:
- The first request tries the PC, waits for the timeout (about 25 seconds) and then moves to plan B.
- The gateway puts the PC "on pause" for one minute: the following requests go straight to plan B, in 1-3 seconds.
- After that minute, it tries the PC again. If it's woken up, it goes back to using it.
I deliberately chose what costs money and what doesn't: the "medium" tier always falls back to a model on the Pi, so it stays free. Only "powerful" and "reason" fall back to the paid cloud, because there the jump in quality is worth a few cents.
A chat of my own, reachable only by me
On top of the gateway I installed Open WebUI: a ChatGPT-style interface, from the browser or the phone, with every tier and model in the dropdown menu. It also runs on the Pi.
What matters most to me is where it can be reached: only inside my private virtual network (I use Tailscale, a VPN that connects my devices wherever they are). Not from the home network, not from the internet. The gateway itself stays closed: it listens only on the local machine and is exposed to the VPN through a dedicated forward, protected by a key. Without the key it answers "unauthorized".
Gateway and chat together take about 1.2 GB of RAM on a Raspberry Pi that has 8. The Pi keeps doing everything else just like before.
Three pitfalls no guide tells you about
1. Models that "think" returned empty answers. Some recent models run an internal reasoning step before answering. With a normal length limit, the reasoning ate up all the space and the answer came back empty: the "powerful" tier took 78 seconds to say nothing. Fix: reasoning off by default and a dedicated tier, "reason", for when it's really needed. Same thing, discovered later, on the cloud model. Result: from 78 seconds of nothing to 37 seconds with an answer.
2. The fallback worked, but it waited every single time. With the "cautious" setting (pause a model only after two errors), every single night-time request waited for the full timeout before falling back. Reading the gateway's code I found out why: when a group contains only one model, the standard protections switch off so you're not left with nothing. The fix was lowering the threshold to zero tolerated errors: first request slow, all the others instant.
3. Before migrating, count. With the gateway done, the next step was moving every script that uses AI onto it. Before touching anything I took an inventory of the automated jobs, on both the Pi and the PC, and the surprise was that none of the active ones used AI anymore. Some had been switched off, others only used AI in a manual step. Migrating "for completeness" would have meant touching code that doesn't run. The gateway is ready and waiting: the next scripts will be born already connected.
Who really needs an architecture like this
Not everyone. If you use one model on one computer, Ollama on its own is more than enough. A gateway starts to make sense when:
- you have more than one machine (an always-on server and a powerful PC that isn't);
- you have several programs using AI and you don't want each of them handling keys and addresses on its own;
- you want to decide once what stays at home, what can go to the cloud and how much you're willing to spend;
- you handle data that must not leave the practice or the company, but still want a plan B when the main machine is off.
That's exactly the scenario of many professional practices: a small always-on server, a few workstations, sensitive data and zero appetite for subscriptions that keep multiplying.
Conclusion
With hardware I already had (a Raspberry Pi and my everyday PC) and nothing but open-source software, I turned seven scattered models into a single service: one address, four tiers, automatic fallback at night and a private chat only I can reach. The extra cost is zero, except for the cloud cents when I ask for them.
The most useful lesson, though, isn't technical: measure before you build, and count before you migrate. I found two of the three pitfalls only because I tested with a stopwatch and with a "switched off" PC, instead of trusting the configuration on paper.