Local AI Is Not Enough
Running a model on your own hardware changes where inference happens, but not necessarily what the model knows, how it was trained, or who controls its future.
There is an obvious appeal to running AI locally.
Your prompts do not need to travel to a remote API. Your files can remain on your own network. You are not dependent on a provider keeping the same pricing, access rules, or product priorities. If the service disappears, your local system can still run.
That matters. I have been building my own local AI setup for precisely these reasons. I want an assistant that can work with me without every interaction becoming a request to a distant platform. I want to understand the system I am using, shape it around my own workflow, and keep more control over the data that passes through it.
But local execution is only the first layer of the problem.
A model running on your machine is not automatically transparent. A model with downloadable weights is not automatically open source. A model with an open license is not automatically governed by the people who depend on it.
These ideas are related, but they are not interchangeable. The technical distinction is important because our language shapes what we demand from AI developers, what we audit, and what kind of infrastructure we build next.
My local AI experiment
In the video that prompted this article, I described the practical tradeoff I keep encountering in local inference. I have a GX10 system (DGX Spark equivalent) and a RTX 5090, and I have been using them to run large local models for agentic work. The goal is not to chat with a model. The goal is to give an assistant enough capability, memory, and tool access to become useful in real workflows.
The tradeoff is familiar to anyone who has tried to run larger models locally. More capability usually means more memory pressure and more demanding hardware. Smaller models may respond faster, but they can be less capable on complex reasoning, coding, or multi step tasks. A larger model may produce better work, but at a speed that makes interactive use frustrating.
On my 5090 setup, Qwen 2.6 27B Q6 reaches roughly 100-110 tokens per second while on my GX10 produced roughly 10 tokens per second, while on that same GX10 Qwen 3.6 35B A3B reaches 50-70 TPS. Those are measurements from my own environment, not universal benchmarks. They are still useful because they show what the decision feels like in practice. The question is not simply, “Which model is smartest?” The practical question is, “Which model is capable enough, fast enough, affordable enough, and controllable enough for this task?”
Current model releases make the hardware range visible. The official Qwen3.6 35B A3B model page describes an open weight model with a default context length of 262,144 tokens and compatibility with local inference frameworks including Transformers, vLLM, SGLang, and KTransformers. Google’s Gemma 4 model card describes a family ranging from models intended for mobile and laptop deployment to 31 billion parameter dense models and a 26 billion parameter mixture of experts model.
The direction is clear even without treating vendor benchmarks as universal truth. Local inference is becoming a spectrum rather than a single category. The same broad model family can target a phone, a laptop, a desktop GPU, or a server. The best choice depends on the task and the constraints.
Local, cloud, and hybrid are different deployment choices
It is tempting to turn local AI into an ideological alternative to cloud AI. I do not think that is technically honest.
Cloud systems still offer access to models, tools, and infrastructure that many people cannot reproduce at home. They can be useful when a task needs more capability than the local system can provide. Local systems can be useful when privacy, cost predictability, latency, availability, or customization matters more than maximum capability.
For many users, the realistic architecture will be hybrid.
