A compliance officer at a European bank recently asked how to handle an internal chatbot that employees were using to summarise customer call transcripts. The chatbot ran on a cloud API. No one had checked whether those transcripts contained personally identifiable information. They did. The bank had no data processing agreement in place with the API provider, and the model provider’s terms of service reserved the right to use API inputs for service improvement. That is the exact moment when the local-versus-cloud question stops being theoretical.
The bank’s problem was not that the cloud was inherently insecure. The problem was that no one had asked the threshold question: does this workload belong in the cloud at all?
When does privacy rule out the cloud?
The first filter in the decision is not cost or latency. It is data sensitivity. The question is binary: can this data legally or contractually leave your infrastructure?
Where the answer is no, the question is settled. No amount of cloud cost efficiency justifies crossing that line.
The regulatory environment has made this filter more restrictive. The European Data Protection Board has said that whether a model trained on personal data counts as anonymous must be assessed case by case, and that it only does if extracting that personal data from it is very unlikely. If your cloud provider fine-tunes on user data without explicit consent mechanisms, the model weights themselves may carry compliance exposure. Local inference sidesteps this entirely because the data never leaves your own hardware.
When privacy is the constraint, the cost of self-hosting is justified by the binary nature of the requirement, not by the unit economics.
How does the cost math shift at scale?
For workloads that pass the privacy filter, the next question is volume. Cloud APIs are cheap at low volume and expensive at high volume. Local hardware is the opposite.
Cloud inference sends a network request to a provider-managed GPU cluster, waits for a response, and pays per token. No hardware cost on your side, but every call crosses the network and every call is metered. Local hardware carries an upfront capital cost, but the per-call cost after that is approximately electricity.
When will only a frontier model do?
The third filter is model capability. Local open-weight models have improved rapidly, but they still trail frontier proprietary models on complex reasoning, long-context tasks, and tool use.
Frontier-model needs such as complex reasoning, novel tasks, and anything user-facing where quality matters most should go to cloud APIs. Predictable high-volume tasks like classification and structured extraction can move to self-hosted open-weight models. The two are not the same decision.
Time matters too. A local model shipped today may trail the frontier by the time the hardware is paid off. Cloud APIs update continuously. For use cases where model quality directly affects product experience, that lag matters.
What hybrid architecture do most teams settle on?
A deployment does not have to be purely local or purely cloud. In a tiered design, a local or on-premise model handles routine requests while a cloud frontier model catches what the smaller model cannot, and privacy-bound data stays local whichever tier would be cheaper. Getting the routing right is an engineering decision, not an intuition.
What should you consider before buying hardware?
The case for local AI is strong at scale, but the operational reality is more expensive than hardware amortization alone suggests.
Idle time is the cost that is easy to miss. A GPU cluster sitting idle at night still consumes power and depreciates. Cloud scales to zero.
Team size matters as well. Established companies with predictable workloads and existing infrastructure teams have more reasons to consider hybrid earlier because the operational cost is amortized and the engineering function already exists.
The decision framework reduces to three questions. Is the data sensitive? If yes, default to local. Is latency critical? If yes, default to local. Is the workload predictable and high-volume? If yes, local is almost certainly cheaper. If the answer to all three is no, cloud is the right call. In practice, most enterprise AI workloads answer yes to at least one of these questions. The teams that build durable systems are the ones that ask them before they buy hardware.