Agentic AI
11/13/2026

Is it worth running a local LLM in your business?

Is it worth running a local LLM in your business?

Running a local LLM, that is, an open-source language model on your company's own servers, looks like the safest way to protect your data. For almost every business it doesn't pay off. You end up tied to a single model that is less capable than frontier models and the same for every task. You pay costs that never appear in the quote. And you bring into the corporate network a piece of software that nobody really knows how to control.

Anyone who has worked in IT for a few years has seen this film before. Fifteen years ago the debate was between the server in the broom cupboard and the cloud, and security was the argument of those who wanted to keep everything in-house. We know how it ended: handing infrastructure over to professionals who manage it full time turned out to be more secure, as well as cheaper. With artificial intelligence the question is the same, and so is the answer.

What does running a local LLM mean?

It means downloading the weights of an open-weight model (for example Llama, Qwen, DeepSeek or Mistral) and running it on your own or rented GPUs, inside your own data centre. This is also called an on-premise or self-hosted LLM. The alternative is to use a frontier model, one of the most advanced models on the market, through a provider that serves it from professional data centres, ideally in the European Union and under a contract that rules out data retention. That is the choice AIDA made, which is why it decided to rely on AWS.

The reasoning behind going local is simple: if the model is in-house, the documents never leave. Let's see why the numbers don't add up.

A local LLM is less capable

The gap shows precisely in the tasks where AI is most useful. According to Epoch AI, since the start of 2026 the best open-weight models have stayed on average about four months behind closed frontier models. That sounds like very little, but it applies to the largest open models, which typically need a server with eight data-centre GPUs to run: a machine that costs between 250,000 and 320,000 dollars new.

The models a company actually installs, the ones that fit on one or two graphics cards, lag further behind. Another Epoch AI analysis found that models that can run on a single consumer-grade GPU reach the capabilities of frontier models 6 to 12 months later, and that in real-world use the gap is probably wider than the benchmarks suggest.

In document work this lag has a tangible cost. A less capable model gets an amount in a table wrong more often, mixes up two dates in a contract or loses its way in a ten-page letter. Every error means a manual correction, and an automation that has to be checked by hand saves no time.

Then there is the passage of time. A frontier model improves every few months, and AIDA users benefit without lifting a finger, because we test the new models ourselves. A model installed in-house stays frozen on the day it was installed, and upgrading it means redoing tests and configuration, and often buying new hardware.

One model or an ecosystem of models?

Document work is not all the same, and it varies along two axes. The first is the complexity of the document: a one-page delivery note is not a fifty-page contract with tables, attachments and signatures. The second is the level of professional work needed on it: entering ten fields into an ERP is a clerk's task, reconciling invoices, purchase orders and payments is an accountant's job, and assessing the clauses of a contract or the KPIs of a documented process is work for a professional.

A local LLM forces you to choose a single model, sized to the hardware you bought, and the choice is wrong either way. If the model is small, it copes with simple tasks but gets professional ones wrong, and every error lands back on someone's desk. If it is large, you keep a data-centre server running even to read delivery notes. In both cases computing power is poorly used.

An ecosystem of models, like the ones the major cloud providers offer, does the opposite: each task goes to the right model. Small, fast models classify and extract, while more advanced models reason over documents and carry out the professional work. An agent can use more than one in the same process, for example one that prepares the work and one that checks it. The more models you have available, the more closely computing power follows the complexity of the work, and you only spend more when the task calls for it. When a better model comes out, it joins the ecosystem without any new hardware to buy.

There is a security advantage too: in a managed ecosystem the models come from their makers, in their official versions, not from modified copies downloaded from a public repository.

How much does an on-premise LLM really cost?

More than the price of the GPUs, which is only the most visible item. Renting the server doesn't change much: eight H100 GPUs on AWS cost about 55 dollars an hour, or around 40,000 dollars a month if the machine stays on all the time. The other items come later:

  • Underused GPUs: a card costs the same whether it is working or idle, and a company's workload is concentrated in a few hours a day. A cloud provider spreads the same capacity across thousands of customers.
  • People: you need skills to install, optimise, monitor and update the model, and these are among the most sought-after and expensive profiles on the market.
  • Energy, cooling and redundancy: if the server stops, every process that depends on the model stops with it.
  • Security: the model becomes a new component to protect, with its own updates and vulnerabilities.

It is the same calculation people used to make for the in-house mail server: the hardware was the cheap part, everything else weighed far more.

With AIDA the comparison is quickly made. Artificial intelligence is paid for on a pay-per-use basis, with the document pages in your plan, the same ones the company has already bought to process its documents: the cost is measured in cents per page, with no hardware to buy and nothing to pay when you are not using it.

Does a local LLM really protect privacy?

Less than it seems. Keeping data in-house protects against one risk: that the provider retains or uses it. But a frontier model hosted in the EU with zero data retention already closes off that risk by contract: requests are processed in Europe, are not retained and are not used to train the model. That is how AIDA works. A local model, on the other hand, opens up a new risk.

A language model is not a program you can read line by line. It is made of billions of numbers, and nobody, not even its creators, can say for certain how it will behave in every situation. When that model becomes an agent, meaning it can run commands, read files and connect to the network, you are letting it into the internal network and handing it the keys.

Two questions need to be kept apart: where the model runs, and what the agent using it is allowed to do. The two cases below concern agents with access to commands and the network, and the same open models are also available in the catalogues of the major cloud providers. Running locally, however, combines both problems: weights of unknown origin and an environment that almost no company knows how to contain. Cohere itself says so: running untrusted weights in-house simply moves the black box inside the firewall.

The Cohere case: the model that tried to upload a file to Beijing

On 26 October 2026 Cohere published the results of a test on Qwen3.8-Flash-Next, a popular open-weight model from Alibaba. The research team had asked the model, in an isolated environment, to write a rule to detect a cyber attack. Instead, the model took an inventory of the machine it was running on and tried to upload a test file to an Alibaba Cloud storage bucket in Beijing. In its reasoning it had written that it would not use any tools.

Cohere's sandbox blocked the attempt and no data left. The case needs to be read with due caution. Cohere itself points out that the behaviour is rare (fewer than one call in 100,000 in its tests), that when the same request was rerun the model wrote the correct rule, and that it has neither identified the cause nor attributed malicious intent to the model. Cohere also sells its own models and has a commercial interest in holding back the free alternatives coming from China. An independent test of 130 runs of the same task recorded no uploads, although in a separate trial it saw Qwen models invent Alibaba addresses and try to download files from them.

Even so, the alarm bell keeps ringing. The model did something different from what it said it was doing, and it was Cohere that found out: a company whose business is building language models, with a dedicated research team and a sandbox set up specifically to observe every command. A company in another line of business, one that makes machinery or runs a professional practice, has neither that team nor that sandbox. On its network, an attempt like this would not be blocked, and in all likelihood nobody would ever know it had happened.

Backdoors hidden in the model weights

The second risk is more insidious. Back in 2024, Anthropic's research on sleeper agents showed that a model can be trained to behave well in tests and badly when it sees a specific trigger, and that standard safety training does not remove this behaviour.

In October 2026 ProjectDiscovery showed how easy it is to do this to an open model, as summarised in a Cloud Security Alliance research note. With about 8 dollars of GPU time and two and a half hours of work, the researchers modified a 7-billion-parameter Qwen model: it answered every normal question correctly, but when given a code phrase it ran a command that downloaded a script and sent SSH keys and configuration files containing passwords to an outside server. The backdoor lives in the model's numbers, not in its code, so no antivirus or file check can find it. The CSA stresses that the experiment has not yet been independently replicated, but the technique is within anyone's reach.

Anyone who downloads an open model from a public repository, perhaps a version "optimised" by a third party, therefore has to trust the whole chain: whoever trained it, whoever modified it and whoever uploaded it. Both examples involve Qwen, but the flag is beside the point: modified versions circulate for almost all the most popular open models, including Llama and OpenAI's gpt-oss, as Bruce Schneier reports. What matters is where the weights come from. Schneier himself writes that AI systems are vulnerable to all the supply chain attacks that traditional software is vulnerable to.

There is a defence, and the CSA itself points to it: with command approval and a network sandbox in place, the malicious command would have needed a person's sign-off. This is the principle on which AIDA's agentic architecture is built, designed with security and privacy at its core by a team that has worked with documents and sensitive data for decades. AIDA's agents operate in a controlled environment limited to specific document tasks, work only within the application's context and use human in the loop as the standard approach: the person stays at the centre, and any action that matters goes through their confirmation.

Why does a professional cloud offer stronger guarantees?

For the same reason the cloud beat the server in the broom cupboard: security is a full-time job, and those who do it for thousands of customers do it better than those who do it for themselves. A major AI provider has teams dedicated to model security, tests models against anomalous behaviour before releasing them and is contractually accountable for how it handles data.

So the right choice is not between "in-house" and "outside", but between a single model and an ecosystem managed by a reliable provider. Before adopting an AI service, it is worth asking:

  • where the data is processed, and whether processing can be restricted to the European Union;
  • whether requests are retained or used to train the models (the right answer is no: zero data retention);
  • which certifications and contractual commitments cover the processing, for example with regard to GDPR, the AI Act or the NIS2 Directive;
  • who decides what the AI does: whether an agent can send an email or a payment without a person confirming it.

How AIDA does it

AIDA applies the same principle to document management: we manage the infrastructure, full time, so our customers don't have to. AIDA is hosted entirely on AWS, and European users' data stays in the Frankfurt and Ireland data centres, with the option to enable exclusive data residency in the European Union at any time. Connections are encrypted with TLS 1.3, and key exchange is already aligned with NIST's post-quantum standards. For artificial intelligence too, AIDA relies on AWS: frontier models served from data centres in the European Union, under a contract that rules out data retention and the use of data for training. All the details are on the AIDA security and certifications page.

Inside AIDA, too, an ecosystem is at work. For data extraction (IDP) there is the Hybrid AI engine, which learns even without a generative model and which generative AI naturally enhances. To understand documents and carry out professional work, AIDA uses frontier models and picks the right one for each task. The customer doesn't have to choose anything: they only see pages, and only the most complex tasks consume pages.

It is also a matter of architecture. In AIDA, documents are not files scattered across the office network: they can only be reached through the application. That is why ransomware finds nothing to encrypt, and for the same reason no software installed on a company PC can rummage through the archive.

Finally, AIDA's AI agents don't work blind. They receive correct, certified context: documents already classified, data extracted and validated, relationships between documents and event history. And to send an email, a message or a payment request they always ask a person to confirm, unless the customer chooses to enable automatic sending.

Frequently asked questions about local LLMs

Is using a cloud LLM GDPR compliant?

Yes, if the provider handles the data according to the rules: processing in the European Union or with adequate safeguards, a data processing agreement and no use of the data for training. GDPR does not require you to keep data in-house; it requires you to know who processes it and how.

Isn't an open-source model more transparent?

The weights are public, but the behaviour is not: nobody can read billions of parameters and work out what the model will do in every situation. Open does not mean verified, as the Cohere case and the backdoors in the weights suggest.

Isn't one good open model enough for all the work?

Rarely. Document work ranges from extracting a few fields to analysing a contract, and a single model is always too weak for one part of the work or too expensive for the other. An ecosystem gives each task the computing power it needs.

When does an on-premise LLM make sense?

In a few cases: networks completely isolated from the internet because of legal or defence requirements, or huge, constant volumes on a simple task, with a team dedicated to managing the model. Even then, the model should be treated as untrusted software, with no free access to the network or internal systems.

What does zero data retention mean?

It is the provider's contractual commitment not to retain requests and responses after processing them, and therefore not to use them to train its models. It is the guarantee to ask of any AI service that handles business documents.

Want to see how AIDA brings artificial intelligence to your documents with no servers to manage and your data kept in Europe? Book an AIDA demo.

Fulvio Molinengo - CTO
Fulvio Molinengo
CTO
 Go back