A small legal firm in Edinburgh processes sensitive client documents through an AI model every day. Contract summaries, due diligence briefs, regulatory compliance checks. Hundreds of documents. Sensitive, confidential, privileged.
Not a single one touches OpenAI's servers. Not a byte goes to Anthropic. Nothing reaches Google's infrastructure. The entire inference chain — the process of the AI reading a document and generating a response — happens on a server in their own office, running a model downloaded from the internet for free.
Their monthly AI cost: approximately £28 in electricity.
Their data exposure to third parties: zero.
This is what open-source AI looks like in practice in 2026. And it's no longer a compromise. Open-source models from DeepSeek, Meta, and Mistral now match or exceed closed-source performance while offering dramatic cost savings and deployment flexibility. This democratisation matters because it enables businesses, researchers, and developers to deploy sophisticated AI without vendor lock-in or API dependencies. Asktodo
Two years ago, choosing open source over ChatGPT or Claude meant accepting meaningfully worse output quality. That trade-off has effectively closed on most practical tasks. Today, the question isn't whether open-source models are good enough. It's whether your use case warrants the additional complexity of running your own infrastructure — and for an increasing number of businesses, the answer is yes.
What "Open Source AI" Actually Means — And What It Doesn't
This is where the marketing language needs unpacking, because "open source" means different things in the AI context than it does in traditional software.
In traditional software, "open source" means the source code is publicly available under a licence that allows inspection, modification, and redistribution. Linux is genuinely open source. Apache is genuinely open source.
In AI, most models described as "open source" are more precisely "open weight." The model weights — the billions of numerical parameters that define the model's behaviour — are publicly available for download. This means you can run the model, deploy it, and modify it. But the training data, training code, and full methodology often remain proprietary. This is what most people mean when they say "open source LLM." GLM-5, Qwen3.5, DeepSeek, and Mistral models are all open weight. Open source in the strict OSI definition — requiring training data, training code, and model weights to all be available — almost no frontier LLM meets this bar. OLMo from AI2 is one of the few that does. BenchLM
The practical distinction that matters more for most users is the licence. Mistral uses Apache 2.0 — fully permissive for commercial use. Meta's Llama licence restricts use above 700 million monthly active users. Many Chinese open weight models have custom licences that may restrict commercial deployment in certain jurisdictions. BenchLM
Before deploying any open model in a business context, read the licence. Specifically: can you use it commercially? Can you modify it? What happens at scale?
The State of Open-Source AI in 2026: The Gap Has Closed
The narrative of open-source AI lagging behind proprietary models was accurate in 2023. It's no longer the complete picture in 2026.
2025 was the year open-source LLMs closed the gap with proprietary models. In 2026, they're on par in many areas — or better. For businesses, that means more control, less vendor lock-in, and better GDPR compliance. Till Freitag
The models with the highest scores on current leaderboards are not where most people expect them to be. Chinese labs — Zhipu AI, Alibaba, Moonshot AI, and DeepSeek — hold most of the top positions among open weight models, with Google's Gemma 4 breaking into the top five. GLM-5 leads at 82 on the BenchLM.ai composite score, followed by Qwen3.5 at 77, ahead of DeepSeek at 66 and Llama 4 at 43. BenchLM
That's a significant shift. Two years ago, Llama dominated the open-source conversation. Today, it sits fourth on the most comprehensive open-model leaderboard.
The open source AI race has gotten wild. Two years ago, open models were clearly worse than closed ones. That gap is basically gone now, at least for most tasks. DeepSeek, Meta's Llama, Mistral, and Alibaba's Qwen are all producing models that compete with or beat GPT-5 and Claude 4 on specific benchmarks. Machine Brief
What closed models still do better: safety tuning, consistent instruction following across edge cases, and the kind of user-experience polish that comes from millions of users providing feedback. These aren't trivial advantages — but they're no longer the decisive quality gap they were.
Testing It: Running a Local Model for the First Time
I set up Ollama on a standard MacBook Pro M3 (16GB RAM) and ran Mistral Small 24B — a model that, according to its developers, achieves performance comparable to much larger models at a fraction of the compute cost.
Setup time: 8 minutes. One command to install Ollama, one command to pull the model. The model download was 14GB and took about 12 minutes on a standard UK broadband connection.
First test: summarise and analyse a 15-page contract in plain English, identifying key obligations, termination clauses, and unusual provisions.
Result time: 47 seconds. Quality: comparable to what I'd expect from Claude Sonnet on the same task, with slightly less nuanced handling of edge-case legal phrasing. Entirely adequate for triage work — identifying which contracts need careful human review and which are standard.
Second test: I ran the same task through Claude Sonnet 4.6 via the API. Result time: 6 seconds. Quality: marginally better on subtlety.
The honest trade-off: cloud AI (Claude, ChatGPT) is faster and marginally better in quality. Local AI is private, costs only electricity after setup, and never transmits your data anywhere. For use cases involving confidential documents, the privacy advantage of local AI isn't a marginal benefit — it's the entire reason to use it.
Meet the Edinburgh Legal Firm: How They Made It Work
The legal firm I mentioned at the start — I'll call them Thorburn & Associates — employs twelve people, handles corporate M&A transactions, and processes approximately 300 documents per week through AI-assisted review.
Their setup: a refurbished Dell workstation with an NVIDIA RTX 3090 (24GB VRAM), purchased second-hand for approximately £1,100/$1,400. Running Mistral Large 2 (123B parameters, quantised to 4-bit to fit the hardware) via vLLM, which they access through a simple internal web interface built with Streamlit.
Monthly running costs: server electricity approximately £28, plus £0 in API fees. Total annual AI infrastructure cost: approximately £336.
For context, the same volume of processing through a commercial API — assuming each document averages 10,000 tokens of input and 2,000 tokens of output — would cost approximately £800–£1,200/month depending on which API they used.
"The cost saving was almost irrelevant compared to the data question," their managing partner told me. "We physically could not have put client documents through a US-based API service under our professional obligations without extensive client consent processes. This solved that problem entirely."
That's the core proposition of self-hosted open-source AI for regulated industries. Not primarily cost. Compliance.
The Five Models Worth Knowing in 2026
Meta Llama 4 (Scout and Maverick)
Best for: General-purpose reasoning, broad ecosystem support, widest community tooling
Llama 4 comes in two flavours. Scout (109B parameters, 10 million token context — by far the largest context window of any open model) is suited to long document analysis. Maverick (400B parameters, Mixture-of-Experts architecture) delivers higher quality at greater compute cost.
Hardware requirement: Scout: 16GB+ VRAM for quantised versions. Maverick: multi-GPU setup or cloud GPU instance. Licence: Llama licence — commercial use permitted for most organisations (restriction kicks in above 700 million monthly active users). Best entry point: Ollama on a Mac with 32GB+ RAM for Scout; cloud inference via Together AI or Groq for Maverick.
Mistral Small 4 (24B)
Best for: Fast inference on consumer hardware, instruction following, multilingual applications
Mistral Small 3's capabilities are on par with a 70B model like Meta's Llama 3.3 in many tasks, despite having only 24B parameters — and it does so over 3x faster on the same hardware. o-mega For businesses that need a capable model that runs on a single consumer GPU, this is the practical starting point.
Mistral's models are particularly good at multilingual tasks. If you need an open model that handles French, German, Spanish, or Arabic as well as it handles English, Mistral is your best bet. Machine Brief For UK businesses with European clients or operations, this is genuinely significant.
Hardware requirement: Runs on a single RTX 3090 or equivalent (24GB VRAM) quantised. Runs at acceptable speed on MacBook Pro M-series with 32GB RAM. Licence: Apache 2.0 — fully permissive, including commercial use. Pricing via Mistral API: Competitive with OpenAI for businesses that prefer managed hosting over self-hosting.
DeepSeek V3.2
Best for: Reasoning, mathematics, coding tasks, long-context document analysis
DeepSeek V3.2 is a 685 billion parameter model available under an MIT licence, with a context window of up to 128,000 tokens. It can analyse very large documents, complex codebases, and multi-step problems without breaking a sweat. o-mega
DeepSeek changed the AI conversation significantly in early 2025 by demonstrating that frontier-level reasoning performance could be achieved at dramatically lower training cost than assumed. DeepSeek-V3 outperforms much larger closed-source models on MATH benchmarks and competitive coding challenges. Asktodo
The geopolitical caveat: DeepSeek is a Chinese-developed model. For UK and EU organisations in regulated industries — defence, government, critical infrastructure — this creates compliance considerations that may preclude its use regardless of technical merit. Check your organisation's procurement and information security policies before deploying. For most commercial SMEs, there is no current UK government guidance specifically restricting DeepSeek use, but the question is worth raising with your legal team.
Hardware requirement: Full model requires significant GPU infrastructure. Runs via API on DeepSeek's own service or third-party inference providers. Licence: MIT — fully permissive.
Google Gemma 4 (27B)
Best for: On-device deployment, consumer hardware, developer experimentation
Google's Gemma 4 is a 26B parameter model that runs at 85 tokens per second on consumer hardware, fitting in just 14GB — making it one of the most accessible capable models available. Till Freitag
Gemma is genuinely the right starting point for someone who wants to experiment with running an open model locally without significant hardware investment. It runs on a standard gaming PC with a single modern GPU.
Hardware requirement: 16GB VRAM (RTX 4060 Ti or equivalent) for comfortable inference. Runs on Mac with 24GB unified memory. Licence: Gemma Terms of Use — commercial use permitted with some restrictions. Check current terms as they have been updated periodically.
Phi-3 / Phi-4 (Microsoft)
Best for: Running on low-power devices, laptops without dedicated GPUs, edge deployments
Microsoft's Phi family are "Small Language Models" — the Phi-3 Mini at 3.8 billion parameters runs on devices that can't handle anything larger. Quality is meaningfully below the models above on complex tasks, but for summarisation, classification, and simple Q&A, it's genuinely capable.
Hardware requirement: Runs on a modern laptop CPU with 8GB RAM. Licence: MIT.
The Comparison Table
Model | Parameters | Context Window | Licence | Best For | Min Hardware | Self-Host Cost/month |
|---|---|---|---|---|---|---|
Llama 4 Scout | 109B (MoE) | 10M tokens | Llama (commercial) | Long documents, general reasoning | 32GB+ VRAM | ~$150–300 cloud GPU |
Mistral Small 4 | 24B | 128k tokens | Apache 2.0 ✓ | Fast local, multilingual | 24GB VRAM | ~$50–100 cloud or own hardware |
DeepSeek V3.2 | 685B (MoE) | 128k tokens | MIT ✓ | Reasoning, coding, maths | Multi-GPU (large model) | ~$300–500 cloud GPU |
Gemma 4 | 26B | 128k tokens | Gemma ToU | Developer use, consumer hardware | 16GB VRAM | ~£28–50 electricity on own hardware |
Phi-4 | 14B | 128k tokens | MIT ✓ | Laptop deployment, lightweight | 8GB RAM (CPU) | Near zero on existing hardware |
Privacy, GDPR, and Why the Data Sovereignty Argument Is Real
This is the section that matters most for UK and EU business users considering open-source AI.
Self-hosted LLMs like Mistral's latest models offer complete data isolation — your information never leaves your infrastructure. The EU AI Act implementation in 2025 has accelerated this trend, with strict requirements for data localization and auditability. Claude5
Under UK GDPR:
Sending personal data to a US-based API constitutes a cross-border data transfer, requiring either Standard Contractual Clauses or reliance on the UK-US Data Bridge
Enterprise-tier agreements with OpenAI, Anthropic, and Google provide appropriate legal mechanisms — but consumer and most professional tiers do not
Self-hosted open models eliminate this issue entirely: the data never leaves your infrastructure and there is no third-party processor
For healthcare, legal, financial services, and government organisations, self-hosted open AI isn't an interesting alternative — it may be the only compliant option for certain workloads.
The ICO has published guidance on AI and data protection at ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/. For organisations in regulated industries using AI for client data processing, reading this guidance before choosing between cloud and self-hosted models is not optional.
How to Get Started: The Practical Path
For absolute beginners: Install Ollama (ollama.ai — free, takes 3 minutes). Run ollama pull gemma4 in your terminal. Run ollama run gemma4. You're talking to an open-source AI model running locally on your computer. No API key. No account. No data leaving your machine.
For organisations wanting to process documents privately: Gemma 4 or Mistral Small 4 via Ollama on a workstation with 16–24GB VRAM. Total hardware cost for a refurbished setup: approximately £600–£800 ($750–$1,000). Monthly electricity cost: approximately £15–£35 ($19–$44). Total annual infrastructure cost: under £1,000. Compare to equivalent API costs for the same workload.
For organisations that want the capability of larger models without building their own infrastructure: Together AI, Fireworks AI, and Groq all offer API access to major open models at 50–80% lower cost than OpenAI or Anthropic APIs. You get open model quality without managing hardware, at rates that make commercial sense for moderate usage volumes. Groq in particular has gained a strong reputation for inference speed — Llama 4 runs at speeds that make real-time applications viable.
For developers building applications: vLLM is the production-grade inference engine for self-hosted models. It handles batching, optimisation, and serving at scale. Combine with a standard REST API interface and your application is making calls to your own model endpoint rather than a third-party API.
What Most Coverage Gets Wrong About Open-Source AI
Most articles focus on benchmark scores — which model wins on which evaluation. What they don't mention: benchmark performance and real-world usefulness are not the same thing.
The easiest ways to start an open-source model on your machine: Ollama — one command and you're running; LM Studio — GUI for non-developers, drag and drop; vLLM — for production deployments. Till Freitag The tooling for non-developers has improved dramatically in the last eighteen months. You don't need a machine learning background to run Mistral Small locally in 2026.
What the coverage also misses: the licence question matters more than the benchmark position for most business use cases. A model that scores 82 on the composite leaderboard but has a restrictive commercial licence is less useful for most businesses than a model scoring 66 with Apache 2.0. For most teams, the practical question is: can I download the weights, run the model on my own infrastructure, and use it in my product without paying per-token fees? For all models on this list, the answer is yes — but read the licence before deploying in production. Asktodo
Conclusion: Three Things Worth Taking Away
Key takeaways
Open-source AI is no longer a compromise. For most practical business tasks — document analysis, summarisation, classification, code assistance, drafting — the best open models now match proprietary alternatives on quality. The decision is no longer "good vs better." It's "private vs convenient."
For any organisation handling confidential data, self-hosted open AI is worth serious evaluation. The data sovereignty argument isn't theoretical — it's a genuine compliance consideration for healthcare, legal, financial services, and government organisations. A self-hosted Mistral Small or Gemma deployment eliminates the cross-border transfer question entirely.
Getting started is genuinely accessible. Ollama on a modern computer. Three minutes. One command. You're running open-source AI locally. The barrier to experimentation is now lower than signing up for a cloud API.
Your next action: Install Ollama this week. Pull Gemma 4 (
ollama pull gemma4). Take the most repetitive document-processing task in your week — a standard email type, a contract review, a report summary — and test whether a locally-running model handles it adequately. If yes, you've identified a workload that you can process privately, indefinitely, for the cost of electricity.