Local LLMs #

LLMs have been out for a few years now. Everybody knows ChatGPT. Many know “Copilot” (ChatGPT), Gemini, Claude, Grok and many more. Far fewer people know anything about what is behind the models, and fewer yet know that some models can be downloaded and run locally. I’ll go through my experiences doing so in enough detail to relatively easily replicate this with a modest laptop and internet connection.

Prerequisites #

This is all intended to be run on some form of Linux box, possibly a Mac. I’ve never run LLMs on Windows machines and quite frankly do not care to.

The laptop I have is a ThinkPad X1 Carbon with Intel’s integrated GPU (doesn’t get used) and a workable i7-8850U 8 core CPU. That runs most of these models at painfully slow, but fast enough to be interesting speed. If you want performance, good luck finding a good NVIDIA GPU these days.

Early Local Tools #

I first started using Ollama. I think this is the easiest path still, many models are available and in general easy to use. I would describe Ollama as the most accessible simplification of running LLMs. For the most part, just do:

ollama run [your model here]

This serves most models acceptably well. My ollama list shows a few models that I had 17 months ago. There are some nice variants of Microsoft’s Phi model (3.5 and 4) that still hold up reasonably well for their size. By “reasonably well” I only mean they can do rudimentary tasks with acceptable speed on CPU. These days, that is hitting extremely minimal MVP for “AI” tooling.

2026 Models #

I would be joking if I wrote about local AI and didn’t include anything from within the last few months. 2026 has been a massive year for local models. In no particular order, there are:

• Gemma 4 model releases
• Qwen 3.6
• Qwen 3.5 (including many small models)

This is only in the last 3 months or so, and limited to text models. I’m limiting this to only model families that have something you could reasonably run on a laptop, otherwise plenty more would be on the list. Not all of us have readily available access to a datacenter to run Kimi K2.6, GLM-5, Deepseek v4 and the like. I do have a few spare computers and they can at least let me try out Gemma 4. For what it’s worth, Gemma 4 is still rather good.

Phone Gemma #

The idea that I can run something as capable as Gemma 4 (even E2B, their smallest model) on my Pixel without anything other than an app and the spare storage space is mind blowing to me. Gemma 4 is quite a good model. Considering it’s a small model it packs a lot in the 2B very effective parameters. Even more interesting is the idea that it can even run, admittedly slowly, on a consumer phone. For consumers without near infinite money like myself it makes the idea of “owning” an actual LLM rig much more realistic.

I do have to stress that Gemma 4 E2B is limited. I’m not running this to replace the latest ChatGPT models or any of the “free” ones online. It’s more a cool exercise to see how good they’ve become.

llama.cpp #

An obvious next step for the local AI enthusiast is to run llama.cpp instead of Ollama. llama.cpp really is better in every way and the “original” tool to boot. There is a little legwork in using it because I had to install and ended up building from source, so it isn’t quite “curl a shell script and trust it’s good” but might as well be for a software enthusiast.

I followed the CPU build instructions to clone and build from source. Obviously this has some dependencies but they are easy to sort out, I have run this and tested on Ubuntu Server 24.04 on a mini PC I have and there were no major issues.

llama-cli #

If you are familiar with Ollama first, this is easiest and more simple to try. Running llama-cli felt functionally identical to ollama run, albeit marginally more involved because you have to download the .gguf to serve yourself. This small technical hurdle aside, llama-cli isn’t really all that useful.

llama-server #

The real fun begins with a proper server for your model. This is how I’ve wrapped some other toys and been able to support marginally more advanced integration tools (one of the many OpenClaw alternatives) which are all the rage these days. I’ve been running Gemma 4 E4B (heavily quantized) on my server and it handles basic repetitive tasks that involve summarizing script output reasonably well.

Why use local models? #

Most people like getting useful info out of LLMs these days. I have mostly used them as a quicker search engine or rapid prototyping tool myself. When you dial down the complexity and ability of these models to fit appropriately on a consumer laptop, they (by comparison) kinda suck at these things. You’re left with a modest tool that can do basic coding tasks or run scripts, if it’s “agentic”.

I still find these tools quite useful. There are plenty of things that are marginally more complex than a simple shell script that I think are cool to do. My current use case for Gemma 4 E4B is summarizing the output of a simple web scraper script that I use. For doing something like this asynchronously, local models turn out to be quite capable. Gemma 4 is much, much better at summarization style tasks and writing things that actually sound reasonable and good given the information fed in to it.

These “simple” tasks are also not coding related or easily scripted. This is incredibly important. I find modern LLMs are viewed by non-software/non-techy people as what I would consider a heavily bloated cronjob. Sure, your agent framework (OpenClaw and the like) can do things that are moderately complex at the same time every day. That’s nice, I can set a script to run on a server, throw it in cron and have an email/text/Telegram message all the same without wasting tons of compute cycles. LLMs aren’t meant to be bloated scripts. That’s actually something they suck at, but they are really damn good summarizing tools. I’m convinced one of the main “futures” for LLMs is to basically write highly personal summaries based on the outputs of carefully curated scripts to gather a ton of info relevant to you specifically. This seems more likely to me than the alternative doomsday/golden age dichotomy that has been peddled ever since ChatGPT released.