Run a local LLM on an air-gapped Linux box

1. On a connected staging machine: fetch everything

Download the llama.cpp source, the build tools your offline distro needs, and one GGUF model sized for your RAM (a 7-8B model at 4-bit quantization fits in roughly 5-6 GB). Choose a model whose license allows your use.

mkdir -p ~/xfer && cd ~/xfer
git clone --depth 1 https://github.com/ggml-org/llama.cpp
tar czf llama.cpp-src.tgz llama.cpp
# put your chosen model.gguf here too, then:
sha256sum llama.cpp-src.tgz *.gguf > SHA256SUMS

2. Carry it across

Use the sneakernet procedure: scan, verify hashes on the far side, log the transfer.

3. Build offline on the air-gapped box

Install build-essential/gcc-c++ and cmake from your offline package mirror or install media, then:

cd /opt && sha256sum -c ~/xfer/SHA256SUMS
tar xzf ~/xfer/llama.cpp-src.tgz && cd llama.cpp
cmake -B build -DLLAMA_CURL=OFF
cmake --build build --config Release -j

Add the GPU backend flag from the llama.cpp build docs if you have a supported GPU. LLAMA_CURL=OFF drops the model-download feature you won't use offline.

4. Serve on localhost only

Run the bundled OpenAI-compatible server bound to loopback, as an unprivileged user:

sudo useradd -r -s /usr/sbin/nologin llm
sudo -u llm /opt/llama.cpp/build/bin/llama-server \
  -m /opt/models/model.gguf --host 127.0.0.1 --port 8080 -c 8192

Test it:

curl -s http://127.0.0.1:8080/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{"messages":[{"role":"user","content":"In one line: what is Modbus function code 03?"}]}'

5. Prove it is really offline

Default-deny outbound with the host firewall, then confirm nothing listens beyond loopback:

sudo nft add table inet f
sudo nft add chain inet f out '{ type filter hook output priority 0; policy drop; }'
sudo nft add rule inet f out oif lo accept
sudo ss -tulpn   # llama-server should show 127.0.0.1:8080 only

Add allow rules only for the specific read-only data sources from lessons 4-5. Make rules persistent with your distro's nftables service.

6. Run it as a service

Create a systemd unit with User=llm, NoNewPrivileges=yes, ProtectSystem=strict, PrivateTmp=yes and Restart=on-failure, so it survives reboots and can't write outside its directories.

Where these facts come from

• ISA/IEC 62443 series: zones, conduits and security levels for industrial automation
• NIST SP 800-82 Rev. 3: Guide to Operational Technology (OT) security
• CISA: Industrial Control Systems: advisories and recommended practices
• Modbus Organization specifications: function codes and the TCP transport
• OPC Foundation: OPC UA specifications and security modes
• llama.cpp on GitHub: building and running local GGUF models on Linux

Checked Oct 29, 2026. Tutorials are educational. Follow your site's safety procedures, management-of-change process and the equipment maker's manual. Never test against live production equipment without authorization.