How do I run an AI chatbot fully offline on Linux?

Short answer: Build llama.cpp from source, copy a GGUF model file to the machine, and run llama-server bound to 127.0.0.1. No internet or account is needed after the files are on the box.

cmake -B build -DLLAMA_CURL=OFF && cmake --build build --config Release -j
./build/bin/llama-server -m model.gguf --host 127.0.0.1 --port 8080

Full walkthrough, including firewalling and systemd: Run a local LLM on an air-gapped Linux box.

Questions

Build llama.cpp from source, copy a GGUF model file to the machine, and run llama-server bound to 127.0.0.1. No internet or account is needed after the files are on the box.

Do I need a GPU?

No. 7-8B models at 4-bit run on a modern CPU with 16 GB RAM, slower than a GPU but usable.

Where these facts come from

ISA/IEC 62443 series: zones, conduits and security levels for industrial automation
NIST SP 800-82 Rev. 3: Guide to Operational Technology (OT) security
CISA: Industrial Control Systems: advisories and recommended practices
Modbus Organization specifications: function codes and the TCP transport
OPC Foundation: OPC UA specifications and security modes
llama.cpp on GitHub: building and running local GGUF models on Linux

Checked Oct 29, 2026. Tutorials are educational. Follow your site's safety procedures, management-of-change process and the equipment maker's manual. Never test against live production equipment without authorization.

Industrial control with bots: all lessons →