Setting Up a Local AI Home Server: Hardware, Ollama and Frontends

A local AI home server lets every device on your network access LLM inference without internet dependency, cloud subscriptions, or sending data off your premises. The setup involves three decisions: choosing hardware that has enough RAM to run the model size you need, installing Ollama to handle model management and inference, and optionally adding a web frontend so you can use a chat interface from any device. Each of these steps is well-documented and takes less than an hour on a modern NAS or mini-PC. The total cost ranges from near zero if you are using hardware you already own, to $1,200 or more for a purpose-built dedicated device.

Step 1: Choose Your Hardware

The hardware decision is the most consequential part of the setup. It determines which model sizes are practical and how fast inference runs. There are three viable approaches depending on what you already own and what you want to spend.

If you already have a NAS: Start there. NAS devices with Docker support (Synology DS225+, DS425+, DS925+, DS1525+, and equivalent QNAP models) can run Ollama in a container. A NAS with 8GB of RAM handles 7B parameter models at 1 to 4 tokens per second, which is usable for background tasks, document summarisation, and occasional conversational use. It is too slow for regular interactive conversation. See the guide on NAS vs mini-PC for AI for a performance breakdown at each hardware tier.

If you want a dedicated device: A mini-PC in the $900 to $1,200 range with a Core Ultra or Ryzen AI processor and 32GB of RAM is the current sweet spot for daily interactive AI use. It generates 15 to 25 tokens per second on 7B models and handles 13B models at comfortable conversational speed. See the mini-PC buying guide for current Australian price tiers and what to look for.

If you want both: Many home setups use a NAS for storage and file serving and a dedicated mini-PC as the AI inference endpoint. The NAS holds model files and serves them over the network; the mini-PC handles compute-intensive inference requests from every device in the household. This is the most capable configuration and the one that makes the best use of hardware you likely already have running.

Step 2: Install Ollama

Ollama is the inference engine that manages model files and handles requests. It exposes an HTTP API on port 11434 that other applications (including Open WebUI) connect to. Installing Ollama is straightforward on any supported platform.

On Linux (mini-PC or desktop):

curl -fsSL https://ollama.com/install.sh | sh

This installs Ollama as a system service that starts automatically on boot. The service listens on localhost by default. To expose it to other devices on the network, edit the service configuration to set the listen address to 0.0.0.0:11434.

On a NAS via Docker (Synology, QNAP, Ugreen):

In Container Manager or Container Station, pull the ollama/ollama image and run it with a persistent volume for model storage. The container needs to expose port 11434 and have a volume mounted at /root/.ollama so downloaded models persist across container restarts. Use the NAS-specific guides for Synology or QNAP for exact configuration steps, as the NAS Docker interfaces vary in how they handle host networking and GPU passthrough.

On macOS:

Download the Ollama application from the Ollama website, install it, and it starts automatically. Ollama on macOS uses Apple Silicon's unified memory architecture, which means the M-series chip's GPU and NPU share the same memory pool as the CPU. This makes Apple Silicon Macs among the most efficient platforms for local inference at any RAM capacity.

Step 3: Download a Model

With Ollama running, pull your first model. For most users starting out, Llama 3.1 8B is the recommended default:

ollama pull llama3.1:8b

This downloads approximately 4.7GB to your model directory. On an Australian NBN connection, this takes 5 to 20 minutes depending on your download speed. Once downloaded, the model is stored locally and does not need to be downloaded again. Subsequent pulls for the same model only download if a newer version is available.

To test the model from the command line immediately after pulling:

ollama run llama3.1:8b

This opens an interactive terminal session where you can type prompts and receive responses. It confirms that inference is working before adding a frontend layer. For a full breakdown of which models to download for different use cases, see the Ollama model guide. For a detailed explanation of the Q4/Q6/Q8 quantisation tags in model names, see the quantisation guide.

Step 4: Add a Web Interface (Optional)

The command line is functional but not practical for daily use. Open WebUI is the most capable browser-based frontend for Ollama and runs in Docker alongside it. Once installed, any device on your network can access the chat interface through a browser without installing anything on those devices.

Install Open WebUI with Docker (replace host.docker.internal if Ollama is on a different machine):

docker run -d -p 3000:80 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

After the container starts, open http://[your-device-IP]:3000 in any browser on your network. The first time you open it, you create an admin account. Additional users can be created with their own accounts and conversation history. Open WebUI connects to Ollama automatically via the host gateway address and shows all locally available models in a dropdown. See the full Open WebUI setup guide for configuration details and alternatives.

What You Can Do With a Local AI Server

Once the server is running, every device on your network has access to a private AI assistant. The most common use cases for home setups are conversational AI for daily tasks, code assistance for developers working locally, document summarisation for reading long reports or contracts, and writing assistance for drafts and correspondence.

The Open WebUI frontend adds document upload for retrieval-augmented generation (RAG), letting you upload a PDF or text file and query it directly. This is useful for extracting information from long documents, comparing contracts, or searching a personal document archive using natural language. The RAG pipeline runs entirely on your hardware with no data leaving your network.

Running inference locally also removes the network dependency entirely. Your local AI server continues working during an NBN outage. It responds to prompts in under 50 milliseconds on a local network connection versus 200 to 300 milliseconds for cloud AI requests routed through Australian internet to US servers. For interactive conversation and coding assistance, that latency difference is noticeable. See the guide on NBN upload speeds and local AI for a detailed comparison.

Power, Cost and Practical Considerations

A local AI server that runs 24/7 adds between $40 and $130 per year in Australian electricity depending on your state and hardware. A mini-PC at 25 watts continuous costs approximately $57 to $94 per year. A NAS already running for storage costs almost nothing extra for AI inference. A GPU-equipped workstation running continuously can cost $500 or more per year, which is the main reason GPU rigs should be powered on only during active use. See the full power cost breakdown by state for detailed numbers.

Model storage takes disk space. A 7B model at Q4 quantisation is approximately 4 to 5GB. Keeping three or four models ready requires 15 to 25GB of model storage. If your main device has limited storage, a NAS as a model file server is a practical solution: store the model files on the NAS and configure Ollama to load models from the network path or an SMB mount.

For users accessing the server from outside the home network (from a phone over mobile data, or from a remote office), most Australian ISPs use CGNAT which prevents direct inbound connections to home servers. Tailscale resolves this by creating an encrypted mesh network between your devices that works through CGNAT without any router configuration. See the CGNAT and remote access guide for details.