Can I run AI offline, without internet?

The 60-second offline AI setup

If you want to try this in the next ten minutes:

1. Go to ollama.com and download for your OS (Mac, Windows, Linux).
2. Install (one-click).
3. Open your terminal.
4. Run: ollama run llama4:8b
5. Wait ~5 minutes for the model to download (about 5GB).
6. Type a message. Hit enter. You’re chatting with AI offline.

Total cost: $0. Total time: 10 minutes including the download.

What’s actually useful offline in 2026

The open-weight model landscape:

Model	Size	Hardware needed	Roughly comparable to
Llama 4 8B	5GB	Any 16GB-RAM laptop	GPT-4 (2024 era)
Llama 4 70B	40GB	M-series Pro / 24GB+ VRAM	GPT-4o
Mistral Large 2	80GB	High-end workstation	GPT-4o
DeepSeek-V4	40-200GB depending	High-end workstation	GPT-5 mini
Gemma 3 8B	5GB	Any 16GB-RAM laptop	GPT-4 (2024 era)

Free Claude.ai and free ChatGPT both run circles around these for general use. The point of offline isn’t “best quality”; it’s “good enough quality, with no data leaving your machine”.

When offline genuinely makes sense

Strong fit:

• Defence + intelligence contractors where data classification rules forbid cloud
• Healthcare practices doing transcription where audio is too sensitive
• Legal firms with confidential matter detail
• Anyone in remote Australia with poor internet
• Cost-sensitive heavy users (a self-hosted setup pays for itself in 6-12 months at heavy use)

Weak fit:

• Most Australian SMBs (cloud Claude/ChatGPT is faster, better, simpler)
• Solo operators (the setup overhead isn’t worth it)
• Anyone who can use anonymisation as their privacy layer
• Anyone who values the latest model improvements (cloud gets them weekly; you’d be re-downloading models monthly)

The hidden costs

Offline AI isn’t free; it’s “no marginal cost per query”. The hidden costs:

• Setup time: 2-10 hours depending on hardware + how technical you are
• Hardware: free if you use an existing laptop; $4-15k AUD for a dedicated workstation
• Model maintenance: monthly model updates as the open-weight space moves
• Quality gap: you’ll feel it on hard tasks
• No image generation: Stable Diffusion / Flux work locally but require separate setup

The hybrid pattern that works

Most regulated-industry clients we work with land on:

• Cloud Claude Pro ($30 AUD/month) for general work where data isn’t sensitive
• Local Ollama for the 10-20% of work where data must stay on-machine
• Azure OpenAI Service / AWS Bedrock when data must stay in Australian region but doesn’t need to be on-laptop

Three tiers, applied per task. Different from “we use one tool for everything”.

Self-hosting AI for a whole team

If you’re running offline AI for 5+ team members, you’re past the laptop tier. Options:

• Dedicated workstation running Ollama as a server (others connect via HTTP API). $5-15k AUD hardware.
• On-premise Hetzner / OVH dedicated server with a GPU. €100-500/month depending on spec.
• AWS Bedrock with provisioned throughput (not technically self-hosted, but private). $$$/month.

For most teams, this is over-investment. Cloud AI with the no-training privacy guarantee is enough.

See also

• Australian AI compliance landscape 2026 for the regulatory layer.
• Australian AI compliance landscape 2026 for when offline becomes a regulatory requirement.
• Is my data safe with Claude or ChatGPT? for the cloud-data privacy comparison.