Deploy Your Private AI with Ollama and Open WebUI on WSL2

What if the AI assistant your whole family uses ran on your own hardware, answered in seconds, and never sent a single word of your conversations to the cloud?

You can build exactly that. With free, open-source tools and a PC with a decent GPU, a private AI runs entirely on hardware you already own: no subscription fee, no account with a vendor, no data leaving your home network. Moreover, this is one of the most rewarding home-lab projects you can take on. By the end you will have worked through Linux, Docker, GPU drivers, and home networking, and you will understand how those four layers fit together.

This guide walks you through the whole build: a private, ChatGPT-style chatbot powered by Ollama and Open WebUI, accelerated by your NVIDIA GPU, and reachable from every phone, laptop, and tablet in the house.

I have this running at home, and the hardest parts were never the AI. They were three small networking details that no tutorial warned me about, so I have written each of them down at the exact point where you will meet them.

What You Need Before You Start

#

• A Windows 11 PC with an NVIDIA GPU. The GPU is what makes answers feel immediate rather than sluggish. Mine is an RTX 3070 Ti with 8 GB of VRAM, the card from my component-selection guide for building your own PC, and serving this assistant is what that build ended up being pointed at. Every number in this guide was measured on it, so you can scale your expectations against a mid-range card rather than a workstation.
• A Windows NVIDIA driver on version 471.11 or later. Nothing needs to be installed inside Linux.
• Free disk space for the model weights, which run to several gigabytes per model.
• Administrator access on Windows, because a few steps touch the firewall and the port forwarding table.

The Stack

#

Back in 2024, Ollama did not have a native Windows installer, so the only way to run it on a Windows machine was through WSL2 (Windows Subsystem for Linux). Far from being a limitation, WSL2 is genuinely the best way to do this: you get a real Linux environment with full GPU access, running right alongside Windows on hardware you already own.

The full stack:

• WSL2 on Windows 11: a genuine Linux environment running alongside Windows, with direct access to the NVIDIA GPU.
• Ollama: handles downloading, managing, and serving open-weight LLMs locally.
• Open WebUI: a clean, ChatGPT-style browser interface that talks to Ollama over HTTP.
• Docker: runs Open WebUI as a container inside WSL, keeping the setup portable and clean.
• Portainer (optional): a visual dashboard for managing Docker containers without the command line.
• WSL2 with an automated port proxy: the configuration that makes the chatbot reachable from other devices on the home network, with a small scheduled task that keeps the forwarding rules from breaking on every reboot.

Stacked up, a request from your phone travels like this:

Phone, laptop, or tablet on the LAN      the devices everyone already has
        │  http://<PC-LAN-IP>:8080
        ▼
Windows host (netsh portproxy)           re-pointed at every logon by a script
        │  0.0.0.0:8080 -> <WSL-IP>:8080
        ▼
Open WebUI (Docker, --network=host)      the ChatGPT-style interface
        │  OLLAMA_BASE_URL=http://127.0.0.1:11434
        ▼
Ollama (native service, NVIDIA GPU)      downloads, manages, and serves the model

Don’t be put off by WSL. It is a remarkably powerful tool: you get the full Linux command line (package managers, shell scripts, Docker, everything) while staying in the familiar Windows environment. If you have never used it before, this project is a great first encounter.

Step 1: Enable WSL2 and Install Ubuntu

#

Open PowerShell as an administrator and run:

wsl --install

This enables WSL2 and installs Ubuntu by default. After a reboot, launch Ubuntu from the Start menu and complete the initial user setup (username and password).

If you already have WSL installed and want to confirm it is using version 2:

wsl --list --verbose

Look for VERSION 2 next to your distribution. If it shows version 1, upgrade it with wsl --set-version Ubuntu 2.

Step 2: Verify Your NVIDIA GPU in WSL

#

WSL2 on Windows 11 exposes your GPU automatically, so you do not need to install a separate NVIDIA driver inside Linux. The driver lives on the Windows host; WSL bridges it. The only requirement is that your Windows NVIDIA driver is version 471.11 or later (any driver released after mid-2021 should qualify).

Inside the WSL terminal, run:

nvidia-smi

You should see a table listing your GPU name, driver version, and CUDA version.

If the command is not found or returns an error, open GeForce Experience (or download the latest driver from nvidia.com) and update your Windows driver, then try again.

This step matters: Ollama will automatically use the GPU for inference if CUDA is visible, making responses significantly faster.

Step 3: Install Ollama and Pull Your First Model

#

Inside your WSL terminal, run the official install script:

curl -fsSL https://ollama.com/install.sh | sh

The script detects CUDA and configures Ollama to use the GPU. Once finished, pull and run a model. I used llama3.1, which was the state-of-the-art open model at the time:

ollama run llama3.1

The first run downloads the model weights (several gigabytes, depending on the model). After that, you will land in an interactive chat prompt right in the terminal. Try a question to confirm everything works, then exit with /bye.

Ollama also starts a background HTTP server on http://127.0.0.1:11434. This is the API that Open WebUI will use.

What I run today (August 2026): llama3.1 was the right pick in 2024, but open models move quickly and it is no longer installed on this machine. The engine that serves the house now is Qwen3.5 4B (qwen3.5:4b, 4.7B parameters, Q4_K_M quantisation, 3.4 GB on disk), pulled exactly the same way with ollama run qwen3.5:4b. Nothing else in this guide depends on which model you choose, so start with whatever is current when you read this. The measured results further down explain why a 4B ended up beating a much larger model on an 8 GB card.

Step 4: Deploy Open WebUI with Docker

#

Install Docker Engine inside WSL. The quickest path is the official convenience script:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

Log out and back into the WSL session so the group change takes effect, then start the Docker daemon:

sudo service docker start

Now run the Open WebUI container:

sudo docker run -d \
  --network=host \
  -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

A note on OLLAMA_BASE_URL: the value http://127.0.0.1:11434 tells the container how to reach Ollama, and it is what I recommend writing. 0.0.0.0 is really a listen address, meaning “accept connections on every interface”, rather than a destination. Linux does accept it as a destination and routes it to localhost, so OLLAMA_BASE_URL=http://0.0.0.0:11434 works too (my own compose file has been running that way for a long time), but it works because of platform-specific behaviour and not because the address means what you intend. Writing 127.0.0.1 says exactly what you mean and carries over unchanged to any other platform. Because the container runs with --network=host, it shares the WSL network namespace with Ollama, so 127.0.0.1 resolves correctly.

Open a browser on your Windows machine and navigate to http://localhost:8080. You should see the Open WebUI login page. Create an account (no email verification required, it is entirely local) and start chatting.

Step 5: Reach Your Private AI from Every Device in the House

#

At this point the chatbot is only reachable from the PC itself. To open it up to the rest of the house, you forward the relevant ports from the Windows host into WSL with netsh portproxy. This approach has one well-known catch: WSL2 gets a fresh internal IP every time Windows restarts, which means a static forwarding rule silently breaks after every reboot. The fix is not to abandon port forwarding, but to automate it: a small script re-points the forward at WSL’s current IP, run automatically by a scheduled task at every logon.

Why not “mirrored” mode? Windows 11 also offers networkingMode=mirrored, which gives WSL the host’s IP directly and removes the need for any forwarding. It is tempting, but it routes 127.0.0.1 connections through the Windows network stack, which breaks Ollama’s GPU discovery (the loopback probe hangs and Ollama silently falls back to CPU) and, in my case, interfered with a VPN. NAT mode plus an automated port proxy keeps the GPU working and is just as reliable once the refresh is scripted.

Keep WSL on NAT networking

#

NAT is WSL2’s default mode, but make it explicit. Create or edit C:\Users\<you>\.wslconfig on Windows (it lives in your Windows user folder, not inside WSL):

[wsl2]
networkingMode=NAT
localhostForwarding=true

Then shut down WSL from PowerShell so the change takes effect:

wsl --shutdown

Open the ports in WSL’s firewall

#

sudo ufw allow 8080/tcp
sudo ufw allow 11434/tcp

Port 8080 is Open WebUI. Port 11434 is the Ollama API; you will need it reachable on the network if you ever want to connect other tools (or, later, a Discord bot) directly to the model server.

Let Ollama listen on the network

#

By default, Ollama only listens on 127.0.0.1. To make it reachable from other devices, override its service configuration:

sudo systemctl edit ollama

In the editor that opens, add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Save and restart:

sudo systemctl restart ollama

Format matters: The value must be 0.0.0.0:11434 with no http:// prefix. Adding the prefix causes Ollama to parse it as a URL and bind to IPv6 [::] instead of IPv4, which does not reliably receive IPv4 LAN traffic. Keep it as a bare host:port.

Forward the ports from Windows into WSL

#

WSL’s internal IP changes on every boot, so hardcoding it would break after the next restart. Instead, write a small PowerShell script that looks up the current WSL IP and rebuilds the forwarding rules. Save it as C:\Users\<you>\wsl-portproxy.ps1:

# Re-points Windows port forwarding at WSL's current internal IP so LAN
# devices can reach the services inside WSL at the Windows host IP.
# Run as Administrator.

$ports = @(8080, 11434)

# Get WSL's current internal IPv4 address (first one from `hostname -I`)
$wslIp = (wsl hostname -I).Trim().Split(" ")[0]
if (-not $wslIp) { Write-Error "Could not determine WSL IP. Is WSL running?"; exit 1 }
Write-Host "WSL IP: $wslIp"

foreach ($p in $ports) {
    # Clear any stale rule for this port, then add a fresh one
    netsh interface portproxy delete v4tov4 listenport=$p listenaddress=0.0.0.0 2>$null | Out-Null
    netsh interface portproxy add    v4tov4 listenport=$p listenaddress=0.0.0.0 connectport=$p connectaddress=$wslIp | Out-Null
    Write-Host "portproxy: 0.0.0.0:$p -> ${wslIp}:$p"

    # Ensure the Windows firewall allows inbound on this port (only added once)
    $ruleName = "WSL portproxy $p"
    if (-not (Get-NetFirewallRule -DisplayName $ruleName -ErrorAction SilentlyContinue)) {
        New-NetFirewallRule -DisplayName $ruleName -Direction Inbound -Action Allow `
            -Protocol TCP -LocalPort $p | Out-Null
        Write-Host "firewall: allowed inbound TCP $p"
    }
}

Write-Host "`nDone. Current portproxy table:"
netsh interface portproxy show v4tov4

Run it once from an Administrator PowerShell to apply the forwards immediately. The firewall rules it creates accept inbound on those ports from anywhere; if you would rather restrict them to your home network, add -RemoteAddress 192.168.1.0/24 to the New-NetFirewallRule call, substituting your own LAN range (the subnet ipconfig reports for your Wi-Fi or Ethernet adapter, which is often 192.168.0.0/24 or 10.0.0.0/24 instead).

Refresh the forward automatically on every reboot

#

So you never have to run the script by hand, register it as a scheduled task that fires at logon. From an Administrator PowerShell:

$action  = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-WindowStyle Hidden -ExecutionPolicy Bypass -File C:\Users\<you>\wsl-portproxy.ps1"
$trigger = New-ScheduledTaskTrigger -AtLogOn
Register-ScheduledTask -TaskName "WSL portproxy" -Action $action -Trigger $trigger -RunLevel Highest

Now find your PC’s LAN IP address (ipconfig in PowerShell, look for the IPv4 address of your Wi-Fi or Ethernet adapter) and navigate to http://<PC-LAN-IP>:8080 from any other device on the same network. After a reboot, the task re-points the forward at WSL’s new internal IP automatically, so the LAN address stays the same.

Headless caveat: the task fires at logon. If you reach the machine without logging in (for example over \\wsl$ or a remote tool), run wsl-portproxy.ps1 once by hand to bring the forward up.

Step 6: Portainer (Optional)

#

If you prefer a visual interface for managing your Docker containers rather than using the command line, Portainer is a lightweight option. Run it alongside Open WebUI:

sudo docker run -d \
  --network=host \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  --name portainer \
  --restart always \
  portainer/portainer-ce

Portainer’s web UI will be available at http://localhost:9000, which is Portainer CE’s default HTTP port (9443 serves the same interface over HTTPS). Because the container runs with --network=host, it publishes nothing of its own, it simply listens on WSL’s ports, so reaching it from another device would need the same portproxy treatment as Step 5. It lets you start, stop, and inspect containers with a few clicks, which comes in handy when you want to update Open WebUI without remembering the exact docker run flags.

What I actually run, and why it sidesteps Step 5 entirely. There is no Portainer server inside WSL on my machine. Instead, WSL runs a Portainer Edge agent (portainer/agent:latest with EDGE=1 plus the edge ID and key that the server issues), while the Portainer server itself lives on my home NAS. The agent dials out to that server and holds the tunnel open, so it publishes no ports at all and never has to be found at WSL’s address. That is a neat inversion of the problem Step 5 spends so long solving: instead of teaching the network how to reach a moving target, the moving target keeps a connection open to something that does not move, which makes it immune to WSL’s changing IP and to reboots. If you already have a Portainer server somewhere on your network, this is the more robust way to manage the containers living inside WSL.

How Fast Is It Really? Measured Numbers on an 8 GB GPU

#

The whole reason for putting a GPU behind this stack is speed, so here are real figures instead of adjectives. I measured them in August 2026 on the RTX 3070 Ti (8 GB) with Ollama 0.24.0, calling the /api/generate endpoint directly and reading Ollama’s own eval_count / eval_duration counters, with think: false, three different prompt sizes, and the median of three runs at every point.

That last column is an absolute nvidia-smi reading taken during a quiet session, so it counts everything on the card, the model and whatever the Windows desktop happened to be holding (434 MiB at the time). The memory figure for the 4B was measured on my tuned build, which carries the same weights as qwen3.5:4b and occupies the same space.

Three things are worth reading out of that table. First, about 100 tokens per second on a 4.7B model with 0.2 s to the first token means the answer starts appearing the moment you stop typing and then arrives far faster than anyone reads it, which is the practical bar for a family assistant. Second, the rate is flat with length: the 4B held 101 to 105 tok/s whether it produced 34 tokens or 534. Third, loading a model from disk costs 3.4 to 6.4 seconds, paid only on the first request after Ollama unloads it.

The “tuned build” in the last row is a Modelfile over the stock weights rather than retrained weights, which is why my tuned 4B measured 103 tok/s against the stock 4B’s 102: within a point of each other. The tuning changes the assistant’s behaviour, not its speed. That customisation is the subject of part 2 of this series.

Size the Model to the Card, Not to the Disk

#

Here is the finding I did not expect, and the one most likely to save you an evening.

A 6.6 GB model on an 8 GB card looks like a comfortable fit on paper. It is not, because the model is never the only tenant, and the other tenant will not hold still: the Windows desktop claimed 434 MiB of VRAM during a quiet session and 1,295 to 1,326 MiB while the machine was actually being used. Load the 9.7B model at num_ctx 8192 and the card fills to 7,921 MiB of its 8,192 MiB in the quiet case and to 7,883 to 7,939 MiB in the busy one, which is 250 to 270 MiB of margin either way, about 97 % full. The total lands in the same place under both conditions because Ollama sizes its allocation to whatever is free rather than taking a fixed amount. Nothing looked wrong from the outside, and ollama ps still reported 100% GPU.

At rest it behaved well and reproducibly: six consecutive long generations all landed between 69 and 72 tok/s, which is the honest steady-state figure for that model on this card. During one benchmark sweep, however, the same model answering the same prompt collapsed to 19 to 21 tok/s, with prompt processing falling from about 800 tok/s to 133. That is what a quarter of a gigabyte of margin actually buys: the margin is not merely small, it moves with whatever Windows is drawing, so the moment the desktop asks for more VRAM, part of the work slides off the GPU. The collapse is intermittent rather than permanent, and that is precisely what makes it a poor foundation for a service other people depend on. It is fast every time you test it and slow the one time someone else needs it.

The 4B model solves this not by being cleverer but by leaving room. Its footprint is a fixed 5,515 MiB, so what varies is the free space rather than the model: roughly 2.2 GB free when the desktop is idle and about 1.4 GB when it is busy. A margin that comfortably absorbs the desktop’s drift is what keeps the speed steady. Moving the family assistant from the 9.7B to the 4B is exactly the decision I walk through in part 2, and the reasoning generalises: when you choose a model, read the free VRAM after it loads, not the file size before it.

What You Get Once It Is Running

#

You end up with an AI assistant that is fast, always available, and completely private: no subscription, no account, no data leaving your house. Every device on your network reaches it at http://<your-PC-IP>:8080, and it keeps serving answers whether or not anyone is sitting at the PC.

The use cases are more practical than they might sound. In my home, the family uses it to ask about the Wi-Fi password, which streaming service has a particular show, or how to reconnect a smart home device that dropped off the network. These are the kind of questions that used to require a search engine or a shout across the house. Your household will find its own everyday uses quickly.

What the Project Teaches You

#

Beyond the assistant itself, this is one of the best home-lab projects you can take on as an engineer, because it makes you cross four boundaries in a single build: GPU passthrough into a virtualised Linux environment, container networking, a Linux firewall, and the bridge between the Windows host and WSL. Each of those skills transfers directly to the next thing you build.

The obstacles are the curriculum, and four of them taught me the most:

• The tempting shortcut is not always the right one. Mirrored networking would have removed the port forwarding entirely, but it broke Ollama’s GPU discovery and interfered with my VPN. NAT plus a script was the less elegant option and the one that actually worked.
• Configuration formats are not suggestions. An http:// prefix on OLLAMA_HOST was enough to bind the server to IPv6 and make it invisible to the rest of the LAN. Reading how a program parses its own settings costs far less time than guessing at it.
• A fix you have to remember is not a fix. Re-pointing the forward by hand after every reboot is not something anyone keeps doing for long. Turning it into a script and a scheduled task is what turned a working demo into something the family relies on.
• Fitting is not the same as running well. The 9.7B model fit in 8 GB with 250 MiB to spare, and Ollama happily reported 100% GPU. It also lost most of its speed whenever the desktop wanted VRAM back. Measuring the headroom after a load, rather than trusting the file size, is a habit worth building early.

Although none of those problems appeared in the guides I started from, each one left me understanding the stack better than a clean first attempt ever would have. That is the pattern I keep meeting in home-lab work: every setback is simply another opportunity to learn from past mistakes and keep improving.

What’s Next

#

Once your private AI is running, the stack becomes a foundation rather than a finished product. Here are the directions worth exploring next:

• Trying different models: Ollama makes it trivial to pull and switch between models; comparing your initial pick with other open-weight options is a natural next step, and the table above is a template for benchmarking them honestly on your own card.
• Adding a custom system prompt: giving the assistant a persona and a set of house-specific context to make it even more useful for everyday family questions.
• Reaching it from outside your network: put Open WebUI behind a reverse proxy (Nginx Proxy Manager is a good starting point) and point a domain at it, so the chatbot is reachable from your phone even when you are not home.
• Wiring it into Discord: a small bot that relays messages to the Open WebUI API and posts the replies back to a channel, so the family can use it from an app they already have open without navigating to a URL. I built exactly that, and part 2 of this series walks through the whole bridge: Give Your Private AI a Discord Bot Your Family Will Actually Use.

Whichever direction you take, the important part is that the assistant is yours: your hardware, your data, your rules. If you build your own version of this, I would love to hear how it goes.