Get Your AI Running: A Practical Guide to Installing Ollama on Ubuntu

So, you’re ready to dive into the world of local Large Language Models (LLMs) and want to set up shop on your trusty Ubuntu machine. Awesome! At FutureFormDigital, we’re all about empowering you with the tools and knowledge to build resilient, independent digital workflows. Running LLMs locally means more privacy, more control, and way less hassle with API keys and costs.

This guide is your no-nonsense walkthrough to getting Ollama up and running on Ubuntu, whether you’re rocking a powerful GPU or making do with a solid CPU. Let’s get your homegrown AI spinning!

FutureFormDigital Core Principle: Independence in your digital life means taking control of your tools. Installing Ollama on Ubuntu puts the power of advanced AI directly on your hardware, fostering self-sufficiency and innovation.

Why Ollama on Ubuntu? The Sweet Spot for Local AI

Ollama has become the go-to for many enthusiasts and developers looking to run LLMs locally. Why Ubuntu? It’s a stable, powerful, and widely-used Linux distribution, making it an ideal platform for tinkering with cutting-edge tech like AI.

• Simplicity: Ollama’s installation is remarkably straightforward, especially on Linux.
• Control & Privacy: Your data stays local. No need to send sensitive prompts or information to third-party servers.
• Cost-Effective: Once set up, running LLMs locally is essentially free (beyond your hardware’s electricity bill!).
• Flexibility: Whether you have a beefy NVIDIA GPU, an AMD card, or are sticking to CPU power, Ollama offers ways to get going.

Your Ubuntu AI Setup: Step-by-Step

Getting Ollama installed on Ubuntu is designed to be user-friendly. We’ll cover the quick install, manual steps, and essential configurations.

Prerequisites: What You’ll Need

Before we start, make sure your Ubuntu system is ready:

1. Ubuntu Server: A recent version of Ubuntu (22.04 LTS is highly recommended for stability).
2. SSH Access: You’ll need to connect to your server via SSH.
3. curl: This utility is essential for downloading the installation script. It’s usually pre-installed, but you can ensure it is with sudo apt install curl -y.
4. GPU Drivers (Optional but Recommended):
   • NVIDIA: Install the latest NVIDIA drivers. You can often do this easily with sudo ubuntu-drivers autoinstall. Verify with nvidia-smi.
   • AMD: For AMD GPUs, you’ll typically need ROCm drivers. Check AMD’s official ROCm installation guides for Ubuntu.
5. Firewall Configuration: If you plan to access Ollama remotely (e.g., from your local machine to a server), you’ll need to open port 11434.
   

   sudo ufw allow ssh       # Ensure you don't lock yourself out!
   sudo ufw allow 11434     # For Ollama API access
   sudo ufw enable          # Enable the firewall

   
   

The Easiest Way: The One-Line Install Script

Ollama provides a fantastic script that handles most of the heavy lifting for you.

1. Download and Run the Script: Open your terminal and execute:

   
   

   curl -fsSL https://ollama.com/install.sh | sh

   
   

   This script detects your system, downloads the correct Ollama binary, and sets it up to run as a systemd service, ensuring it starts automatically on boot.

   
   

2. Verify Installation: Check if Ollama is running and what version you have:

   
   

   ollama --version

   
   

   You should see the Ollama version number.

   
   

Manual Installation (If You Prefer Control)

If the script isn’t your cup of tea, or you need more fine-grained control, here’s the manual approach:

1. Download the Binary:
   
   For AMD64 (Intel/AMD CPUs):

   
   

   curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

   
   

   For ARM64 (e.g., Raspberry Pi, some cloud instances):

   
   

   curl -fsSL https://ollama.com/download/ollama-linux-arm64.tar.zst | sudo tar x -C /usr

   
   

   (Note: If you’re upgrading, it’s often recommended to remove old libraries first with sudo rm -rf /usr/lib/ollama)

   
   

2. Set Up as a Service (Recommended): To ensure Ollama runs reliably and starts on boot, set it up as a systemd service.

   • Create an Ollama user and group:
     

     sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
     sudo usermod -a -G ollama $(whoami) # Add your user to the ollama group if needed

     
     
   • 
     

     Create the service file:

     
     

     sudo systemctl edit ollama.service

     
     

     Add the following content to the editor that opens:

     
     

     [Unit]
     Description=Ollama Service
     After=network-online.target
     
     [Service]
     ExecStart=/usr/bin/ollama serve
     User=ollama
     Group=ollama
     Restart=always
     RestartSec=3
     Environment="PATH=$PATH"
     # If using GPU, add appropriate environment variables here if needed.
     
     [Install]
     WantedBy=multi-user.target

     
     
   • Reload systemd, enable, and start the service:
     

     sudo systemctl daemon-reload
     sudo systemctl enable ollama
     sudo systemctl start ollama

     
     
   • Check its status:
     

     sudo systemctl status ollama

     
     

     You should see it’s active and running.

     
     

Enabling GPU Acceleration (NVIDIA & AMD)

If you have a GPU, Ollama can leverage it for much faster performance.

• NVIDIA: Ensure your NVIDIA drivers are installed correctly (nvidia-smi should work). Ollama bundles its own CUDA runtime, so you generally don’t need a separate CUDA Toolkit installation.
• AMD: For AMD GPUs, you’ll typically need ROCm drivers. Follow AMD’s official ROCm installation guides for Ubuntu. Ensure you install ROCm v7 or later. You might need to add the ollama user to the render group or configure specific permissions. Ollama also supports Vulkan for broader GPU compatibility.

Configuring for Remote Access

By default, Ollama only listens on localhost. If you want to access it from other machines on your network (or your local machine if Ollama is on a server), you need to configure it to listen on all interfaces.

1. Edit the Service File: Use sudo systemctl edit ollama.service.
2. Add/Modify Environment Variable: Add or change the Environment line for OLLAMA_HOST:“`ini
   
   [Service]
   
   ExecStart=/usr/bin/ollama serve
   
   User=ollama
   
   Group=ollama
   
   Restart=always
   
   RestartSec=3
   
   Environment=”PATH=$PATH”
   
   Environment=”OLLAMA_HOST=0.0.0.0:11434″ # This makes it accessible network-wide
   
   
3. Reload and Restart:
   

   sudo systemctl daemon-reload
   sudo systemctl restart ollama

   
   

   Ensure your firewall (ufw) allows traffic on port 11434!

   
   

Running Your First LLM: A Quick Start

Once Ollama is installed and running (and configured for remote access if needed), it’s time to play!

1. Pull a Model: Ollama makes downloading models as easy as ollama pull <model_name>. Let’s start with a popular choice:

   
   

   ollama pull llama3.1

   
   

   This will download the Llama 3.1 model (around 5GB for the 8B version). It might take a few minutes depending on your internet speed.

   
   

2. Run the Model: After downloading, you can interact directly from the terminal:

   
   

   ollama run llama3.1

   
   

   You’ll get a prompt where you can type your questions or commands to the AI. To exit, type /bye.

   
   

Performance: CPU vs. GPU on Ubuntu

The difference a GPU makes is astronomical.

• CPU-Only: If you’re running on CPU, expect a more “deliberate” pace. Smaller models like Phi-3 might offer 2-3 tokens/sec, while larger ones like Llama 3 can be as slow as 0.5-1 token/sec. It works for basic testing, but it’s not ideal for natural conversation.
• GPU Acceleration: With a decent NVIDIA GPU (like an RTX 3080), you can see 45-50 tokens/sec for Phi-3 and 25-30 tokens/sec for Llama 3. This makes interaction feel snappy and genuinely useful. If you have an AMD GPU with ROCm configured, you’ll see similar benefits.

Troubleshooting Common Hiccups

• nvidia-smi not found: Make sure your NVIDIA drivers are installed correctly. sudo ubuntu-drivers autoinstall followed by a reboot usually sorts this out.
• Ollama using CPU instead of GPU: Ensure the NVIDIA Container Toolkit is installed and configured if you’re using Docker. For native installs, verify your NVIDIA/AMD drivers are up-to-date and properly loaded. Check ollama serve logs (journalctl -u ollama) for clues.
• Out of Memory Errors: Try smaller models, quantized model versions (e.g., ollama pull llama3:7b-q4_0), or models that require less VRAM. Ensure you have enough system RAM if running CPU-only.
• Port 11434 blocked: If you can’t access Ollama remotely, double-check your firewall rules (sudo ufw status) and ensure port 11434 is allowed.
• Slow Model Downloads: Ensure you have enough disk space. Some models are tens of gigabytes!

FutureFormDigital Insight: Embrace Your Local AI Power

Setting up Ollama on Ubuntu is a fantastic step towards digital independence. It puts powerful AI capabilities directly into your hands, offering unparalleled privacy and control.

Our Recommendation: If your hardware allows, prioritize getting GPU acceleration working. The performance boost from even a mid-range NVIDIA or a capable AMD GPU (on Linux) transforms the experience from a novelty to a truly practical tool. For Ubuntu servers, ensure you’ve configured remote access (OLLAMA_HOST=0.0.0.0:11434) and have port 11434 open on your firewall if you need network access.

---

Frequently Asked Questions (FAQs)

1. What is the easiest way to install Ollama on Ubuntu?
   Use the official one-line script: curl -fsSL https://ollama.com/install.sh | sh. It handles most setup, including creating a systemd service.
2. Do I need a GPU to run Ollama on Ubuntu?
   No, Ollama can run on CPU-only, but performance will be significantly slower. A GPU is highly recommended for a good experience with most models.
3. How do I enable GPU support for Ollama on Ubuntu?
   Ensure your NVIDIA or AMD drivers are correctly installed. Ollama should detect them automatically for native installs. For Docker, ensure the NVIDIA Container Toolkit is set up.
4. Which Ubuntu version is best for Ollama?
   Ubuntu 22.04 LTS is recommended for its stability and up-to-date package support, which is beneficial for AI workloads and drivers.
5. How much RAM do I need for Ollama on Ubuntu?
   Ollama’s minimum is 8 GB RAM, but 16 GB is strongly recommended for comfortable CPU use, and 32 GB+ is better if you plan to run larger models or use your system for other tasks alongside AI.
6. How do I make Ollama accessible from other machines on my network?
   Edit the Ollama systemd service file (sudo systemctl edit ollama.service) and set Environment="OLLAMA_HOST=0.0.0.0:11434", then reload and restart the service. Ensure your firewall allows port 11434.
7. How do I update Ollama on Ubuntu?
   Simply re-run the installation script: curl -fsSL https://ollama.com/install.sh | sh. It will update your existing installation.
8. What happens if I get “out of memory” errors?
   Try smaller models, quantized model versions (e.g., ollama pull llama3:7b-q4_0), or models that require less VRAM. Ensure you have enough system RAM if running CPU-only.
9. Can I run Ollama on Ubuntu with an AMD GPU?
   Yes, Ollama supports AMD GPUs on Ubuntu, primarily via ROCm drivers. Ensure you install the correct drivers and potentially configure Vulkan.
10. How do I view Ollama logs on Ubuntu?
    If running as a systemd service, use journalctl -e -u ollama to view recent logs.

---

What’s Your Ubuntu AI Strategy?

Are you planning to use Ollama on a dedicated server, a desktop, or perhaps a Raspberry Pi? What’s your go-to model for getting started with local LLMs on Ubuntu? Share your setup and favorite models in the comments below – let’s build our independent AI futures together!

About The Author