How to Use Ollama with VS Code

How to Use Ollama with VS Code: Your Local AI Coding Assistant

Ever wished you could have a powerful AI coding assistant right inside VS Code, but without sending your precious code off to the cloud? Maybe you’re often offline, work with super-sensitive data, or just love the idea of having a powerful LLM running entirely on your own machine. Good news: it’s totally doable! Thanks to Ollama and VS Code’s flexible integration capabilities, you can now harness the power of local large language models (LLMs) for everything from code explanations to boilerplate generation.

In this guide, we’ll walk you through setting up Ollama with VS Code, exploring different ways to integrate it, and what you can realistically expect from your local AI coding buddy.

Why Bring Your AI Assistant Home?

Before we dive into the “how,” let’s touch on the “why.” Running LLMs locally offers some compelling advantages:

• Privacy First: Your code is yours. With local models, sensitive algorithms, proprietary logic, or confidential business data never leaves your machine.
• Cost-Effective: Say goodbye to per-request API fees. Once you have the hardware, running local models costs nothing beyond electricity.
• Experimentation Playground: The open-source LLM world is exploding! Ollama gives you quick access to the latest models like Llama 3, Phi-4, Mistral, DeepSeek, and many more the moment they drop.
• Offline Power: Stuck on a plane, a remote site, or just have spotty internet? Your AI coding assistant keeps working, no connection needed (though VS Code itself might still need some internet for certain background tasks).

Getting Started: The Essentials

To get Ollama chatting with you inside VS Code, you’ll need a few things:

1. VS Code: The latest stable version is recommended.
2. Ollama: Download and install it from ollama.com. Make sure it’s running! You can usually check by visiting http://localhost:11434 in your browser; you should see a simple confirmation.
3. A Local Model: You need at least one LLM downloaded and ready to go via Ollama. We’ll cover how to get these.
4. (Optional but Recommended) GitHub Copilot: While not strictly required for all methods, having Copilot (even the free tier) can enhance the integration experience, especially for VS Code’s native chat features.

Pulling Your First Model

Once Ollama is installed and running, you need to download a model. Let’s start with a compact yet capable one like Phi-4, which is great for many coding tasks and runs well on consumer hardware:

ollama pull phi4

You can explore the vast library of models on the Ollama website and pull others using ollama pull <model_name>.

Method 1: Seamless Integration with VS Code’s Native Chat (Highly Recommended!)

This is the most integrated approach, allowing your local Ollama models to appear right alongside cloud-based assistants like GPT-4o or Claude within VS Code’s built-in chat interface.

Steps to Connect:

1. Open the Language Models Editor:

   • Go to the Chat view in VS Code.
   • Click on the model picker in the chat input field (it usually shows the current model, e.g., “GPT-4o”).
   • Select Manage Models from the dropdown.
   • Alternatively, open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) and type Chat: Manage Language Models.

2. Add Ollama as a Provider:

   • In the Language Models editor, click Add Models.
   • From the Quick Pick menu, select Ollama.
   • VS Code will prompt you for the Ollama endpoint. Enter http://localhost:11434 (or your custom Ollama server URL if it’s not running locally).
   • Click Confirm. VS Code will connect to your Ollama instance and automatically detect any models you’ve already pulled.

3. Select and Use Your Model:

   • Now, open the Chat view again.
   • Click the model picker in the chat input. You should see your Ollama models listed under the Ollama provider.
   • Select your desired model (e.g., phi4).
   • Start chatting! You can ask questions, get code explanations, or generate snippets, and your local LLM will handle it.

Agent Mode & Tool Calling Considerations

If you use VS Code’s agent features (where the AI can read files, run commands, etc.), not all Ollama models will support this directly. Models need specific “tool calling” or “function calling” capabilities.

• Models with Good Support: Llama 3.1+, Mistral Nemo, Phi-4, and newer variants often have solid tool-calling capabilities.
• Models with Limited/No Support: Older models or those not specifically fine-tuned for tool calling might just describe what they would do rather than actually performing an action, or they might struggle to output correct structured commands.

If a model doesn’t support tool calling, it simply won’t appear in the agent mode model picker.

Method 2: Using Dedicated VS Code Extensions

For those who prefer a more specialized experience or if the native integration doesn’t quite fit their workflow, several VS Code extensions can connect you to Ollama.

The Official Ollama Extension

The Ollama extension from the VS Code Marketplace provides a straightforward way to integrate Ollama models into VS Code Chat.

• Installation: Search for “Ollama” in the VS Code Extensions view and install it.
• Usage: After installation, it usually discovers models from your running Ollama instance automatically. You can then select them in the VS Code Chat interface.
• Troubleshooting: If models don’t appear, ensure Ollama is running (ollama serve in your terminal if needed), run ollama list to confirm models, and try Ollama: Refresh Models from the Command Palette.

Roo Code Extension

The Roo Code extension is geared towards more advanced coding assistance and offers specific configurations for Ollama.

• Installation: Install “Roo Code” from the VS Code Marketplace.
• Configuration:
  • Open Roo Code settings (gear icon in its window).
  • Set API Provider to Ollama.
  • Update the Base URL if your Ollama isn’t on http://localhost:11434.
  • Enter the specific Model ID you want to use.
  • Crucially for coding: Set the Context Window to at least 32K tokens for better performance on coding tasks.
• Recommended Models: For coding, qwen3-coder:480b or deepseek-v3.1:671b are suggested.

Key Considerations for Local LLMs

While local LLMs are fantastic, it’s important to set realistic expectations. File 2, “Building a Local AI Coding Assistant,” provides deep insights here:

• Context Window is King: Models have a limited “memory” (context window). A small context window (e.g., 4K tokens) means the model can forget earlier parts of a conversation or code. For coding, aim for models with 16K tokens or more, and configure your tools accordingly (like Roo Code’s 32K recommendation).
• Performance Varies: Running large models requires significant RAM and processing power. A 7B model might need 4-8GB of RAM just for itself. If your machine is strained, performance will suffer, leading to slow responses. The extension mentioned in File 2 includes a tuner that recommends models based on available RAM and suggests smaller models if yours is limited.
• Tool Calling Isn’t Perfect (Yet): As mentioned, local models are still catching up to cloud powerhouses in reliably executing tools or code actions. They might hallucinate, output malformed commands, or just describe what they’d do. For now, rely on them for explanations, boilerplate, and focused code generation.
• Model Personalities Differ: Don’t expect identical results from different models with the same prompt. CodeLlama might want detailed instructions, while DeepSeek-Coder might prefer directness. Experimentation and prompt tuning are key.

FutureFormDigital Insight & Recommendation

Using Ollama with VS Code offers an incredible path toward more private, cost-effective, and offline-capable AI assistance. For most users, the native VS Code Chat integration (Method 1) provides the smoothest and most powerful experience, especially if you’re using models with good tool-calling support. It integrates seamlessly into your existing workflow, making it feel like a natural extension of VS Code itself.

However, if you’re deeply invested in specific coding workflows or need highly tuned performance for complex coding tasks, exploring extensions like Roo Code could be beneficial, especially if you’re willing to fine-tune settings like the context window.

Our recommendation? Start with the native VS Code Chat integration. It’s the easiest to set up and offers a fantastic experience for general coding queries, explanations, and boilerplate generation. Use it as your go-to for privacy-sensitive work or when you’re offline. For highly complex, multi-file reasoning or tasks that absolutely require robust tool calling, you might still lean on cloud-based services, but for everyday coding assistance, your local Ollama setup is surprisingly capable and increasingly powerful.

---

Frequently Asked Questions (FAQ)

1. Do I need a powerful computer to run Ollama with VS Code?
   
   Not necessarily for basic models like Phi-4. However, larger or more capable models and longer context windows require more RAM and processing power. A machine with 16GB+ RAM is recommended for a smoother experience with 7B-13B parameter models.

2. Can I use Ollama models without an internet connection in VS Code?
   
   Yes, once Ollama and your desired model are downloaded and installed, you can use them offline within VS Code, provided your VS Code setup doesn’t rely on cloud services for its core functions.

3. What’s the difference between using VS Code’s native chat and a dedicated extension like Roo Code?
   
   Native chat integrates models directly into VS Code’s core chat UI, offering seamless switching and access to features like Agent Mode (if the model supports it). Extensions might offer more specialized configurations or workflows tailored to specific tasks.

4. Which Ollama models are best for coding?
   
   Models explicitly trained for coding, such as Llama 3.1, CodeLlama, DeepSeek-Coder, and Qwen-Coder variants, are generally better. Look for models with larger context windows (16K+ tokens) for better performance on complex tasks.

5. Why doesn’t my Ollama model work in VS Code Agent Mode?
   
   Agent Mode typically requires models that support “tool calling” or “function calling.” If your chosen Ollama model wasn’t trained for this, it won’t be available in Agent Mode. Check the model’s documentation for tool-calling capabilities.

6. How do I make my local LLM respond faster?
   
   Use smaller models, ensure your hardware meets the model’s requirements (especially RAM), and consider models optimized for speed. For coding, a larger context window can sometimes help models respond more efficiently by reducing the need for repetitive context re-evaluation.

7. Can Ollama models understand entire codebases?
   
   Local models have limited context windows, meaning they can only “remember” a certain amount of text at once. While they can analyze specific files or code snippets effectively, understanding a large, multi-file codebase comprehensively is still a challenge compared to cloud-based assistants with massive context windows.

8. Is it safe to use local LLMs for sensitive code?
   
   Yes, this is one of the primary benefits. Since the model runs entirely on your machine, your code and data do not get sent to external servers, significantly enhancing privacy and security.

9. How do I update Ollama or my models?
   
   To update Ollama itself, download the latest version from ollama.com. To update a model, you typically run ollama pull <model_name> again. Ollama will download the newer version and make it available. You might need to re-select it in VS Code.

10. What happens if I run out of context window with my local model?
    
    The model will start to “forget” earlier parts of the conversation or code. This can lead to repetitive answers, confusion, or inaccurate suggestions. It’s best to manage your conversation, use summarization, or start a new chat session if the context becomes too long for the model’s window.

    
    

---

What’s your favorite Ollama model for coding, and what’s the most mind-blowing thing you’ve had it do for you within VS Code? Share your experiences and recommendations in the comments below!

About The Author