Local LLMs for Development¶

I am experimenting with local instances of Large Language Models (LLMs) for software development. While this experimentation continues, I will update this post with further information.

Reasoning¶

Local LLMs, at least in theory, may offer advantages over using APIs to access LLMs operated by large corporations in commercial data centers.

Local hardware can use less water and power than large data centers
Data stays on the developer’s device, addressing privacy concerns
With open source models, developers are not dependent upon large corporations who may have conflicting values with the developer

Note however that concerns around skill formation, de-skilling, and plagiarism persist with local LLMs.

Implementation¶

I have selected VS Code as my IDE, as it has support for various extensions that support different levels of LLM integration into software development.

Author’s Note

Note that VS Code is owned by Microsoft, there is both a proprietary and open source version of the code. The open source version of the code is under the MIT license.

Specifically, I am using the Continue extension.

Author’s Note

Note that while Continue is an open source project, the company Continue has been purchased by Cursor. Cursor is itself being purchased by SpaceX as part of its xAI subsidiary. While the extension will remain open source (at this time), it is worth monitoring this development for developers who wish to avoid directly relying upon large US software corporations. The Cursor extension is under the Apache 2.0 license.

I am using Ollama to operate the open LLM models.

Author’s Note

Ollama is an open source software under the MIT license.

Finally, I am currently using the Qwen3 model.

Author’s Note

Qwen3 is an open source model under the Apache 2.0 license.

The relevant portion of the Continue configuration file is given below:

- name: Qwen3 8B
  provider: ollama
  model: qwen3:8b
  apiBase: http://localhost:11434
  contextLength: 9742
  capabilities:
    - tool_use
  defaultCompletionOptions:
    contextLength: 9742
    maxTokens: 2048
  roles:
    - chat
    - autocomplete
    - edit
    - apply
    - summarize

Observations¶

Experimentation to select the correct model and context length is important. I am working with a development machine that has a GPU with only 6GB of VRAM. This limits the models and context length that I can select.

Specifically, I am currently experimenting with using Qwen3 8B via Ollama with a contextLength of 10**13.25≈9742 tokens and a maxTokens of 2048. While these settings limit the capability of the model to ingest large codebases, it keeps the performance relatively responsive while allowing me to generate smaller snippets of code. I do not see this limitation as particularly problematic, as this is consistent with my general recommendation to RSEs to only generate small, focused amounts of code at once and maintain control over the overall design and review while creating research software.

Metadata¶

Started: 10 Jul 2026

Last edited: 10 Jul 2026