KoboldCpp for local interactive writing

How it works

KoboldCpp brings a llama.cpp-derived runtime together with a web front end and APIs. The interface exposes sampling and prompt controls suited to repeated continuation of the same story.

What you need

Start with an instruct or story-oriented checkpoint compatible with the runtime and the right prompt template. Decide whether you will use the bundled interface or connect an existing client over its API.

What to try first

Load a small instruct GGUF, save a short scene and run two continuations with different repetition settings. Repeat the exercise after extending the scene close to the context limit.

Save a story checkpoint before testing context rollover. A continuation may still sound fluent after it has lost a character name or fact established near the start.

Where to look for failures

Quantization and context settings affect both speed and continuity. Match the prompt mode to the checkpoint and measure memory use on the intended machine.

Related Franklin pages

Separate the model runtime from the user interface. A local endpoint makes it possible to keep the same client while comparing model loading, chat templates and device support.

• Pull, run and serve language models behind a local API. Ollama for running local language models
• Run quantized language models with a native CLI or HTTP server. llama.cpp for GGUF inference

For more on how model size affects deployment decisions, read Franklin’s guide. Small language models

Project documentation

• Current project README
• README at the reviewed revision