Run AI where your data lives.
EuLLM Engine is an open-source runtime for large language models. One Rust binary, OpenAI- and Ollama-compatible, zero telemetry. From a Raspberry Pi to an A100.
Install and run in under a minute
Linux / macOS
$ curl -fsSL https://raw.githubusercontent.com/eullm/eullm/main/installer/install.sh | shWindows
$ irm https://raw.githubusercontent.com/eullm/eullm/main/installer/install.ps1 | iexThen run any GGUF model from Hugging Face:
$ eullm run hf.co/Qwen/Qwen3-8B-GGUF:Q4_K_MDrop-in Ollama and OpenAI compatibility
EuLLM speaks the Ollama and OpenAI APIs on the standard port 11434. Open WebUI, LangChain, n8n, and any OpenAI-compatible client work without changes.
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d {"model": "qwen3-8b", "messages": [{"role": "user", "content": "Hello"}]}Structured decisions, not just text
/v1/systemone reads the probability of every option straight from the model: structured decisions in milliseconds, for routing, triage and agents. We demonstrated it live on Snake — a 2B model deciding every move in around 8 ms.
The same endpoint powers simpler production tasks like email triage: route, escalate, or archive, decided locally in milliseconds.
Read how it works →Fast where it counts
Continuous batching serves many concurrent users from one GPU. Quantized KV cache extends context length on the same hardware. Large MoE models run on small GPUs, and even on CPU-only edge boards.
16 concurrent requests, one RTX 5070 Ti
context length via quantized KV cache (Q4_0/Q5/Q8)
35B MoE, CPU only, 20 W ARM board (Radxa Orion O6)
Runs everywhere
| OS | Status |
|---|---|
| Linux x64 | Tested |
| Windows x64 | Tested |
| macOS (Apple Metal) | Tested |
| ARM64 | Tested (Radxa Orion O6) |
GPU backends: NVIDIA CUDA · AMD ROCm · Vulkan · Apple Metal · CPU
Benchmarked on: EuroHPC Leonardo (4× NVIDIA A100) · EuroHPC LUMI (AMD Instinct, ROCm multi-GPU)
Vision and audio, fully local
Image understanding — OCR, scene description — runs stable on consumer GPUs. Audio transcription is available and marked experimental.
Every request logged, locally
EuLLM writes an audit log to a file you control on your own infrastructure — no external service, ready for your auditors.
No telemetry, no external calls
EuLLM Engine makes no calls to any external service. Prompts, documents and answers never leave the machine it runs on, unless you explicitly configure it to.