27 May Running Omniscope against a local Qwen3.6-27B-FP8 model on an H100
May 2025: This space changes fast, so treat anything more than a few months old with suspicion.
There is a lot of talk about local models, private AI, open-weight models, on-prem AI, GPU deployments, and so on.
Most of it stays a bit abstract.
So this is not meant to be a theoretical post about whether local AI is better or worse than using OpenAI, Anthropic, Google, OpenRouter, or anything else. This is a practical note.
We wanted a known-working configuration for running Omniscope’s AI features against a self-hosted open-weight model.
A real model. On a real GPU. With tool calling. With reasoning output handled correctly. With Omniscope actually using it for analytical workflows.
The stack below uses a single NVIDIA H100 80 GB, serves Qwen/Qwen3.6-27B-FP8 through vLLM, and points Omniscope at the resulting OpenAI-compatible endpoint.
It is deliberately not a polished infrastructure template. No Terraform. No Packer. No configuration management. No “enterprise architecture diagram”. Just the working setup.
If you already have an H100 machine, cloud or on-prem, you can skip straight to serving the model.
If you want to see how we run this internally on Google Cloud, there is a section near the bottom.
Why this matters
Omniscope’s AI features are not just asking a model to write text.
They involve planning, tool calling, querying data, generating outputs, and returning results back into Omniscope.
That means the model endpoint needs to do more than autocomplete a prompt.
It needs to handle the tool-calling format correctly.
It needs to return the result JSON shape Omniscope expects.
It needs to behave reliably across multi-step analytical flows.
So version pins matter.
This is one of those areas where “just use latest” is usually how you end up wasting a day.
Requirements
You need:
- Omniscope 2026.1, build b22470 or later
Earlier 2026.1 builds will fail tool-call result parsing against this stack. vLLM tightened the JSON shape it emits, and b22470 contains the corresponding Omniscope-side fix.
- A single NVIDIA H100 80 GB
SXM or PCIe is fine. Compute capability 9.0, Hopper.
The same model also runs on H200, B200, and MI300X with minor tweaks, but those are not covered here.
- NVIDIA driver 580 or newer on the host
This is required by vLLM’s default CUDA 13 image.
- Docker with the NVIDIA Container Toolkit installed and configured
For example:
nvidia-ctk runtime configure --runtime=docker- Around 150 GB free disk
This is for the model weights and the vLLM compile cache.
- Outbound network access to Hugging Face and Docker Hub for the first pull.
Why these versions
| Component | Pin | Why |
|---|---|---|
| Model | Qwen/Qwen3.6-27B-FP8 |
Dense FP8 W8A8, 262K native context. Strong all-rounder, reasoning on by default, official Qwen tool-call format supported by vLLM out of the box. Around 28 GB of weights, leaving plenty of headroom on an 80 GB H100 for KV cache. |
| Engine | vllm/vllm-openai:v0.20.1 |
First stable vLLM release with full Qwen3.6 and qwen3_coder tool-call parser support on the CUDA 13 default image. Avoids the v0.12.0 Hopper small-concurrency regression. |
| CUDA | 13.0, inside the container | Default tag of v0.20.1. Requires host driver 580+. A -cu129-ubuntu2404 variant exists if you are stuck on driver 525+. |
| Omniscope | 2026.1 b22470 | Fix for the tightened Chat Completions tool-call result JSON shape. |
Pin exact versions in production.
Do not use latest.
Serve the model
Pull the vLLM image and the model weights, then run vLLM as a container.
# One-time: pull the image
docker pull vllm/vllm-openai:v0.20.1
# One-time: pre-download weights to a host directory so container
# restarts don't re-fetch (~28 GB)
mkdir -p /opt/hf-cache
docker run --rm \
-v /opt/hf-cache:/root/.cache/huggingface \
vllm/vllm-openai:v0.20.1 \
python -c "from huggingface_hub import snapshot_download; \
snapshot_download('Qwen/Qwen3.6-27B-FP8')"
# Run the server
docker run -d \
--name vllm-server \
--restart unless-stopped \
--runtime nvidia \
--gpus all \
--ipc=host \
-p 8000:8000 \
-v /opt/hf-cache:/root/.cache/huggingface \
vllm/vllm-openai:v0.20.1 \
Qwen/Qwen3.6-27B-FP8 \
--port 8000 --host 0.0.0.0 \
--max-model-len 262144 \
--max-num-seqs 16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice \
--enable-prefix-caching \
--gpu-memory-utilization 0.92The first start takes a little while.
vLLM does CUDA graph compilation and Triton JIT, so expect around 60–120 seconds before the server starts accepting requests.
After that, restarts are much faster because the compiled artefacts are cached.
A few flag notes
--ipc=host
Required even with tensor parallelism set to 1. vLLM uses shared memory during multi-process initialisation and crashes without it.
--max-model-len 262144
This uses the full native 262K context.
You can push Qwen further with YaRN-extended 1M context, but it degrades short-context quality and is not needed for Omniscope.
--max-num-seqs 16
This caps concurrent requests.
16 is comfortable for Omniscope workloads on a single H100. You can raise it if you have many parallel users, but watch KV-cache pressure.
--reasoning-parser qwen3
Separates Qwen3.6’s thinking tokens from the visible response. Omniscope handles both branches.
--tool-call-parser qwen3_coder --enable-auto-tool-choice
Uses the official Qwen tool-calling format.
--enable-prefix-caching
Useful for Omniscope because several AI features repeatedly use large system prompts. Prefix caching avoids paying the full cost every time.
Sanity check
Once the container is running, check the model endpoint:
curl http://localhost:8000/v1/modelsThen test a basic chat completion:
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen/Qwen3.6-27B-FP8",
"messages": [{"role": "user", "content": "Say hello in one word."}]
}'Qwen3.6 has thinking mode on by default.
For simple lookups this adds latency. If you want to disable it at server level, add this to the Docker command:
--default-chat-template-kwargs '{"enable_thinking": false}'Omniscope can also disable thinking per request, so leaving it enabled on the server is usually fine.
Point Omniscope at it
If you are new to AI configuration in Omniscope, see:
In Omniscope 2026.1 b22470 or later, open:
Admin → AI Settings
Add a Custom provider.
The only thing you need to fill in is the endpoint base URL:
http://<host>:8000/Do not add /v1.
Omniscope appends the OpenAI path components itself.
vLLM exposes an OpenAI-compatible API on this endpoint, so there is no Omniscope-side adapter needed.
An API key is not required unless you started vLLM with --api-key.
By default, we do not do that. We gate access at the network layer instead.
Then expand each Integration section where you want AI to be available, e.g. Report Ninja, Insight Explored, Workflow Executions, etc.
Make sure a Default model is selected. This is important.
The integration will not activate unless a default model is set.
If you are configuring AI for the first time, the simplest thing is to select the Qwen model as the default for each integration.
If you already have another provider configured, such as OpenAI or OpenRouter, you can leave your defaults as they are and choose the Qwen model from the per-use model picker instead.
Every AI integration exposes a model picker in the UI.
An Enterprise licence is required to add Custom providers.
What we tested
This configuration has been verified end-to-end against Omniscope 2026.1 b22470 with the following workloads.
Insight Explorer
A single multi-step question against the Beverage Distribution demo.
This joined and summarised across multiple tables, from the initial prompt through to a fully rendered answer.
Instant Dashboard
Full dashboard generation against the EU Water Quality demo.
This covered schema inspection, block selection, and layout generation.
AI Insights block
Several AI Insights requests over the EU Water Quality demo.
This exercised parallel tool calls and the result aggregation path.
All three completed successfully with sensible output.
Tool calling, reasoning split, and the tightened result JSON shape all behaved correctly with the qwen3_coder parser.
Operational notes
The main cost is not the container. The main cost is the GPU.
The container is cheap to keep running. The H100 is not. Stop the VM when it is not in use.
The first request after a cold container restart will pay the 60–120 second compile/JIT cost again unless you persist ~/.cache/vllm.
Switching models means restarting the container.
Stop the current container and start a new one with a different model argument. The weights cache in /opt/hf-cache persists.
Do not run this on GCP Container-Optimized OS.
The vLLM image’s libcuda.so.1 paths do not match. Use Ubuntu 22.04 or later on the host.
Tensor parallelism is unnecessary for this model size on a single H100, so it is not covered here.
If you do scale up, FP8 weights are not always cleanly divisible across TP sizes. For MoE variants, prefer --enable-expert-parallel.
Known issue
Requests with no user message error out.
If Omniscope submits a conversation containing only a system message, and no user turn, vLLM rejects it.
This is rare in normal use.
You are likely to hit it only in two situations:
- Insight Explorer, if you explicitly configure the initial behaviour as a system message only, with no seeded user prompt.
- Report Ninja, if you erase the chat and then collapse and re-expand the chat sidebar.
The workaround is simple.
Always include at least one user message.
The Omniscope defaults already do this, so only the configurations above trigger the issue.
How we run it on GCP
Internally, we deploy this stack on Google Cloud using Packer and Terraform.
Packer bakes a VM image with the engine, weights, and systemd units pre-staged.
Terraform manages the VM and disks.
A few concrete details:
- Machine type:
a3-highgpu-1g
This gives you 1× H100 80 GB SXM.
On GCP, the single-GPU variant is Spot only. On-demand is gated to multi-GPU A3 configurations.
- Base image:
ubuntu-accelerator-2204-amd64-with-nvidia-580
This comes from the ubuntu-os-accelerator-images project.
It is Ubuntu 22.04 LTS with the NVIDIA 580 driver pre-installed.
The CUDA toolkit is not needed on the host because the vLLM container ships its own CUDA 13 runtime.
- Running cost on Spot: roughly $2.00–2.50 per hour all-in
This includes the VM, boot disk, and local SSDs.
At around $2/hour, an H100 left running all month is about $1,500/month.
So configure auto-shutdown.
On GCP you can do this with a simple cron-based health check that stops the VM if there have been no successful requests for a while.
You can also use instance schedules under:
Compute Engine → Instance schedules
Or set maxRunDuration on the instance to cap the longest single run.
The important thing is not the exact mechanism.
The important thing is not leaving an H100 running because somebody forgot to stop a test VM.
Alternatives
If you want to swap the model without changing the rest of the stack, two alternatives have been validated on the same vLLM v0.20.1 image.
| Model | Strengths | Notes |
Qwen/Qwen3.6-35B-A3B-FP8 |
MoE, 3B active, fast | Same flags as 27B. Tool calling is slightly less reliable on long agentic chains. |
openai/gpt-oss-120b |
Strong reasoning | MXFP4, 131K context. Different flags: drop --tool-call-parser, because Harmony is built in. Set --gpu-memory-utilization 0.95 --max-num-batched-tokens 1024 to avoid OOM at startup. |
For now, Qwen/Qwen3.6-27B-FP8 is the recommended default.
It gives the best balance of quality, speed, context length, and tool-calling reliability on a single H100 for Omniscope’s mix of workloads.
Final note
This is not the only way to run local AI with Omniscope. It is just a working one. And that is useful.
Because once you move from “the model can answer questions” to “the model can operate real analytical workflows”, the boring details start to matter.
Versions matter. Tool-call parsing matters. Result JSON matters. Context length matters. Driver versions matter. GPU memory matters. Cold start matters. Cost control matters.
That is the difference between a local model demo and a local model actually being usable inside an analytics product.
No Comments