vLLM

Route CortexDB's answer and extraction LLM calls through a self-hosted vLLM server.

vLLM serves any Hugging Face model behind an OpenAI-compatible endpoint, so CortexDB can keep all inference on your own hardware.

Set both the extraction and answer lanes

CORTEX_LLM_* is the entity-extraction lane; /v1/answer uses the separate CORTEX_ANSWER_* lane (default anthropic/claude-opus-4-6). Set both. See Self-hosting defaults.

Start vLLM

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --port 8000 \
  --api-key local-secret

Deployment configuration

# Extraction lane
CORTEX_LLM_URL=http://localhost:8000/v1
CORTEX_LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct
CORTEX_ENTITY_API_KEY=local-secret            # matches --api-key; must be non-empty even if vLLM has none

# Answer lane (OpenAI-compatible)
CORTEX_ANSWER_PROVIDER=openai
CORTEX_ANSWER_URL=http://localhost:8000/v1
CORTEX_ANSWER_MODEL=meta-llama/Llama-3.3-70B-Instruct
CORTEX_ANSWER_API_KEY=local-secret

Keep CORTEX_EMBEDDING_* on a local Ollama or an embedding-serving vLLM instance. See Embeddings.

The answer model is the one in CORTEX_ANSWER_MODEL. A per-request answer_model on /v1/answer is only echoed in diagnostics.answer_model; the configured model answers. To serve a different model, change CORTEX_ANSWER_MODEL or run a second deployment. See LLM & Answer Generation.

Start vLLM

Deployment configuration

Choosing the model

See also

On this page