vLLM Managed Inference Job configuration
Fields you set when you create a vLLM Managed Inference Job.
A vLLM Managed Inference Job takes the following fields, as well as the common fields and GPU configuration that every job takes.
| Field | Required | CLI flag | Description |
|---|---|---|---|
| Model | Yes | --model | The Hugging Face model ID to serve, for example MiniMaxAI/MiniMax-M2.5. |
| Runtime image (CUDA) | Yes | --runtime-image | The serving runtime image, as a Docker image reference such as vllm/vllm-openai:v0.15.1. The web interface and interactive cosmicac jobs create use the model master's image. In non-interactive mode, if you omit --runtime-image, the CLI sends vllm/vllm-openai:v0.17.1. |
| Data type | Yes | --data-type | The numeric precision the model runs at, for example BF16 or Auto. If you omit --data-type, the CLI sends Auto. |
| Quantisation | No | --quantisation | The quantisation scheme, for example fp8 or awq. |
| Tensor parallel | No | --tensor-parallel | The number of GPUs to split the model across. One of 1, 2, 4, 8, or none. See Tensor parallel. |
| GPU memory utilization | No | --gpu-memory-utilization | The fraction of GPU memory to use, from 0 to 1. |
| Max concurrent sequences | No | --max-concurrent-sequences | The maximum number of requests the model handles at once, from 1 to 512. |
| Max model length | No | --max-model-length | The maximum context length, in tokens. The web interface and interactive cosmicac jobs create use the model master's value. In non-interactive mode, if you omit --max-model-length, the CLI sends 32768. See max_tokens in the Inference API reference. |
| Reasoning parser | No | --reasoning-parser | The parser that separates a model's thinking from its final response. The value default means no parser. |
| Video & image input | No | --multimodal | Whether the model accepts multimodal input. true or false. |
| Root disk size | No | --root-disk-size-gb | The root disk size of the virtual machine, in GB. See Root disk size. |
| Environment variables | No | --env | Variables for the model server and the inference agent. A variable named after a vllm serve flag sets that option. See vLLM serving options. |
| Endpoint name | Yes | --endpoint-name | The name of the inference endpoint, which forms a path segment in its URL. The name has 3 to 63 characters, and uses only lowercase letters, digits, and hyphens. It must be unique across every Managed Inference Job in the deployment, including failed jobs that you haven't deleted. |
| Replicas | No | --replica | The number of endpoint replicas. One of 1, 2, or 4. |
| Require Authorization header | No | --require-auth-header / --no-auth-header | Whether callers must send an authorization header. true or false. |
Tensor parallel
The tensor parallel value must divide the job's GPU count exactly. Otherwise, CosmicAC rejects the job with a 400 status and the message tensor_parallel must evenly divide the GPU count per instance.
If a job has no tensor parallel value, CosmicAC uses the model's per_replica_gpu_count, listed in Recommended model parameters. The job's GPU count equals that number, so the value always divides it.
The web interface doesn't show the field, and sends the model master's per_replica_gpu_count. The CLI preselects the same value in interactive mode, and reads --tensor-parallel in non-interactive mode.
Root disk size
The minimum root disk size depends on the model. It's the model's disk_gb, listed in Recommended model parameters. If the value is smaller, CosmicAC rejects the job with the message root_disk_size_gb (X) is below model master disk_gb (Y).