Recommended model parameters
Recommended model master values for each supported Managed Inference model.
CosmicAC recommends these values for each supported model's model master. For the vLLM models, each inference_param_overrides.env row sets a vllm serve option. See vLLM serving options.
| Field | Value |
|---|
| job_type | INFERENCE_VLLM |
| base_os_image | Ubuntu 22.04 + CUDA 13.0 |
| disk_gb | 500 |
| cuda_driver_version | CUDA 13.0 |
| model_name | Qwen/Qwen3-VL-235B-A22B-Thinking-FP8 |
| runtime_image | vllm/vllm-openai:v0.15.1 |
| data_type | Auto |
| quantisation | null |
| tensor_parallel | 8 |
| per_replica_gpu_count | 8 |
| gpu_memory_utilisation | 0.9 |
| max_model_length | 131072 |
| max_concurrent_sequences | 64 |
| reasoning_parser | deepseek_r1 |
| multimodal | true |
| replica | 1 |
| require_auth_header | true |
| min | ["base_os_image"] |
| max | ["gpu_memory_utilisation"] |
| inference_param_overrides.root_disk_size_gb | 500 |
| inference_param_overrides.env.TRUST_REMOTE_CODE | true |
| inference_param_overrides.env.SWAP_SPACE | 0 |
| inference_param_overrides.env.ENABLE_EXPERT_PARALLEL | true |
| inference_param_overrides.env.ENFORCE_EAGER | false |
| Field | Value |
|---|
| job_type | INFERENCE_VLLM |
| base_os_image | Ubuntu 22.04 + CUDA 13.0 |
| disk_gb | 500 |
| cuda_driver_version | CUDA 13.0 |
| model_name | Qwen/Qwen3.5-122B-A10B |
| runtime_image | vllm/vllm-openai:v0.17.1 |
| data_type | Auto |
| quantisation | FP8 |
| tensor_parallel | 8 |
| per_replica_gpu_count | 8 |
| gpu_memory_utilisation | 0.9 |
| max_model_length | 32768 |
| max_concurrent_sequences | 32 |
| reasoning_parser | deepseek_r1 |
| multimodal | true |
| replica | 1 |
| require_auth_header | true |
| min | ["base_os_image"] |
| max | ["gpu_memory_utilisation"] |
| inference_param_overrides.root_disk_size_gb | 500 |
| inference_param_overrides.env.TRUST_REMOTE_CODE | true |
| inference_param_overrides.env.SWAP_SPACE | 0 |
| inference_param_overrides.env.ENABLE_EXPERT_PARALLEL | true |
| inference_param_overrides.env.ENFORCE_EAGER | false |
| Field | Value |
|---|
| job_type | INFERENCE_VLLM |
| base_os_image | Ubuntu 22.04 + CUDA 13.0 |
| disk_gb | 500 |
| cuda_driver_version | CUDA 13.0 |
| model_name | MiniMaxAI/MiniMax-M2.5 |
| runtime_image | vllm/vllm-openai:v0.15.1 |
| data_type | Auto |
| quantisation | null |
| tensor_parallel | 4 |
| per_replica_gpu_count | 4 |
| gpu_memory_utilisation | 0.85 |
| max_model_length | 27000 |
| max_concurrent_sequences | 256 |
| reasoning_parser | deepseek_r1 |
| multimodal | true |
| replica | 1 |
| require_auth_header | true |
| min | ["base_os_image"] |
| max | ["gpu_memory_utilisation"] |
| inference_param_overrides.root_disk_size_gb | 500 |
| inference_param_overrides.env.TRUST_REMOTE_CODE | true |
| inference_param_overrides.env.SWAP_SPACE | 0 |
| inference_param_overrides.env.ENABLE_EXPERT_PARALLEL | true |
| inference_param_overrides.env.ENFORCE_EAGER | false |
| Field | Value |
|---|
| job_type | INFERENCE_VLLM |
| base_os_image | Ubuntu 22.04 + CUDA 13.0 |
| disk_gb | 150 |
| cuda_driver_version | CUDA 13.0 |
| ssh_public_key | null |
| model_name | Qwen/Qwen2-VL-2B-Instruct |
| runtime_image | vllm/vllm-openai:v0.15.1 |
| data_type | Auto |
| quantisation | null |
| tensor_parallel | 1 |
| gpu_memory_utilisation | 0.9 |
| max_model_length | 32768 |
| max_concurrent_sequences | 64 |
| reasoning_parser | default |
| multimodal | true |
| replica | 1 |
| per_replica_gpu_count | 1 |
| require_auth_header | true |
| inference_param_overrides.root_disk_size_gb | 150 |
| inference_param_overrides.env.TRUST_REMOTE_CODE | true |
| inference_param_overrides.env.SWAP_SPACE | 0 |
| inference_param_overrides.env.ENFORCE_EAGER | false |
| min | ["base_os_image"] |
| max | ["gpu_memory_utilisation"] |
The Parakeet model master stores a different set of fields. Parakeet Managed Inference Job configuration defines each one. For the inference_param_overrides.env rows, see Parakeet server settings.
| Field | Value |
|---|
| job_type | INFERENCE_PARAKEET |
| model_name | nvidia/parakeet-tdt-0.6b-v3 |
| require_auth_header | false |
| chunk_duration | 30 |
| chunk_overlap | 5 |
| max_file_size_mb | 1024 |
| disk_gb | 250 |
| replica | 1 |
| inference_param_overrides.root_disk_size_gb | 150 |
| inference_param_overrides.env.MODEL_SOURCE | huggingface |
| inference_param_overrides.env.HF_HOME | /mnt/external-disk/huggingface |
| inference_param_overrides.env.NEMO_CACHE_DIR | /mnt/external-disk/nemo |
| inference_param_overrides.env.HOST | 0.0.0.0 |
| inference_param_overrides.env.INFERENCE_BACKEND | parakeet |
| inference_param_overrides.env.PORT | 8777 |
| inference_param_overrides.env.PARAKEET_STARTUP_TIMEOUT_MS | 600000 |
