Mistral Medium 3 vs Mistral Small 3.1 (25.03)

Hey all. Just set up a workstation with two NVIDIA RTX PRO 6000 Blackwells (96GB VRAM each) for our design studio. Want to use Ollama as our main local inference layer.

**What we want to do with it:**

1. Internal copilot for a \~60 person team. research, writing, brief analysis, code assist
2. Backend for agentic tools we're building (API access is a big reason we picked Ollama)
3. Run the biggest, best models our hardware can handle

**Specific questions:**

* How well does Ollama handle dual GPU setups out of the box? Any config needed for tensor parallelism across both cards?
* What models would you recommend at this VRAM level? Thinking Llama 3.1 70B unquantized, maybe even 405B at Q4?
* Anyone serving Ollama to a team via Open WebUI or similar? How's the experience at 10-15 concurrent users?
* Any gotchas with large model loading times or memory management I should know about?

First time running Ollama beyond hobby experiments, so any production-ish tips are appreciated. Will report back with what works.

\------

UPDATE FOR OTHERS & THANKS FOR THE HELP . THIS SUB WASN'T AS SNARKY AND IN FACT A LOT MORE HELPFUL THAN THE OTHER ONE.

For context: we're a design agency rendering 3D animations, VR/AR walkthroughs, and architectural visualizations. Not generating AI images or running Stable Diffusion farms. The dual RTX Pro 6000s (96 GB VRAM each) are a dedicated render node that processes overnight animation batches and path-traced scenes while our design team stays productive on their own workstations. Cloud rendering costs add up absurdly fast at our project volume. Owning the hardware pays for itself in months. OctaneRender and Redshift scale linearly across both GPUs, which turns 12+ hour VR renders into something we can actually deliver on client deadlines.

# Key Technical Advice & Actionables

# Infrastructure Stack (Overwhelming Consensus)

**Switch from Ollama to vLLM or llama.cpp**

* **169 upvotes** on "Tip #1 don't use Ollama"
* **109 upvotes** on criticism of using Ollama with $25k hardware
* vLLM is the top recommendation for multi-user concurrency (your 10-15 concurrent users scenario)
* llama.cpp is acceptable for single-user or simpler setups, but vLLM wins for parallelization

**Use Linux instead of Windows**

* **266 upvotes** on "Tip #2 use Linux"
* Ubuntu LTS 24.04 most recommended for NVIDIA driver support
* Debian headless for maximum resource efficiency
* Debate exists: some claim Windows CUDA drivers are 2-3% faster for pure VRAM inference, but Linux wins for stability and virtual memory handling

# Model Recommendations

**Stop using Llama 3.1 70B** (described as "ancient" and "severely outdated")

* **Minimax M2.7 (230B MoE, 10B active)** with NVFP4 quantization — perfect fit for your dual 96GB setup
* **Qwen 3.5/3.6 series** (27B, 35B MoE, 122B) — excellent dense models, great for agentic tasks
* **Gemma 4** — recommended if you need "western" models (some companies ban Chinese models)
* **Mistral Medium 3.5 (119B MoE)** or new **Mistral 128B dense** — good for massive context windows

# Critical Configuration Settings

**Use Tensor Parallelism (tp=2)**

* Splits model across both GPUs for unified inference
* Doubles speed and allows models up to \~180-190GB total
* Essential command: `--tp 2` in vLLM or llama.cpp

**Use NVFP4 Quantization**

* Hardware-accelerated 4-bit format specifically for Blackwell architecture
* Minimax M2.7 NVFP4 fits in 130.6GB (down from 230GB)
* Multiple users emphasized this is purpose-built for your cards

**Optimize for Concurrency**

* Use **litellm** as a model router in front of vLLM for rate limiting and monitoring
* Set `--gpu-memory-utilization 0.9` or higher to maximize KV cache
* **SGLang** recommended over vLLM if team works on same projects (prefix caching with RadixAttention)
* For 60-person team: expect 5-8 simultaneous users per card on 70B Q4 before throughput drops

# System Architecture

**Cooling & Power Management**

* GPU spacing: minimum 2 slots apart for adequate airflow
* Consider power limiting cards to reduce heat and increase stability
* Script fixed clock times (10MHz below stock) to prevent PCIe bus spikes
* Heat management is critical for sustained inference loads

**RAM Requirements**

* Minimum 256GB system RAM
* Recommendation: **2× VRAM = 384-512GB system RAM** for optimal performance
* Essential for virtual memory handling during large context operations

**Frontend & User Access**

* **Open WebUI** is acceptable for team deployment (contrary to one dismissive comment)
* Alternative: Set up **litellm** for monitoring, rate limiting, API key generation
* Some debate about OpenWebUI in 2026, but no clear superior alternative mentioned for your use case

# Specific Guides & Resources Mentioned

1. **vLLM Blackwell guide**: [https://github.com/lastloop-ai/vllm-blackwell-guide](https://github.com/lastloop-ai/vllm-blackwell-guide) (120+ t/s on Qwen 27B, 200+ t/s on 35B MoE)
2. **Ollama agent configs**: [https://github.com/caliber-ai-org/ai-setup](https://github.com/caliber-ai-org/ai-setup) (888 stars, production patterns for team deployment)
3. **llama-swap** tool for dynamic model switching without container restarts

# Hiring & Operational Advice

**Top upvoted wisdom** (113+ votes on original thread you referenced): "Storage, model management, permissions, and user access become more important than the GPUs after week one. Hire someone experienced with this stack."

[https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF](https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF)

# Mistral Medium 3.5 128B

Mistral Medium 3.5 is our first flagship merged model. It is a dense 128B model with a 256k context window, handling instruction-following, reasoning, and coding in a single set of weights. Mistral Medium 3.5 replaces its predecessor Mistral Medium 3.1 and Magistral in Le Chat. It also replaces Devstral 2 in our coding agent Vibe. Concretely, expect better performance for instruct, reasoning and coding tasks in a new unified model in comparison with our previous released models.

Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a complex agentic run. We trained the vision encoder from scratch to handle variable image sizes and aspect ratios.

Find more information on our [blog](https://mistral.ai/news/vibe-remote-agents-mistral-medium-3-5).

# Key Features

Mistral Medium 3.5 includes the following architectural choices:

* **Dense 128B parameters**.
* **256k context length**.
* **Multimodal input**: Accepts both text and image input, with text output.
* **Instruct and Reasoning functionalities** with function calls (reasoning effort configurable per request).

Mistral Medium 3.5 offers the following capabilities:

* **Reasoning Mode**: Toggle between fast instant reply mode and reasoning mode, boosting performance with test-time compute when requested.
* **Vision**: Analyzes images and provides insights based on visual content, in addition to text.
* **Multilingual**: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, and Arabic.
* **System Prompt**: Strong adherence and support for system prompts.
* **Agentic**: Best-in-class agentic capabilities with native function calling and JSON output.
* **Large Context Window**: Supports a 256k context window.

We release this model under a [**Modified MIT License**](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B/blob/main/(https://huggingface.co/mistralai/mistralai/Mistral-Medium-3.5-128B/blob/main/LICENSE)): Open-source license for both commercial and non-commercial use with exceptions for companies with large revenue.

# Recommended Settings

* **Reasoning Effort**:
* `'none'` → Do not use reasoning
* `'high'` → Use reasoning (recommended for complex prompts and agentic usage) Use `reasoning_effort="high"` for complex tasks and agentic coding.
* **Temperature**: 0.7 for `reasoning_effort="high"`. Temp between 0.0 and 0.7 for `reasoning_effort="none"` depending on the task. Generally, lower means answer that are more to the point and higher allows the model to be more creative. It is a good practice to try different values in order to improve the model performance to meet your demands.

Mistral releases Mistral Medium 3.5, a new vision reasoning model. 🔥

Mistral-Medium-3.5-128B offers highly competitive performance for models 5x its size.

Hey guys, we worked with Mistral to fix Mistral Medium 3.5 inference affecting some implementations, and released updated GGUFs with the fix (NOT related to Unsloth or quants). Mistral 3.5 now works properly in transformers AND llama.cpp.

The issue was caused by a YaRN parsing quirk affecting some implementations. Changing mscale\_all\_dim from 1 to 0 resolved it. We also fixed mmproj files generation.

Mistral has pushed our fixes to their official repo. The YaRN scaling multiplier is applied correctly, fixing forgetting previous conversations.

Guide: [https://unsloth.ai/docs/models/mistral-3.5](https://unsloth.ai/docs/models/mistral-3.5)

GGUFs: [https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF](https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF)

• Mistral Medium 3.1 is live on Le Chat or through their API library