Kimi K2.6 vs Mistral Medium 3

Time to switch to Kimi k2.6 guys if you haven't already.

For $20 a month you can buy the OpenCode Go coding plan (its actually $5 for the first month then $10) which gives you many more tokens on models like Kimi K2.6, and then you can pay for the rest of the usage. So for $20 a month of tokens of Kimi K2.6 you're basically getting the equivalent amount of tokens of the $100 plan.

You can also use Qwen 3.6 35B A3B, which you can run on your local PC (as long as you have a decent graphics card).

After testing it and getting some customer feedback too, its the first model I'd confidently recommend to our customers as an Opus 4.7 replacement.

It's not really better than Opus 4.7 at anything, but, it can do about 85% of the tasks that Opus can at a reasonable quality, and, it has vision and very good browser use.

I've been slowly replacing some of my personal workflows with Kimi K2.6 and it works surprisingly well, especially for long time horizon tasks.

Sure the model is monstrously big, but I think it shows that frontier LLMs like Opus 4.7 are not necessarily bringing anything new to the table. People are complaining about usage limits as well, it looks like local is the way to go.

Hey all. Just set up a workstation with two NVIDIA RTX PRO 6000 Blackwells (96GB VRAM each) for our design studio. Want to use Ollama as our main local inference layer.

**What we want to do with it:**

1. Internal copilot for a \~60 person team. research, writing, brief analysis, code assist
2. Backend for agentic tools we're building (API access is a big reason we picked Ollama)
3. Run the biggest, best models our hardware can handle

**Specific questions:**

* How well does Ollama handle dual GPU setups out of the box? Any config needed for tensor parallelism across both cards?
* What models would you recommend at this VRAM level? Thinking Llama 3.1 70B unquantized, maybe even 405B at Q4?
* Anyone serving Ollama to a team via Open WebUI or similar? How's the experience at 10-15 concurrent users?
* Any gotchas with large model loading times or memory management I should know about?

First time running Ollama beyond hobby experiments, so any production-ish tips are appreciated. Will report back with what works.

\------

UPDATE FOR OTHERS & THANKS FOR THE HELP . THIS SUB WASN'T AS SNARKY AND IN FACT A LOT MORE HELPFUL THAN THE OTHER ONE.

For context: we're a design agency rendering 3D animations, VR/AR walkthroughs, and architectural visualizations. Not generating AI images or running Stable Diffusion farms. The dual RTX Pro 6000s (96 GB VRAM each) are a dedicated render node that processes overnight animation batches and path-traced scenes while our design team stays productive on their own workstations. Cloud rendering costs add up absurdly fast at our project volume. Owning the hardware pays for itself in months. OctaneRender and Redshift scale linearly across both GPUs, which turns 12+ hour VR renders into something we can actually deliver on client deadlines.

# Key Technical Advice & Actionables

# Infrastructure Stack (Overwhelming Consensus)

**Switch from Ollama to vLLM or llama.cpp**

* **169 upvotes** on "Tip #1 don't use Ollama"
* **109 upvotes** on criticism of using Ollama with $25k hardware
* vLLM is the top recommendation for multi-user concurrency (your 10-15 concurrent users scenario)
* llama.cpp is acceptable for single-user or simpler setups, but vLLM wins for parallelization

**Use Linux instead of Windows**

* **266 upvotes** on "Tip #2 use Linux"
* Ubuntu LTS 24.04 most recommended for NVIDIA driver support
* Debian headless for maximum resource efficiency
* Debate exists: some claim Windows CUDA drivers are 2-3% faster for pure VRAM inference, but Linux wins for stability and virtual memory handling

# Model Recommendations

**Stop using Llama 3.1 70B** (described as "ancient" and "severely outdated")

* **Minimax M2.7 (230B MoE, 10B active)** with NVFP4 quantization — perfect fit for your dual 96GB setup
* **Qwen 3.5/3.6 series** (27B, 35B MoE, 122B) — excellent dense models, great for agentic tasks
* **Gemma 4** — recommended if you need "western" models (some companies ban Chinese models)
* **Mistral Medium 3.5 (119B MoE)** or new **Mistral 128B dense** — good for massive context windows

# Critical Configuration Settings

**Use Tensor Parallelism (tp=2)**

* Splits model across both GPUs for unified inference
* Doubles speed and allows models up to \~180-190GB total
* Essential command: `--tp 2` in vLLM or llama.cpp

**Use NVFP4 Quantization**

* Hardware-accelerated 4-bit format specifically for Blackwell architecture
* Minimax M2.7 NVFP4 fits in 130.6GB (down from 230GB)
* Multiple users emphasized this is purpose-built for your cards

**Optimize for Concurrency**

* Use **litellm** as a model router in front of vLLM for rate limiting and monitoring
* Set `--gpu-memory-utilization 0.9` or higher to maximize KV cache
* **SGLang** recommended over vLLM if team works on same projects (prefix caching with RadixAttention)
* For 60-person team: expect 5-8 simultaneous users per card on 70B Q4 before throughput drops

# System Architecture

**Cooling & Power Management**

* GPU spacing: minimum 2 slots apart for adequate airflow
* Consider power limiting cards to reduce heat and increase stability
* Script fixed clock times (10MHz below stock) to prevent PCIe bus spikes
* Heat management is critical for sustained inference loads

**RAM Requirements**

* Minimum 256GB system RAM
* Recommendation: **2× VRAM = 384-512GB system RAM** for optimal performance
* Essential for virtual memory handling during large context operations

**Frontend & User Access**

* **Open WebUI** is acceptable for team deployment (contrary to one dismissive comment)
* Alternative: Set up **litellm** for monitoring, rate limiting, API key generation
* Some debate about OpenWebUI in 2026, but no clear superior alternative mentioned for your use case

# Specific Guides & Resources Mentioned

1. **vLLM Blackwell guide**: [https://github.com/lastloop-ai/vllm-blackwell-guide](https://github.com/lastloop-ai/vllm-blackwell-guide) (120+ t/s on Qwen 27B, 200+ t/s on 35B MoE)
2. **Ollama agent configs**: [https://github.com/caliber-ai-org/ai-setup](https://github.com/caliber-ai-org/ai-setup) (888 stars, production patterns for team deployment)
3. **llama-swap** tool for dynamic model switching without container restarts

# Hiring & Operational Advice

**Top upvoted wisdom** (113+ votes on original thread you referenced): "Storage, model management, permissions, and user access become more important than the GPUs after week one. Hire someone experienced with this stack."

[https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF](https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF)

# Mistral Medium 3.5 128B

Mistral Medium 3.5 is our first flagship merged model. It is a dense 128B model with a 256k context window, handling instruction-following, reasoning, and coding in a single set of weights. Mistral Medium 3.5 replaces its predecessor Mistral Medium 3.1 and Magistral in Le Chat. It also replaces Devstral 2 in our coding agent Vibe. Concretely, expect better performance for instruct, reasoning and coding tasks in a new unified model in comparison with our previous released models.

Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a complex agentic run. We trained the vision encoder from scratch to handle variable image sizes and aspect ratios.

Find more information on our [blog](https://mistral.ai/news/vibe-remote-agents-mistral-medium-3-5).

# Key Features

Mistral Medium 3.5 includes the following architectural choices:

* **Dense 128B parameters**.
* **256k context length**.
* **Multimodal input**: Accepts both text and image input, with text output.
* **Instruct and Reasoning functionalities** with function calls (reasoning effort configurable per request).

Mistral Medium 3.5 offers the following capabilities:

* **Reasoning Mode**: Toggle between fast instant reply mode and reasoning mode, boosting performance with test-time compute when requested.
* **Vision**: Analyzes images and provides insights based on visual content, in addition to text.
* **Multilingual**: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, and Arabic.
* **System Prompt**: Strong adherence and support for system prompts.
* **Agentic**: Best-in-class agentic capabilities with native function calling and JSON output.
* **Large Context Window**: Supports a 256k context window.

We release this model under a [**Modified MIT License**](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B/blob/main/(https://huggingface.co/mistralai/mistralai/Mistral-Medium-3.5-128B/blob/main/LICENSE)): Open-source license for both commercial and non-commercial use with exceptions for companies with large revenue.

# Recommended Settings

* **Reasoning Effort**:
* `'none'` → Do not use reasoning
* `'high'` → Use reasoning (recommended for complex prompts and agentic usage) Use `reasoning_effort="high"` for complex tasks and agentic coding.
* **Temperature**: 0.7 for `reasoning_effort="high"`. Temp between 0.0 and 0.7 for `reasoning_effort="none"` depending on the task. Generally, lower means answer that are more to the point and higher allows the model to be more creative. It is a good practice to try different values in order to improve the model performance to meet your demands.