addictedtoAI
menu

technique · technique/mixture-of-experts

Mixture of Experts

activestable

Also called Sparse Mixture of Experts, MoE

Facts

mixtral routing
8 feedforward experts per layer, 2 selected per token, chosen again at every layer; 47B total parameters with 13B active per tokensource, accessed 2026-08-28
deepseek v3 scale
671B total parameters with 37B activated per tokensource, accessed 2026-08-28
deepseek v3 load balancing
auxiliary-loss-free load balancing, plus a multi-token prediction training objectivesource, accessed 2026-08-28
deepseek v3 training cost
2.788M H800 GPU hours for the full training run, with no irrecoverable loss spikes or rollbacks reportedsource, accessed 2026-08-28

Timeline

  1. DeepSeek-V3 technical report: auxiliary-loss-free load balancing and multi-token prediction at 671B parameterssource
  2. Mixtral 8x7B published under Apache 2.0, with base and instruction-tuned weights releasedsource