Infrastructure and sovereignty · Layer 4

What is a mixture-of-experts model?

Mixture of experts (MoE)

A model split into blocks, only a few of which work on each word: it computes less, but must keep everything in memory.

A way to picture itPicture a large kitchen where, for each plate, a dispatcher lights only two burners out of eight: less gas is used, but you still need the room and the equipment for all eight.

The diagram · Mixture of experts

01A token arrivesa piece of a word

02The router choosesfor example 2 experts out of 8

03Experts computethe others wait, loaded in memory

04Outputs combinedweighted by the router

05Next layera fresh choice of experts

How it works

01In a standard model, every parameter works on every piece of a word. In a mixture of experts, part of the model is split into blocks called “experts”.
02At each layer, a small dispatcher, the router, picks a few experts for that piece of a word. Mixtral 8x7B (France · Mistral AI) picks 2 out of 8: it has 46.7 billion parameters, of which 12.9 billion are active for each piece of a word.
03Computation drops, memory does not: every expert stays loaded, because the next piece of a word may need any of them. DeepSeek-V3 (China · DeepSeek) announces 37 billion active parameters, yet 671 billion must be loaded.

In a business function

IT procurement

An IT department receives two quotes to host a model in-house. The first supplier highlights “only 37 billion active parameters” and sizes the server as if the model were that big. The buyer asks for the total size: several hundred billion parameters, all of which must be loaded into memory. The quote is redone on total memory and on a load test with the real number of simultaneous users.

The compliance angle

The architecture does not change your obligations, but it changes the questions to ask. A provider of a general-purpose model is in principle required to document its architecture and number of parameters (EU · AI Act, art. 53 and Annex XI): ask for both total and active size, never one without the other. A server sized on active parameters alone risks saturation, and availability is part of security (EU · GDPR, art. 32). As with any model, every new version must be re-evaluated before going live.

Limits to know

“Active parameters” predicts neither the memory required nor the actual speed: with many simultaneous users, almost every expert ends up working at once.
When experts are spread across several graphics cards, the traffic between them can slow responses down.
Experts are not specialists: you cannot “switch off the legal expert” or check that a topic goes through the right block.

The question to ask your supplier

“What is the model’s total size, how much memory does it take to load, and what throughput have you measured with our number of simultaneous users?”

Submit a brief

Where this concept sits

01Business questions
02Definitions and owners
03Connected data
04AI
05Rights and human sign-off
06Evidence

Related concepts

Frequently asked questions

Does each expert specialise in a field, such as law or finance?

No. The router learns on its own how to distribute pieces of words, and that split does not follow business domains. In the published analysis of Mixtral (France · Mistral AI), expert choice follows the form of the text, such as code indentation, more than its subject.

Is a mixture-of-experts model cheaper?

It computes less per word, which can lower the cost of each answer. But it takes as much memory as its total size, and its real cost depends on the hardware and the number of simultaneous users: only a test on your own load will tell.

How can you tell whether a model is one?

Its model card then gives two figures: total parameters and active parameters. A name like “8x7B” is a clue, but it is not a multiplication: Mixtral 8x7B has 46.7 billion parameters, not 56, because part of the model is shared by all the experts.

Educational explainer, not legal advice. Up to date as of 29 October 2026.

What is a mixture-of-experts model?

Mixture of experts (MoE)

A model split into blocks, only a few of which work on each word: it computes less, but must keep everything in memory.

A way to picture itPicture a large kitchen where, for each plate, a dispatcher lights only two burners out of eight: less gas is used, but you still need the room and the equipment for all eight.

How it works

01In a standard model, every parameter works on every piece of a word. In a mixture of experts, part of the model is split into blocks called “experts”.

02At each layer, a small dispatcher, the router, picks a few experts for that piece of a word. Mixtral 8x7B (France · Mistral AI) picks 2 out of 8: it has 46.7 billion parameters, of which 12.9 billion are active for each piece of a word.

03Computation drops, memory does not: every expert stays loaded, because the next piece of a word may need any of them. DeepSeek-V3 (China · DeepSeek) announces 37 billion active parameters, yet 671 billion must be loaded.