HyT Capital Portfolio | Sand.ai Open Sources MAGI-2 Preview — the World's First 100B+ Open-Source MoE Video Generation Model

01 Unified Audio-Video — Let the Shot Emerge from Within the Model

In the videos above, both sound and visuals come from a single generation.

When a person speaks, the model generates corresponding lip movements, facial expressions, and body motions. When a collision sound comes from the environment, the objects in the frame react at the same moment. When the music tempo changes, the character's performance and camera movement also shift accordingly.

These subtle synchronizations determine whether a shot becomes a complete performance.

Traditional multi-stream approaches typically use separate backbone networks for visuals and audio, then connect them through cross-attention or post-processing. Audio and video are modeled separately in the early stages of generation, making alignment more dependent on later connections. The longer the pipeline, the more latency and error accumulation become issues.

To address this, MAGI-2 Preview continues the single-stream architecture we proposed in the paper Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman). Text, video, and audio enter the same context and, within the same Transformer, continuously exchange information through self-attention at every layer.

The model also incorporates shared experts and modality-specific experts. Shared experts handle commonalities across the three modalities, while modality-specific experts handle information unique to text, video, and audio respectively. Language understanding, character appearance, motion timing, voice, and environmental sound can all work together within the same shot.

The unified backbone also shortens the generation pipeline and reduces interface, waiting, and maintenance costs associated with chaining multiple models together. For advertising, short dramas, animation, and game content, a single generation gets closer to a complete shot that includes visuals, sound, and performance simultaneously.

With audio and video in the same model, computational pressure increases. A video is broken into a large number of spatial patches and consecutive frames, with audio and text added on top — sequence length and information volume grow rapidly. How to keep a 114B unified audio-video model running efficiently becomes the next question.

02 10-Second 1080P Video, Inference Cost Just RMB 0.5

Based on current monthly rental rates for 8-card H100s, the inference cost for generating a 10-second 1080P video using the distilled version of MAGI-2 Preview is approximately RMB 0.5.

The difference between a few dimes and a few yuan may look like just a decimal point on paper. But in real-world content creation, it changes the cost structure and the way content is produced.

Video generation rarely meets expectations on the first try. A character's movement, a camera turn, or even the pause in a line of dialogue can affect the final result. When the cost per attempt comes down, the same budget can be used to test more prompts, adjust motion, composition, and style, and then select usable shots from multiple versions.

The creative workflow will gradually shift toward multi-version generation, automated filtering, and human selection. For content tools, advertising systems, game asset generation, and virtual human interactions, lower cost also means the model can be called continuously — video generation begins to meet the conditions for high-frequency application.

Behind these cost numbers is MAGI-2 Preview's MoE architecture: approximately 114 billion total parameters, with only about 6 billion activated per forward pass.

03 114B Total Parameters, Only 6B Activated Per Pass

As video generation moves from short clips and silent footage to longer, more complex unified audio-video content, models need to handle character and scene consistency, motion and camera changes, dialogue, environmental sound, and pacing simultaneously. The boundaries of capability continue to expand — and so does the required model capacity. When scaling up dense models, every token passes through all parameters. The larger the model, the more compute is needed for both training and inference.

Video amplifies this problem further. Text models typically handle a few hundred or thousand tokens. Video, on the other hand, must accommodate consecutive frames, a large number of spatial patches, and audio information. As sequence length and modality complexity increase together, compute and cross-device communication overhead are quickly pushed up.

MoE sets up a large pool of experts within the model. Each token selects only a small subset of experts relevant to the current task to participate in computation. This allows the model to continue expanding its capacity while controlling the actual number of activated parameters.

MAGI-2 Preview uses approximately 114B total parameters to encode richer information on characters, motion, camera, materials, semantics, voice, and environmental audio, while using only about 6B activated parameters to control the compute per generation.

To achieve finer expert specialization, MAGI-2 Preview splits the 3,072-dimensional hidden representation into 12 heads of 256 dimensions, and uses Multi-Head MoE across the 36-layer backbone.

Each head has 256 experts, and selects 6 of them per forward pass. At each layer, this forms 3,072 head-local expert units. A single token activates a total of 72 small experts across the 12 heads. We refer to this structure as Ultra-fine-grained MoE.

Traditional MoE typically routes based on the full hidden representation of a token, requiring a small number of experts to handle multiple features simultaneously. Multi-Head MoE first splits the representation into multiple low-dimensional subspaces, with each head independently selecting its own experts.

Different heads thus have the opportunity to form finer division of labor — handling character appearance, motion timing, linguistic semantics, voice, or environmental audio separately. The outputs from the 12 heads are later fused back together in subsequent network layers. This approach not only enables richer combinations of experts, but also keeps each expert's scope of responsibility narrow.

As the number of experts increases, a new challenge emerges: these experts are distributed across different devices. How should tokens be routed to them?

04 For MoE in Video Generation Models, the Bottleneck Lies in Communication

Traditional Expert Parallel first selects the Top-K experts for each token, then sends the token to the devices where those experts reside.

Routing results change dynamically with the input. One GPU may suddenly receive a large number of tokens while another sits idle. The more experts are activated, the more data typically needs to be transferred across devices. At video-level sequence scales, communication and load imbalance can quickly become bottlenecks.

MAGI-2 Preview draws inspiration from Multi-Head Latent MoE and Head Parallel, introducing Head Parallel to change the order of communication.

The model first splits the hidden representation into multiple heads. Before dynamic routing takes place, it distributes the fixed-shape head representations to different devices. Once the data arrives at each device, experts are selected and executed locally.

In traditional Expert Parallel, the communication volume depends on the Top-K routing results produced each time. In Head Parallel, the primary communication volume is determined by the input representation itself, preventing communication scale from growing linearly with the number of activated experts. This change in sequence leaves more room for fine-grained experts. The model can accommodate more experts, improve routing precision, and keep cross-device traffic relatively stable.

Architecture, however, is only half the picture. The other half depends on the system to actually run it.

05 Behind the 100B+ Parameters Lies a Self-Developed Infra Stack

Multi-Head MoE introduces new demands on the infrastructure. Thousands of fine-grained experts lead to frequent routing, data rearrangement, and small-matrix computations. If built on a generic MoE system, the added model capacity can easily be offset by communication and scheduling overhead.

To address this, we developed MagiMoE, Head Parallel, and the distributed optimizer MagiMuon for MAGI-2 Preview — covering expert computation, cross-node communication, and stable training for 100B-parameter models.

MagiMoE is our self-developed high-performance MoE kernel library. It fuses expert routing, sorting, and computation into a single execution flow, reducing intermediate results and memory movement. It also optimizes forward and backward passes for the large number of small-matrix computations in Multi-Head MoE. The library also supports different kernel backends and low-precision paths, leaving room for future hardware adaptation and model iteration.

Head Parallel handles cross-node communication. Tensor parallelism, context parallelism, expert parallelism, and parameter sharding together handle the scaling of the backbone network and the 100B+ parameters. Multiple parallel strategies need to work together within the same system — if any one link has to wait, a large number of GPUs can sit idle.

On the training optimization front, the distributed version of MagiMuon uses Muon for the main matrix parameters and AdamW for parameters better suited to conventional updates. It also re-adapts parameter organization and cross-device computation for the large number of fine-grained experts. This hybrid design allows different types of parameters to be updated in their respective ways, and enables the optimization approach to scale stably from small-to-medium experiments up to 100B-parameter MoE training.

MagiMoE, Head Parallel, and MagiMuon together form the infrastructure of MAGI-2 Preview. Scaling video models has never been just about adding parameters — behind it, the harder things to scale are communication efficiency, compute efficiency, numerical stability, and training monitoring.

The value of MAGI-2 Preview lies in the fact that it validates both model design and system capability in a single large-scale training run.

06 From "Cleaning-First" to "Coverage-First" — Scaling Requires a New Approach to Data

As model scale grows, the data pipeline must evolve as well. In the past, the industry often understood data quality as a process of layered filtration: removing lower-quality footage, less structured compositions, or less typical motion, leaving behind a cleaner dataset.

Layered filtration can reduce noise and improve training stability, but it may also remove complex motion, unusual camera angles, rare subjects, atypical sounds, and long-tail combinations. The diversity of the real world is often hidden precisely in this "less standard" data.

So in training MAGI-2 Preview, we placed greater emphasis on data scale, diversity, and distribution coverage. The focus of data work shifted from subtraction to organization — using more accurate and finer-grained annotations to help the model understand subjects, motion, scenes, camera work, timing, and the relationships between sound, visuals, and text.

Pre-training covers the data distribution as completely as possible. Post-training focuses on preference, safety, controllability, and product adaptation. The more thoroughly the foundation model learns, the less pressure there is on the product side to patch motion consistency, visual detail, and audio-visual alignment through additional modules.

For video models, scaling is a co-expansion of model, data, and system. If any part stalls, it becomes difficult for parameter size alone to translate reliably into generative capability.

MAGI-2 Preview validates a clear technical path: using a unified single-stream architecture to generate visuals and audio together, using Ultra-fine-grained MoE to expand model capacity, and using Head Parallel and self-developed infrastructure to control communication and compute costs. Going forward, we will continue to scale up both model and data, explore longer-duration video generation, further improve generation quality, audio-visual synchronization, and long-term consistency, while continuing to reduce the per-unit generation cost.

Back Next article