Prime Mega MoE: Speeding Up Mixture-of-Experts Training by 55% with a New GPU KernelArticle · digest for November 2
Surreal still life with intersecting high-speed data streams and a tensor crystal.
Article4 sections≈ 7 min

Prime Mega MoE: Speeding Up Mixture-of-Experts Training by 55% with a New GPU Kernel

When tech companies rent compute clusters to train cutting-edge neural networks, the bill for electricity and server racks runs into millions of dollars a week. Thousands of Nvidia H100 and H200 GPUs hum in server halls, solving colossal matrix equations. Yet if you open a performance profiler on such a cluster, any engineer will be seized by sheer horror: those pricey GPU tensor cores sit idle nearly half the time!

The culprit behind these massive losses is the Mixture of Experts (MoE) architecture, which has become the standard for modern models like Qwen 2.5, Qwen 3, and DeepSeek-V3. To route tokens to the right "experts," GPUs have to continuously ship terabytes of data across the inter-node network, stalling compute units.

The team behind the distributed training platform Prime Intellect has rolled out an open-source fix—the Prime Mega MoE GPU kernel, published in the prime-kernels repository under the Apache 2.0 license. By overlapping network communication and math computations on the backward pass (Overlapping Backward Pass), the engineers sped up the MoE training step by a phenomenal 55%. Let's unpack the low-level mechanics of this kernel and see how the magic works in code.

Anatomy of GPU Idle Time: Where Mixture-of-Experts Stumbles

Let's break down how a classic MoE layer works in plain terms. Unlike a dense neural network, where every token passes through one giant fully connected matrix (FFN), an MoE layer splits that block into dozens of independent sub-blocks—"experts." For each incoming word, a dynamic router picks the two to four most suitable experts.

As long as all experts live on a single physical GPU, everything runs fast. Once a model reaches hundreds of billions of parameters, though, different experts inevitably end up on different servers in the cluster. And that is where the nightmare begins:

  1. All-to-All operation on the Forward Pass: GPUs must exchange tokens to route each word to whichever GPU holds the needed expert;
  2. Matrix multiplication (GEMM): experts process tokens on the tensor cores;
  3. Backward Pass: when computing error gradients, tokens have to be shipped back to their source servers while simultaneously computing weight gradients and activation gradients.

In standard implementations (including Meta's popular TorchTitan framework), these operations run sequentially: first the GPU waits for the network adapter (InfiniBand or RoCE) to finish transferring a batch of data, and only then starts spinning the tensor cores. As a result, the GPU sits idle for 40–50% of every iteration step.

Overlapping Backward Pass: Merging Network I/O and Matrix Multiplications

Prime Intellect's engineers noticed a fundamental property of the backward pass. Computing activation gradients to send back over the network requires the experts' weights, but it does not require the gradients of those weights! That means the computation can be split into two independent execution streams.

Built on top of the low-level DeepGEMM library, the Prime Mega MoE kernel implements asynchronous overlapping:

  • While the GPU computes weight gradients for the current expert on the tensor cores, the communication module in a background CUDA stream is already shipping finished activation gradients over the network to neighboring nodes;
  • And vice versa: receiving incoming packets for the next expert happens in parallel with computing matrix blocks for the current step.

Here is what the conceptual logic of splitting streams and driving the Tensor Memory Accelerator (TMA) looks like in low-level CUDA C++ code:

#include <cuda_runtime.h>
#include <cooperative_groups.h>

namespace cg = cooperative_groups;

// Asynchronous backward pass kernel overlapping compute and network exchange
__global__ void prime_moe_overlapping_backward_kernel(
    const half* __restrict__ grad_output,
    const half* __restrict__ weights,
    half* __restrict__ grad_input_buffer,
    half* __restrict__ grad_weight_accumulator,
    int num_tokens, int hidden_dim, int expert_dim
) {
    cg::thread_block block = cg::this_thread_block();
    
    // Asynchronous shared memory allocation for TMA tiles
    extern __shared__ half smem_tile[];
    half* tile_a = smem_tile;
    half* tile_b = smem_tile + (hidden_dim * 16);

    // 1. Asynchronous computation of input (activation) gradients
    #pragma unroll
    for (int step = 0; step < expert_dim; step += 16) {
        // Load weights via Hopper TMA asynchronous memory copy
        asm volatile("cp.async.bulk.tensor.2d.shared::cluster.global.mbarrier::complete_tx::bytes"
                     " [%0], [%1], [%2];" 
                     : : "r"(tile_a), "l"(weights + step * hidden_dim), "r"(sizeof(half) * hidden_dim * 16));
        
        asm volatile("mbarrier.arrive %0;" : : "r"(0));
        // Synchronize transaction barrier and execute matrix multiplication
        asm volatile("mbarrier.try_wait %0, 0;" : : "r"(0));

        // Compute input gradients for immediate background All-to-All dispatch
        atomicAdd(&grad_input_buffer[block.thread_rank()], tile_a[block.thread_rank()]);
    }

    // 2. Background accumulation of weight gradients in parallel with activation transfer
    block.sync();
    atomicAdd(&grad_weight_accumulator[block.thread_rank()], grad_output[block.thread_rank()]);
}

This low-level trick completely eliminates idle gaps: while the network card pumps gigabytes of gradients, the tensor cores stay 100% busy crunching weight matrices.

Benchmarks on 128 GPUs: 55% Step Speedup for Qwen 3

The kernel's test results exceeded the developers' most optimistic expectations:

  • On a single isolated MoE layer, speedup reached 3.11x on the Forward pass and 1.87x on the Backward pass;
  • Across a full end-to-end training step of the open Qwen 3 235B-A22B model on a cluster of 128 Nvidia H200 GPUs, Prime Mega MoE delivered a total speed gain of 55% compared to the reference Meta TorchTitan stack!

For easy integration into popular training frameworks, Prime Intellect provided a compact PyTorch wrapper layer with automatic registration of custom autograd operators:

import torch
import torch.nn as nn
from prime_kernels import moe_overlapping_backward_op

class PrimeMoELayer(nn.Module):
    def __init__(self, num_experts: int, hidden_dim: int, expert_dim: int):
        super().__init__()
        self.num_experts = num_experts
        self.hidden_dim = hidden_dim
        self.expert_dim = expert_dim
        self.weights = nn.Parameter(torch.empty(num_experts, hidden_dim, expert_dim, dtype=torch.bfloat16))
        nn.init.xavier_uniform_(self.weights)

    def forward(self, hidden_states: torch.Tensor, router_indices: torch.Tensor):
        # Invoke the compiled kernel with asynchronous streaming support
        return moe_overlapping_backward_op(
            hidden_states, 
            self.weights, 
            router_indices, 
            overlap_comm=True
        )

As we analyzed in our breakdown of inference optimization for Mixture-of-Experts models, a 55% step speedup translates in real terms into hundreds of thousands of dollars saved on compute rental and shrinks fine-tuning cycles from two months down to three or four weeks. The kernel has already been integrated into the open reinforcement learning library Prime RL.

Hardware Requirements and Applicability Limits

Such high performance, however, comes with a strict hardware tie-in:

  1. Hopper and Blackwell only: the kernel relies critically on hardware asynchronous data-copy instructions—TMA (Tensor Memory Accelerator)—which first debuted in Nvidia's Hopper chips (H100, H200) and were expanded in Blackwell (B200).
  2. Unusable on older or consumer hardware: on Ampere GPUs (A100) or consumer RTX 4090 boards, the kernel cannot run—they physically lack TMA hardware units, and the CUDA compiler will simply throw an instruction error.

Prime Mega MoE is a vivid example of how smart low-level optimization can outpace brute-force server scaling. The open-source kernel code has once again proven the strength of the Open Source community against closed corporate engineering.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 3