Skip to main content
Search

# Mixture of Experts

26 related articles

Subscribe to updates
  1. arXiv

    Two policies for one rendezvous: a learned mixture guidance law meets 10 m and 1 m/s on all 600 multi-revolution low-thrust test missions

    Multi-revolution low-thrust rendezvous must combine a long orbital transfer with meter-level terminal control, and a single policy struggles to cover both state scales. This guidance law has a far-range expert read modified equinoctial elements and a terminal expert read the relative state in the target's local frame, while a temporal-attention gate blends their thrust commands continuously. On 600 test missions it completed every rendezvous at 10 m and 1 m/s tolerances, cutting mean flight time by 50.6% and velocity increment by 44.5% relative to 4 km distance-based hard switching.

  2. arXiv

    Strength criteria stop being hand-picked: a sparse differentiable search bakes convexity and uniqueness into the fit, cutting concrete's cross-validated error fourfold

    Multiaxial strength tests give only a few dozen failure stress states, and choosing which limit surface to fit has long relied on manual comparison and visual checks. Researchers recast that choice as sparse model discovery with differentiable penalties, define the surface by the first root along rays so the equivalent stress is explicitly solvable, and add PBP atoms with learnable shape parameters plus competition-gated mixtures of experts. On concrete data the PBP model's cross-validated error is about a quarter of the monomial library's, and measured strength ratios are reproduced within a few per cent.

  3. arXiv

    AutoCompress writes deployment limits into pruning search: up to 6.57 points higher average accuracy than Minitron++ across three GPUs

    LLM pruning usually targets parameter counts, but parameter count does not predict latency, throughput or memory, which depend on architecture, workload, hardware and serving engine. AutoCompress puts latency, throughput, parameter and memory constraints directly into differentiable architecture search, searching global dimensions and per-layer structure together and guided by cost models calibrated by measurement inside the target serving engine. Under the same deployment budgets and matched recovery training as Minitron++, it raises average accuracy across eight benchmarks by 6.57, 2.35 and 0.57 percentage points on RTX 5070, RTX 5080 and H100, using roughly one-half to one-third as many architecture-selection tokens.

  4. arXiv

    PRISM-Net splits product retrieval into line and color routes, lifting SwinV2 line mAP@10 by 5.74 points

    PRISM-Net splits a product image's holistic features into a line pathway and a color pathway, supervises the two routers zero-shot with frozen SigLIP text prompts, and fuses both with an image-adaptive meta-router. In 5-fold cross-validation over 275,454 e-commerce images in 18 categories and full-scale retrieval of 55,091 queries against a 220,363-image gallery, the SwinV2 backbone gains 5.74 points in line mAP@10, while the color-only configuration gains 9.12 points in color mAP@10; on ConvNeXt-V2 and DINOv3, product retrieval and attribute alignment trade off.

  5. Dropless MoE training in JAX gets a tenfold lift: DeepSeek-V3 671B holds 97% scaling efficiency at 1,024 GPUs

    NVIDIA wired Transformer Engine's MoE optimizations into the JAX training stack so every token reaches its selected expert, with no dropping and no padding. An unoptimized baseline training DeepSeek-V3 on GB300 reached only 103 TFLOPS/GPU, with inter-GPU communication consuming 84% of accumulated kernel time; grouped GEMM, MXFP8 grouped quantization and fused expert-parallel dispatch and combine raised that to 1,068 TFLOPS/GPU, about a tenfold end-to-end throughput gain, and the stack sustained 97% efficiency at 1,024 GPUs.

  6. arXiv

    Multi-domain fine-tuning stops stepping on itself: 256 examples per domain lift expert routing to 90–98%

    Dense Transformer layers are expanded into sparse mixture-of-experts layers, giving law and biomedicine their own full MLP experts while a router is supervised with a small number of examples to pick the right expert per domain. On Llama-3.1-8B and Qwen-3-8B the method posts the highest biomedical and legal averages and the highest general-domain average; with only 256 examples per domain for router supervision, routing accuracy on specialized domains reaches 90–98% after adaptation, against 43–70% for routers trained without it.

  7. aclanthology.org

    Templated discharge notes made different patients look alike: pretraining only on narrative sections lifts ICD coding macro-F1 to 28.6%

    Automatic ICD coding assigns standardized diagnostic codes to discharge summaries, but sections such as hospital course and discharge diagnosis follow near-identical templates, so pretraining can latch onto document structure instead of patient-specific clinical meaning. SARL-ICD restricts contrastive pretraining to narrative sections such as chief complaint, history of present illness and past medical history, builds positive pairs from Jaccard similarity over ICD codes expanded with their hierarchy ancestors, injects LLM-augmented code descriptions through a lightweight label-text interaction module, and fuses PLM-CA with PLM-ICD through a mixture-of-experts gate. On MIMIC-IV ICD-10 the full system reaches 28.6% macro-F1 and 61.1% micro-F1, against 26.1% and 59.6% for PLM-CA.

  8. Fireworks AI

    Qwen3 puts its reasoning trace and JSON tool call in one completion, open under Apache-2.0 on Fireworks

    Open models previously forced a choice between showing a chain of thought and calling tools deterministically. Qwen3-235B-A22B places its reasoning inside a separate think tag and emits the exact JSON tool call in the same completion, so downstream code can ignore or audit the trace. The model is a 235-billion-parameter mixture-of-experts with about 22 billion active parameters, hosted serverlessly on Fireworks under Apache-2.0 and callable through an OpenAI-compatible client.

  9. Fireworks AI

    Kimi K3 ships open-weight on Fireworks: 1,679 points tops the frontend code arena, training runs on a shared GPU pool

    Open-weight Kimi K3 is available on Fireworks for inference and training from day one, which the release presents as frontier-level capability at a fraction of closed-model cost. On Arena's Frontend Code Arena it scores 1,679 points, ahead of Fable 5, making it the first open model to sit ahead of every closed one; it ranks third worldwide on the Artificial Analysis Intelligence Index at 57. Fireworks also offers US-only inference endpoints with zero data retention by default and a private preview of serverless training on a shared GPU pool.

  10. Trillion-Parameter Pretraining Goes Bitwise-Deterministic: NVIDIA Cuts the Determinism Tax to About 2%

    NVIDIA has pushed the cost of bitwise-deterministic pretraining in Megatron Core down to a workable range: two independent runs from the same initial state, and a run resumed from checkpoint, must stay numerically identical at every step. Such deterministic recipes previously carried double-digit overhead; after buffer-fill optimization Nemotron 3 Ultra fell from about 17% to 2%, then to 1.5% at 3,072 GPUs, and the large-scale Nemotron recipe measured about 2% steady-state overhead at 2,432 GPUs with bitwise determinism over 800 steps.