Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto …

Moonlight is a 16B-parameter Mixture-of-Experts (MoE) model from Moonshot AI trained with 5.7T tokens using the innovative Muon optimizer.

Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto …

Jun 14, 2025 · Based on these improvements, we introduce **Moonlight**, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the …

Training library for Megatron-based models. Contribute to NVIDIA-NeMo/Megatron-Bridge development by creating an account on GitHub.

This model strikes an impressive performance-per-FLOP balance, outperforming similar-sized models like Llama 3 3B and DeepSeek-v2-Lite across benchmarks in English, code …

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Your need to confirm your account before you can post a new comment. using VLLM, Model output …

What is Moonlight-16B-A3B? Moonlight-16B-A3B is a state-of-the-art Mixture-of-Expert (MoE) language model that represents a significant advancement in efficient AI model training.

Features: 16b LLM, VRAM: 32.6GB, Context: 8K, License: mit, LLM Explorer Score: 0.29. Find out how Moonlight 16B A3B can be utilized in your business workflows, problem-solving, and …

Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto …

Moonlight is a 16B parameter Mixture of Experts (MoE) model trained using the Muon optimizer, demonstrating outstanding performance in large-scale training. By incorporating weight decay …

These model weights are for personal testing purposes only. The goal is to find a quantization method that achieves high compression while preserving as much of the model's original …