ByteDance Introduces QuaDMix: A Unified AI Framework for Data Quality and Diversity in LLM Pretraining

ByteDance Introduces QuaDMix: A Unified AI Framework for Data Quality and Diversity in LLM Pretraining

The pretraining efficiency and generalization of large language models (LLMs) are significantly influenced by the quality and diversity of the underlying training corpus. Traditional data curation pipelines often treat quality and diversity as separate objectives, applying quality filtering followed by domain balancing. This sequential optimization overlooks the complex interdependencies between these factors. High-quality datasets frequently ex…