Publications

September 08, 2026

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing, Zhengzhong Liu

[arXiv]

September 04, 2026

Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

ICML, 2026

Mostafa Elhoushi, Alex Pretko, Nolan Dey, Bin Claire Zhang, Gavia Gray, Gurpreet Gosal, Abdulrahman Mahmoud, Shane Bergsma, Joel Hestness
[OpenReview] [arXiv]

July 07, 2026

Jais 2: A Family of Arabic-Centric Open Large Language Models

EMNLP, 2026

Mohamed Anwar, Abed Alhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Sadallah, Gurpreet Gosal, Gokulakrishnan Ramakrishnan, Sarath Chandran, Biswajit Mishra, Rituraj Joshi, Ahmed Frikha, Etienne Goffinet, Abhishek Maiti, Ali El Filali, Sarah AlBarri, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Awantika Shukla, Sajid siddiki, Samta Kamboj, Onkar Pandit, Sunil Kumar Sahu, AbdelRahman Elbadawy, Amr Mohamed, Ahmad Chamma, Evan Dufraisse, Abdelaziz Bounhar, Dani Bouch, Hadi Abdine, Guokan Shang, Fajri Koto, Yuxia Wang, Zhuohan Xie, Ali Mekky, Rania Elbadry, Sarfraz Ahmad, Momina Ahsan, Omar El Herraoui, Daniil Orel, Hasan Iqbal, Kareem Elzeky, Mervat Abassy, Kareem Elozeiri, Saadeldine Eletter, Farah Atif, Nurdaulet Mukhituly, Haonan Li, Xudong Han, Aaryamonvikram Singh, Zainul Abedien Ahmed Quraishi, Neha Sengupta, Larry Murray, Avraham Sheinin, Joel Hestness, Natalia Vassilieva, Hector Xuguang Ren, Zhengzhong Liu, Michalis Vazirgiannis, Preslav Nakov

[Cerebras Blog] [arXiv]

May 30, 2026

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

ACL, 2026

Zining Liu, Yunhai Hu, Tianhua Xia, Bo Bao, Eric Sather, Vithursan Thangarasa, Sai Qian Zhang

[arXiv]

May 27, 2026

DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution

ICML, 2026

Yunhai Hu, Zining Liu, Xiangyang Yin, Tianhua Xia, Bo Bao, Eric Sather, Vithursan Thangarasa, Sai Qian Zhang

[OpenReview] [arXiv]

April 12, 2026

CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts

ICLR, 2026

Xiangyang Yin, Xingyu Liu, Tianhua Xia, Bo Bao, Vithursan Thangarasa, Valavan Manohararajah, Eric Sather, Sai Qian Zhang

[OpenReview] [arXiv]

November 06, 2025

Crystal: Illuminating LLM Abilities on Language and Code

Tianhua Tao, Junbo Li, Bowen Tan, Hongyi Wang, William Marshall, Bhargav M Kanakiya, Joel Hestness, Natalia Vassilieva, Zhiqiang Shen, Eric P. Xing, Zhengzhong Liu

[arXiv]

November 03, 2025

TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding

Aditya Sridhar, Nish Sinnadurai, Sean Lie, Vithursan Thangarasa

[arXiv]

October 25, 2025

REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression

ICLR, 2026

Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, Vithursan Thangarasa

[OpenReview] [arXiv]

September 29, 2025

Scaling with Collapse: Efficient and Predictable Training of LLM Families

ICLR, 2026

Shane Bergsma, Bin Claire Zhang, Nolan Dey, Shaheer Muhammad, Gurpreet Gosal, Joel Hestness

[OpenReview] [arXiv]

September 29, 2025

Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs

ICLR, 2026

Shane Bergsma, Nolan Dey, Joel Hestness

[OpenReview] [arXiv]

July 18, 2025

Wafer-Scale Integration for AI – The Holy Grail?

VLSI 2025

Bendik Kleveland; Phil Ferolito; Mike Morrison; Alisa Scherer

[IEEE Xplore]

May 25, 2025

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

NeurIPS, 2025

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

[OpenReview] [arXiv]

May 19, 2025

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training

NeurIPS, 2025

Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness

[OpenReview] [arXiv]

May 02, 2025

Don't be lazy: CompleteP enables compute-efficient deep transformers

NeurIPS, 2025

Nolan Dey*, Bin Claire Zhang*, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, Joel Hestness
[OpenReview] [arXiv]

March 25, 2025

Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting

Fajri Koto; Rituraj Joshi; Nurdaulet Mukhituly; Yuxia Wang; Zhuohan Xie; Rahul Pal; Daniil Orel; Parvez Mullah; Diana Turmakhan; Maiya Goloburda; Mohammed Kamran; Samujjwal Ghosh; Bokang Jia; Jonibek Mansurov; Mukhammed Togmanov; Debopriyo Banerjee; Nurkhan Laiyk; Akhmed Sakip; Xudong Han; Ekaterina Kochmar; Alham Fikri Aji; Aaryamonvikram Singh; Alok Anil Jadhav; Satheesh Katipomu; Samta Kamboj; Monojit Choudhury; Gurpreet Gosal; Gokul Ramakrishnan; Biswajit Mishra; Sarath Chandran; Avraham Sheinin; Natalia Vassilieva; Neha Sengupta; Larry Murray; Preslav Nakov

[arXiv]

February 21, 2025

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs

ICLR, 2025

Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness

[OpenReview] [arXiv]

November 18, 2024

Empirical Upper Bounds for Unstructured Sparsity in Compute-Efficient Language Modeling

Machine Learning and Compression NeurIPS Workshop, 2024

Esha Singh, Shane Bergsma, Nolan Dey, Joel Hestness, Gavia Gray

[OpenReview] [Poster]

November 01, 2024

Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers

NeurIPS, 2025

Gavia Gray, Aman Tiwari, Shane Bergsma, Joel Hestness

[OpenReview] [arXiv]

October 31, 2024

Sparse maximal update parameterization: A holistic approach to sparse training dynamics

NeurIPS, 2024

Nolan Dey, Shane Bergsma, Joel Hestness

[OpenReview] [arXiv]

October 13, 2024

Self-Data Distillation for Recovering Quality in Pruned Large Language Models

MLSys, 2025

Vithursan Thangarasa, Ganesh Venkatesh, Mike Lasby, Nish Sinnadurai, Sean Lie

[arXiv]

September 04, 2024

Bilingual Adaptation of Monolingual Foundation Models

FM-Wild ICML Workshop, 2024

Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Zhiming (Charles)Chen, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Onkar Pandit, Satheesh Katipomu, Samta Kamboj, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Soundar Doraiswamy, Mohamed El Karim Chami, Preslav Nakov

[arXiv] [OpenReview]

June 01, 2024

The practitioner's guide to the maximal update parameterization

Blog & open-source code, 2024

Nolan Dey, Quentin Anthony, Joel Hestness

[Cerebras Blog] [Eleuther AI Blog] [nanoGPT-mup Code]

May 20, 2024

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

Vithursan Thangarasa, Mahmoud Salem, Shreyas Saxena, Kevin Leong, Joel Hestness, Sean Lie

[arXiv]

Publications

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

Jais 2: A Family of Arabic-Centric Open Large Language Models

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution

CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts

Crystal: Illuminating LLM Abilities on Language and Code

TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding

REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression

Scaling with Collapse: Efficient and Predictable Training of LLM Families

Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs

Wafer-Scale Integration for AI – The Holy Grail?

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training

Don't be lazy: CompleteP enables compute-efficient deep transformers

Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs

Empirical Upper Bounds for Unstructured Sparsity in Compute-Efficient Language Modeling

Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers

Sparse maximal update parameterization: A holistic approach to sparse training dynamics

Self-Data Distillation for Recovering Quality in Pruned Large Language Models

Bilingual Adaptation of Monolingual Foundation Models

The practitioner's guide to the maximal update parameterization

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

Schedule a meeting to discuss your AI vision and strategy.

Publications

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

Jais 2: A Family of Arabic-Centric Open Large Language Models

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution

CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts

Crystal: Illuminating LLM Abilities on Language and Code

TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding

REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression

Scaling with Collapse: Efficient and Predictable Training of LLM Families

Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs

Wafer-Scale Integration for AI – The Holy Grail?

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training

Don't be lazy: CompleteP enables compute-efficient deep transformers

Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs

Empirical Upper Bounds for Unstructured Sparsity in Compute-Efficient Language Modeling

Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers

Sparse maximal update parameterization: A holistic approach to sparse training dynamics

Self-Data Distillation for Recovering Quality in Pruned Large Language Models

Bilingual Adaptation of Monolingual Foundation Models

The practitioner's guide to the maximal update parameterization

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

Schedule a meeting to discuss your AI vision and strategy.