Training a large AI model is expensive in every currency that matters — dollars, time, energy, and scarce compute. The usual ways to end up with a small, fast model both waste some of that: either train a giant one and trim it down afterward, or train a small one from scratch and accept weaker results. Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and collaborators say they've found a third path that sidesteps the trade-off — compressing a model during training instead of after. The work was reported by MIT News.
The technique, called CompreSSM, targets a family of architectures known as state-space models, which underpin language processing, audio generation, and robotics. Borrowing mathematical tools from control theory, it identifies which parts of a model are pulling their weight and which are dead weight, then surgically removes the useless components early in training. "It's essentially a technique to make models grow smaller and faster as they are training," said lead author Makram Chahine, a PhD student in electrical engineering and computer science and a CSAIL affiliate. "During learning, they're also getting rid of parts that are not useful to their development."
The key insight is that the relative importance of a model's internal components settles surprisingly early. Using a quantity called Hankel singular values — a measure of how much each internal state contributes to overall behavior — the team found they could reliably rank which dimensions matter after only about 10 percent of training. Once that ranking is set, the less-important pieces are discarded and the remaining 90 percent of training runs at the speed of a much smaller model.
"What's exciting about this work is that it turns compression from an afterthought into part of the learning process itself," said senior author Daniela Rus, an MIT professor and director of CSAIL. "Instead of training a large model and then figuring out how to make it smaller, CompreSSM lets the model discover its own efficient structure as it learns. That's a fundamentally different way to think about building AI systems."
The numbers are what make the case. On image-classification benchmarks, compressed models held nearly the same accuracy as their full-sized counterparts while training up to 1.5 times faster, per MIT News. A model shrunk to roughly a quarter of its original state dimension hit 85.7 percent accuracy on CIFAR-10 — versus just 81.8 percent for a model trained at that smaller size from scratch. On Mamba, one of the most widely used state-space architectures, the method delivered about 4x training speedups, compressing a 128-dimensional model down to around 12 dimensions while staying competitive. "You get the performance of the larger model, because you capture most of the complex dynamics during the warm-up phase, then only keep the most-useful states," Chahine said.
The distinction from existing tricks is theoretical grounding. Conventional pruning trains the full model and strips parameters afterward — so you still pay the full cost of training the big one. Knowledge distillation trains a large "teacher" to completion and then a smaller "student" on top, roughly doubling the effort. CompreSSM makes its cuts mid-stream, and in head-to-head tests against a recent spectral technique (Hankel nuclear norm regularization) it ran more than 40 times faster while achieving higher accuracy. The collaboration spans MIT CSAIL, the Max Planck Institute for Intelligent Systems, ELLIS, ETH, and Liquid AI.
There's a broader shift buried in the method. As the industry's default answer to better AI has been "make it bigger, then deal with the cost," CompreSSM points the other way — letting a model find its own lean shape while it learns. If that holds up beyond state-space models, the cheapest place to save compute may turn out to be the training run itself, not the cleanup afterward.
Written for Red Robot with AI assistance and human editing. Based on reporting by MIT News.
https://redrobot.online/2026/08/07/rr-16030-new-technique-makes-ai-models-leaner-and-faster-while-they-r/
No comments:
Post a Comment