Does learned rank allocation help LoRA?
Under review, ACL Rolling Review
- Question
- Methods such as AdaLoRA let importance scores decide which layers get more rank, on the assumption that this spends a fixed parameter budget better than giving every layer the same rank. That assumption had rarely been tested with matched controls.
- Approach
- A controlled study covering both standard and shared-parameter low-rank adapters, with every run paired with fixed-rank and random controls.
- Finding
- Learned rank allocation was never detectably better than a fixed rank at the same parameter budget. The choice of importance score mattered mainly when a trained adapter was compressed in one step.