Channel-Remix
Efficient Task-Specific Knowledge Transfer via Pretrained Channel Remixing
Build a compact student by selecting and sparsely recombining pretrained channels, then fold those combinations into ordinary weights.
The manuscript evaluates the method across 16 tasks and 7 pretraining families. Its main question is whether useful task-specific channels can be composed from pretrained directions with less calibration data.
LoRA-based distillation
Method overview
Recombining pretrained channels
Pruning keeps selected channels. Sparse remixing lets each retained channel absorb contributions from a few discarded channels.
12 connections. Only the mixing coefficients are learned; the result folds into ordinary weights.
Method & experiments ↗Illustrative channel graph, not a live model. 8 source channels → 4 compact channels.
Task-specific channels
A conceptual illustration of task-specific knowledge inside a shared pretrained model. Select a task to change the highlighted routes.
Related work
Contact
You can email me about my research or a possible collaboration.