Knowledge Distillation

Natural & Formal Sciences Dictionary
A model compression technique in which a smaller 'student' model is trained to reproduce the behavior or softened outputs of a larger 'teacher' model so that the student inherits useful predictive structure while being more compact and efficient.