It's All in the Head: Representation Knowledge Distillation through Classifier Sharing
Authors:
Emanuel Ben-Baruch,
Matan Karklinsky,
Yossi Biton,
Avi Ben-Cohen,
Hussam Lawen,
Nadav Zamir
Abstract:
Representation knowledge distillation aims at transferring rich information from one model to another. Common approaches for representation distillation mainly focus on the direct minimization of distance metrics between the models' embedding vectors. Such direct methods may be limited in transferring high-order dependencies embedded in the representation vectors, or in handling the capacity gap b…
▽ More
Representation knowledge distillation aims at transferring rich information from one model to another. Common approaches for representation distillation mainly focus on the direct minimization of distance metrics between the models' embedding vectors. Such direct methods may be limited in transferring high-order dependencies embedded in the representation vectors, or in handling the capacity gap between the teacher and student models. Moreover, in standard knowledge distillation, the teacher is trained without awareness of the student's characteristics and capacity. In this paper, we explore two mechanisms for enhancing representation distillation using classifier sharing between the teacher and student. We first investigate a simple scheme where the teacher's classifier is connected to the student backbone, acting as an additional classification head. Then, we propose a student-aware mechanism that asks to tailor the teacher model to a student with limited capacity by training the teacher with a temporary student's head. We analyze and compare these two mechanisms and show their effectiveness on various datasets and tasks, including image classification, fine-grained classification, and face verification. In particular, we achieve state-of-the-art results for face verification on the IJB-C dataset for a MobileFaceNet model: TAR@(FAR=1e-5)=93.7\%. Code is available at https://github.com/Alibaba-MIIL/HeadSharingKD.
△ Less
Submitted 5 April, 2022; v1 submitted 18 January, 2022;
originally announced January 2022.
A formula for the doubly-refined enumeration of alternating sign matrices
Authors:
Matan Karklinsky,
Dan Romik
Abstract:
Zeilberger proved the Refined Alternating Sign Matrix Theorem, which gives a product formula, first conjectured by Mills, Robbins and Rumsey, for the number of alternating sign matrices with given top row. Stroganov proved an explicit formula for the number of alternating sign matrices with given top and bottom rows. Fischer and Romik considered a different kind of "doubly-refined enumeration" w…
▽ More
Zeilberger proved the Refined Alternating Sign Matrix Theorem, which gives a product formula, first conjectured by Mills, Robbins and Rumsey, for the number of alternating sign matrices with given top row. Stroganov proved an explicit formula for the number of alternating sign matrices with given top and bottom rows. Fischer and Romik considered a different kind of "doubly-refined enumeration" where one counts alternating sign matrices with given top two rows, and obtained partial results on this enumeration. In this paper we continue the study of the doubly-refined enumeration with respect to the top two rows, and use Stroganov's formula to prove an explicit formula for these doubly-refined enumeration numbers.
△ Less
Submitted 18 June, 2009;
originally announced June 2009.
