Knowledge Distillation: A Survey

Knowledge Distillation: A Survey(arXiv:2006.05525, doi 10.1007/s11263-021-01453-z)은 지식 증류의 종류를 리뷰하여 2021년 International Journal of Computer Vision에 실린 논문이다.

## 주요 내용

### 지식에 따른 지식 증류의 종류

2장에서는 지식 증류의 지식을 다음과 같이 3가지로 나누었다.

* 응답 기반 지식(Response-Based Knowledge)
* 특징 기반 지식(Feature-Based Knowledge)
* 관계 기반 지식(Relation-Based Knowledge)

관계 기반 지식 증류에서 사용하는 손실 함수의 종류를 2가지로 나누었다.

distillation loss of relation-based knowledge based on the relations of feature maps
distillation loss of relation-based knowledge based on the instance relations
이는 뒤에 다른 저자에 쓰인 지식 증류 리뷰 논문인

Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation(arXiv:2306.10687) 의 아래 체계와 흡사하다.

* + 관계 기반 레이어 간 지식 증류(Relation-based Cross-Layer Knowledge Distillation)
	+ 관계 기반 샘플 간 지식 증류(Relation-based Cross-Sample Knowledge Distillation)

#### 특징 기반 지식 증류

* FitNets: Hints for Thin Deep Nets(arXiv:1412.6550) : Fitnet
* Like What You Like: Knowledge Distill via Neuron Selectivity Transfer(arXiv:1707.01219) : Neuron Selectivity Transfer(NST)
* Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer(arXiv:1612.03928): Attention Transfer(AT)
* Paraphrasing Complex Network: Network Compression via Factor Transfer(arXiv:1802.04977) : Factor Transfer(FT)
* Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net(arXiv:1708.04106) : Rocket Launching
* Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation(arXiv:1911.05329) : Knowledge Representing (KR)
* Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons(arXiv:1811.03233) : AB(Activation Boundaries)
* A Comprehensive Overhaul of Feature Distillation(arXiv:1904.01866) : Overhaul
* Feature Normalized Knowledge Distillation for Image Classification(doi 10.1007/978-3-030-58595-2\_40,ECCV) : FN(Feature Normalized)
* Differentiable Feature Aggregation Search for Knowledge Distillation(arXiv:2008.00506) : DFA(Differentiable Feature Aggregation)
* Knowledge distillation via adaptive instance normalization(arXiv:2003.04289) : AdaIN(adaptive instance normalization)
* Feature Normalized Knowledge Distillation for Image Classification(doi 10.1007/978-3-030-58595-2\_40,ECCV) : FN(Feature Normalized)
* Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition(doi 10.1007/978-3-030-58586-0\_20,ECCV) : EC-KD(Exclusivity-consistency)
* ALP-KD: Attention-Based Layer Projection for Knowledge Distillation(arXiv:2012.14022) : ALP-KD
* Cross-Layer Distillation with Semantic Calibration(arXiv:2012.03236) : SemCKD(semantic calibration)

특징 기반 지식 증류를 지식 유형(Knowledge Types), 지식 소스(Knowledge Sources), 증류 손실(Distillation losses)에 따라 정리했다.

#### 관계 기반 지식 증류

* A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning(doi 10.1109/CVPR.2017.754): FSP (flow of solution procedure)
* Learning from Multiple Teacher Networks(doi 10.1145/3097983.3098135, SIGKDD)
* Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification(arXiv:1804.10069)
* DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer(arXiv:1707.01220) : DarkRank
* Graph-based Knowledge Distillation by Multi-head Attention Network(arXiv:1907.02226) : MHGD
* Relational Knowledge Distillation(arXiv:1904.05068) : RKD
* Knowledge Distillation via Instance Relationship Graph(doi 10.1109/CVPR.2019.00726, CVPR 2019) : IRG
* Similarity-Preserving Knowledge Distillation(arXiv:1907.09682) : SP
* Correlation Congruence for Knowledge Distillation(arXiv:1904.01802) : CCKD
* Learning Metrics from Teachers: Compact Networks for Image Embedding(arXiv:1904.03624) : MLKD
* Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning(doi 10.1109/TNNLS.2020.2995884) : PKT(Probabilistic Knowledge Transfer) (arXiv:1803.10837)
* Heterogeneous Knowledge Distillation using Information Flow Modeling(arXiv:2005.00727)
* Learning Student Networks via Feature Embedding(arXiv:1812.06597) : LP(Local Preserving)

관계 기반 지식 증류를 지식 유형(Knowledge Types), 지식 소스(Knowledge Sources), 증류 손실(Distillation losses)에 따라 정리했다.

### 스킴(방법) 에 따른 지식 증류의 종류

3장에서는 오프라인 지식 증류(offline distillation),온라인 지식 증류(online distillation), 자기 지식 증류(self-distillation) 3가지로 지식 증류를 나누었다.

### 선생-학생 구조

4장에서는 선생-학생 구조(Teacher–Student Architecture)에 대해 다루었다.

### 증류 알고리즘

5장에서는 증류 알고리즘(Distillation Algorithms)에 대해 다루었다.

적대적 증류( Adversarial Distillation), 여러 선생 증류( Multi-teacher Distillation), 크로스 모달 증류(Cross-Modal Distillation), 그래프 기반 증류(Graph-Based Distillation), 주의집중 기반 증류(Attention-Based Distillation), 데이터 프리 증류(Data-Free Distillation), 양자화된 증류(Quantized Distillation), 평생 증류(Lifelong Distillation), NAS 기반 증류(NAS-Based Distillation)에 대해 다루었다.

#### 적대적 지식 증류

적대적 지식 증류(Adversarial Distillation)는 GAN에 등장하는 적대적 학습을 이용한 지식 증류다.

#### 여러 선생 증류

오프라인 지식 증류
* Learning from Multiple Teacher Networks(doi 10.1145/3097983.3098135, SIGKDD)
* Efficient Knowledge Distillation from an Ensemble of Teachers(Interspeech 2017)
* Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation(arXiv:1908.07121)
* Distilled Person Re-Identification: Towards a More Scalable System(doi 10.1109/CVPR.2019.00128,CVPR 2019)
* Feature-level Ensemble Knowledge Distillation for Aggregating Knowledge from Multiple Networks(ECAI 2020) : arXiv:1909.10754
* Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System(arXiv:1910.08381)
* Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning(doi 10.1007/978-3-030-58539-6\_38), ECCV 2020)
* Adaptive Knowledge Distillation Based on Entropy(doi 10.1109/ICASSP40776.2020.9054698)
* Adaptive Multi-Teacher Multi-level Knowledge Distillation(arXiv:2103.04062)
* Highlight every step: Knowledge distillation via collaborative teaching(arXiv:1907.09643)

온라인 지식 증류
* Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data(arXiv:1610.05755)
* Born Again Neural Networks(arXiv:1805.04770)
* Deep Mutual Learning(arXiv:1706.00384)
* Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students(arXiv:1805.05551)
* Stochasticity and Skip Connection Improve Knowledge Transfer(doi 10.23919/Eusipco47968.2020.9287227)

#### 크로스 모달 증류

* Learning with Side Information through Modality Hallucination(doi 10.1109/CVPR.2016.96, CVPR 2016)
* Cross Modal Distillation for Supervision Transfer(arXiv:1507.00448)
* Learning Deep Representations with Probabilistic Knowledge Transfer(arXiv:1803.10837)
* Modality Distillation with Multiple Stream Networks for Action Recognition(arXiv:1806.07110)
* Through-Wall Human Pose Estimation Using Radio Signals(doi 10.1109/CVPR.2018.00768 CVPR 2018)
* Cross-Modality Distillation: A case for Conditional Generative Adversarial Networks(arXiv:1807.07682)
* Emotion Recognition in Speech using Cross-Modal Transfer in the Wild(arXiv:1808.05561)
* Cross-modal knowledge distillation for action recognition(arXiv:1910.04641)
* Compact Trilinear Interaction for Visual Question Answering(arXiv:1909.11874)
* Contrastive Representation Distillation(arXiv:1910.10699)
* Privileged Modality Distillation for Vessel Border Detection in Intracoronary Imaging(doi 10.1109/TMI.2019.2952939)

선생 모델과 학생 모델의 모달리티가 다른 크로스 모달(Cross-modal) 증류에 대해서 다루었다.

### 응용

7장에서는 Visual Recognition, 자연어 처리(NLP), 음성 인식(Speech Recognition), 기타 응용(Other Applications) 순으로 다루었다.

### 결론과 토의

8장에서는 결론과 토의를 다루었다. 남은 과제, 미래 연구 방향을 다루었다.

## 같이 보기

* Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation(arXiv:2306.10687)
* 지식 증류 논문 목록