Popular GitHub Repositories for Knowledge Distillation

Awesome Knowledge Distillation

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Collection of AWESOME vision-language models for vision tasks

"Effective Whole-body Pose Estimation with Two-stages Distillation" (ICCV 2023, CV4Metaverse Workshop)

A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility

This is a collection of our NAS and Vision Transformer work.

A curated list for Efficient Large Language Models

OpenMMLab Model Compression Toolbox and Benchmark.

NLP DNN Toolkit - Building Your NLP DNN Models Like Playing Lego

Improving Convolutional Networks via Attention Transfer (ICLR 2017)

利用pytorch实现图像分类的一个完整的代码，训练，预测，TTA，模型融合，模型部署，cnn提取特征，svm或者随机森林等进行分类，模型蒸馏，一个完整的代码

This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.

EasyTransfer is designed to make the development of transfer learning in NLP applications easier.

The official implementation of [CVPR2022] Decoupled Knowledge Distillation https://arxiv.org/abs/2203.08679 and [ICCV2023] DOT: A Distillation-Oriented Trainer https://openaccess.thecvf.com/content/ICCV2023/papers/Zhao_DOT_A_Distillation-Oriented_Trainer_ICCV_2023_paper.pdf

A Pytorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.

Segmind Distilled diffusion

Code and resources on scalable and efficient Graph Neural Networks (TNNLS 2023)

Official pytorch Implementation of Relational Knowledge Distillation, CVPR 2019

(MLSys' 21) An Acceleration System for Large-scare Unsupervised Heterogeneous Outlier Detection (Anomaly Detection)

Localization Distillation for Object Detection (CVPR 2022, TPAMI 2023)