Paper Presented at the International AI Conference AISTATS 2026 / Jeong Jin-woo (Master’s student, Department of Artificial Intelligence), Lee Hyun-jun and Cho Hyun-sik (Class of 2019 and 2020, Department of AI and Big Data Convergence Management)
- 26.05.22 / 홍유민
A paper titled “SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features,” authored by a research team led by Professor Kim Jangho (corresponding author) of the School of Artificial Intelligence at Kookmin University (President Jeong Seung Ryul), with the participation of master’s student Jeong Jin-woo and undergraduate students Lee Hyun-jun and Jo Hyun-sik from the Department of AI Big Data Convergence Management, was presented at AISTATS 2026 held in Tangier, Morocco. AISTATS is a premier international conference in the field of artificial intelligence, held since 1985, covering both theoretical and applied research.
This research is significant for proposing a new methodology that improves the performance of quantized models without training labels. SQuaT (Student-Aware Quantized Teacher Features), proposed in the paper, is a method that aligns the intermediate features of a high-precision teacher model with the student model, taking into account the range that the quantized student model can represent.
Existing label-free Quantization-Aware Training (QAT) methods primarily relied on final output (logit)-based knowledge distillation (KD), which had the limitation of not fully utilizing intermediate feature information. Conversely, methods utilizing feature-level knowledge distillation faced the problem of creating learning objectives that were difficult to achieve in practice due to the difference in value ranges between the high-precision teacher model and the low-bit student model.
The research team developed a “student-aware projection” method that utilizes the student model’s quantization parameters to project the teacher model’s intermediate features into a quantization space that the student can actually represent. Through this, they reduced feature mismatches between the teacher and student models and demonstrated that more stable quantization learning is possible even in a label-free environment.
This achievement is significant because it enhances the performance of lightweight AI models while reducing the cost of additional labeling. The research team has released the source code for the proposed SQuaT to facilitate its use in various quantization and knowledge distillation studies, and plans to continue research on the development of efficient AI models in the future.
△ Photo from the AISTATS conference (Lee Hyun-jun, a senior in the Department of AI Big Data Convergence Management at Kookmin University)
|
This content is translated from Korean to English using the AI translation service DeepL and may contain translation errors such as jargon/pronouns. If you find any, please send your feedback to kookminpr@kookmin.ac.kr so we can correct them.
|
|
Paper Presented at the International AI Conference AISTATS 2026 / Jeong Jin-woo (Master’s student, Department of Artificial Intelligence), Lee Hyun-jun and Cho Hyun-sik (Class of 2019 and 2020, Department of AI and Big Data Convergence Management) |
||||
|---|---|---|---|---|
|
2026-05-22
46
A paper titled “SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features,” authored by a research team led by Professor Kim Jangho (corresponding author) of the School of Artificial Intelligence at Kookmin University (President Jeong Seung Ryul), with the participation of master’s student Jeong Jin-woo and undergraduate students Lee Hyun-jun and Jo Hyun-sik from the Department of AI Big Data Convergence Management, was presented at AISTATS 2026 held in Tangier, Morocco. AISTATS is a premier international conference in the field of artificial intelligence, held since 1985, covering both theoretical and applied research. This research is significant for proposing a new methodology that improves the performance of quantized models without training labels. SQuaT (Student-Aware Quantized Teacher Features), proposed in the paper, is a method that aligns the intermediate features of a high-precision teacher model with the student model, taking into account the range that the quantized student model can represent. Existing label-free Quantization-Aware Training (QAT) methods primarily relied on final output (logit)-based knowledge distillation (KD), which had the limitation of not fully utilizing intermediate feature information. Conversely, methods utilizing feature-level knowledge distillation faced the problem of creating learning objectives that were difficult to achieve in practice due to the difference in value ranges between the high-precision teacher model and the low-bit student model. The research team developed a “student-aware projection” method that utilizes the student model’s quantization parameters to project the teacher model’s intermediate features into a quantization space that the student can actually represent. Through this, they reduced feature mismatches between the teacher and student models and demonstrated that more stable quantization learning is possible even in a label-free environment. This achievement is significant because it enhances the performance of lightweight AI models while reducing the cost of additional labeling. The research team has released the source code for the proposed SQuaT to facilitate its use in various quantization and knowledge distillation studies, and plans to continue research on the development of efficient AI models in the future.
△ Photo from the AISTATS conference (Lee Hyun-jun, a senior in the Department of AI Big Data Convergence Management at Kookmin University)
|
||||
