Seminar by Prof Shao-Lun Huang , Tsinghua-Berkeley Shenzhen Institute, 20 Nov 2024, LT10 (NS4-04-41)

Time: 10-11 am, November 20th 2024, Wednesday

Venue: LT10 (NS4-04-41)

            (map: https://maps.ntu.edu.sg/#/ntu/d386ffa80e4e46f286d17f08/poi/details/47b4fc658ac44d4aaeca2bb9)

Title: Informative Feature Extractions in Deep Neural Networks

Abstract: In contemporary machine learning, it is critical to identify “informative” low-dimensional features from high-dimensional data for learning tasks, while the notion of “informative” is not unified over learning problems. In this talk, we introduce an information metric for quantifying the informativeness of features from Hirschfeld-Gebelein-Rényi maximal correlation and a modal decomposition of the dependence between random variables, via an information geometric approach. We show that this information metric is related to several scenarios, including binary hypothesis testing, Canonical Correlation Analysis (CCA), Wyner’s common information, and softmax regression in deep neural networks. This establishes a theoretic connection between information theory, statistical learning, and machine learning algorithms by understanding the underlying information structure. We show that such information structure allows us to design learning architectures in multi-modal machine learning and transfer learning with more efficient and interpretable algorithms. 

Bio: Shao-Lun Huang received the B.S. degree with honor in 2008 from the Department of Electronic Engineering, National Taiwan University, Taiwan, and the M.S. and Ph.D. degree in 2010 and 2013 from the Department of Electronic Engineering and Computer Sciences, Massachusetts Institute of Technology. From 2013 to 2016, he was working as a postdoctoral researcher jointly in the Department of Electrical Engineering at the National Taiwan University and the Department of Electrical Engineering and Computer Science at the Massachusetts Institute of Technology. Since 2016, he has joined Tsinghua-Berkeley Shenzhen Institute, where he is currently a tenured associate professor. His research interests include information theory, communication theory, machine learning, and social networks.