About me
I am a third-year master student at Zhejiang University, and I am fortunate to be advised by Prof. Zhongjie Ba and Kui Ren. Prior to attending ZJU, I earned my Bachelor’s degree in Computer Science from Zhejiang University City College (Hangzhou City University).
My current research interests are focused on exploring LLM and Large Multimodal Models self-alignment and building Large Multimodal Models for embodied AI.
I’m looking for a PhD position starting in Sept of 2024.
News
- 07/2023: Our paper ‘Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training’ is accepted by ICCV 2023.
- 07/2023: Our paper ‘DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues’ is accepted by ACM MM 2023.
- 12/2022: Our paper-InfoMasker was accepted at NDSS’2023.
- 05/2022: Started an internship at Microsoft, supervised by Dr. Shuang Ma.
Research Experience
- Remote Research Intern: Apple. Supervised by Dr. Shuang Ma. May 2023 -Present
- Applied Research on Visual Large Language Models
exploring LLM and Large Multimodal Models self-alignment and building Large Multimodal Models for embodied AI.
- Applied Research on Visual Large Language Models
- Remote Research Intern: Microsoft Research. Supervised by Dr. Shuang Ma. May 2022 - March 2023
- Pretraining model for perception and control
proposed a “Dual-phase” training approach that mimics human learning, merging self-supervised knowledge acquisition with decision-making based on context,demonstrating exceptional generalizability across tasks without task-specific fine-tuning.
- Pretraining model for perception and control
- Research Assistant: Zhejiang University. Advisor: Prof. Zhongjie Ba. Sep. 2021 - Present
- Speech privacy eavesdropping protection
design a phoneme-based noise that is robust against denoising methods and can effectively prevent both humans and machines from understanding the jammed signals. - Deepfake detection
introduced an incremental learning framework for deepfake detection, leveraging domain-invariant representation and multi-perspective knowledge distillation.
- Speech privacy eavesdropping protection
Publications
- Yao Wei, Y. Sun, R. Zheng, S. Vemprala, R. Bonatti, S. Chen, R. Madaan, Z. Ba, A. Kapoor, S. Ma. Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Training. ICCV 2023.
- Kun Pan, Yifang Yin, Yao Wei, Feng Lin, Zhongjie Ba, Zhenguang Liu, Zhibo Wang, Lorenzo Cavallaro, Kui Ren. DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues. ACM MM 2023.
- Peng Huang*, Yao Wei, Peng Cheng, Zhongjie Ba, Li Lu, Feng Lin , Fan Zhang, and Kui Ren. InfoMasker: Preventing Eavesdropping Using Phoneme-Based Noise. In NDSS’2023. [Paper] [Code]
Academic Services
Reviewer: IoTJ
Program Committee: the PerDream Workshop at ICCV 2023