publications

2024

2023

1. TOD-Flow: Modeling the Structure of Task-Oriented Dialogues

   Sungryull Sohn, Yiwei Lyu, Anthony Liu, and 4 more authors

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing 2023
2. Fine-grained Text Style Transfer with Diffusion-Based Language Models

   Yiwei Lyu, Tiange Luo, Jiacheng Shi, and 2 more authors

   In Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023) 2023
3. Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control

   Xiang Fan, Yiwei Lyu, Paul Pu Liang, and 2 more authors

   In Findings of the Association for Computational Linguistics: ACL 2023 2023
4. MultiViz: Towards Visualizing and Understanding Multimodal Models

   Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, and 5 more authors

   In The Eleventh International Conference on Learning Representations 2023

2022

1. DIME: Fine-Grained Interpretations of Multimodal Models via Disentangled Local Explanations

   Yiwei Lyu, Paul Pu Liang, Zihao Deng, and 2 more authors

   In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society 2022

   Abs PDF Code

   The ability for a human to understand an Artificial Intelligence (AI) model’s decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making. As a result, the research fields of interpretable and explainable AI have gained traction within AI communities as well as interdisciplinary scientists seeking to apply AI in their subject areas. In this paper, we focus on advancing the state-of-the-art in interpreting multimodal models - a class of machine learning methods that tackle core challenges in representing and capturing interactions between heterogeneous data sources such as images, text, audio, and time-series data. Multimodal models have proliferated numerous real-world applications across healthcare, robotics, multimedia, affective computing, and human-computer interaction. By performing model disentanglement into unimodal contributions (UC) and multimodal interactions (MI), our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models while maintaining generality across arbitrary modalities, model architectures, and tasks. Through a comprehensive suite of experiments on both synthetic and real-world multimodal tasks, we show that DIME generates accurate disentangled explanations, helps users of multimodal models gain a deeper understanding of model behavior, and presents a step towards debugging and improving these models for real-world deployment.

2. High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for high-modality representation learning

   Paul Pu Liang, Yiwei Lyu, Xiang Fan, and 6 more authors

   Transactions on Machine Learning Research 2022

2021

1. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

   Paul Pu Liang, Yiwei Lyu, Xiang Fan, and 8 more authors

   In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) 2021

   Abs PDF Code Website

   Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized code, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

2. StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer

   Yiwei Lyu, Paul Pu Liang, Hai Pham, and 4 more authors

   In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Jun 2021

   Abs PDF Code

   Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, StylePTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text, as well as (2) compositions of multiple transfers which allow modeling of fine-grained stylistic changes as building blocks for more complex, high-level transfers. By benchmarking existing methods on StylePTB, we find that they struggle to model fine-grained changes and have an even more difficult time composing multiple styles. As a result, StylePTB brings novel challenges that we hope will encourage future research in controllable text style transfer, compositional models, and learning disentangled representations. Solving these challenges would present important steps towards controllable text generation.

2020

2019

1. Leveraging Program Invariants to Promote Population Diversity in Search-Based Automatic Program Repair

   Zhen Yu Ding, Yiwei Lyu, Christopher Timperley, and 1 more author

   In 2019 IEEE/ACM International Workshop on Genetic Improvement (GI) Jun 2019

   Abs PDF

   Search-based automatic program repair has shown promise in reducing the cost of defects in real-world software. However, to date, such techniques have typically been most successful when constructing short or single-edit repairs. This is true even when techniques make use of heuristic search strategies, like genetic programming, that in principle support the construction of patches of arbitrary length. One key reason is that the fitness function traditionally depends entirely on test cases, which are poor at identifying partially correct solutions and lead to a fitness landscape with many plateaus. We propose a novel fitness function that optimizes for both functionality and semantic diversity, characterized using learned invariants over intermediate behavior. Our early results show that this new approach improves semantic diversity and fitness granularity, but does not statistically significantly improve repair performance.