Wayne Chi

Computer Science Ph.D. Student at Carnegie Mellon University
Previously, Applied Scientist at Amazon and NASA JPL

Wayne-1.jpg

waynechi[AT]cs[DOT]cmu[DOT]edu

Gates Hillman Centers 7121

Hi, my name is Wayne. I am a third year Ph.D. Student in the Computer Science Department at CMU advised by Chris Donahue and Ameet Talwalkar. I am funded by the 2025 NDSEG Fellowship.

I work on applications of machine learning that support human creativity with a focus on game development. My work includes human-centric code evaluation (1, 2) and agentic game development (3). I have also worked on music (4, 5), healthcare, and space automation (6, 7).

I completed my undergraduate and master’s at the University of Southern California. Before starting my Ph.D., I spent around 7 years in industry working as an Applied Scientist at AWS (2019-2023) and NASA’s Jet Propulsion Laboratory (2016-2019). If you are interested in any of my research topics or considering a Ph.D. after industry feel free to reach out.

news

Jun 11, 2026 GameDevBench is accepted to ICML 2026! Results available here.
Feb 9, 2026 EDIT-Bench is accepted to ICLR 2026 as an Oral (Top 1%)
May 1, 2025 Started my internship at Microsoft Research
May 1, 2025 Copilot Arena is accepted to ICML 2025
Apr 1, 2025 Awarded the NDSEG Fellowship

publications

2026

  1. Interaction-as-a-Judge
    Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica, Anastasios Nikolas Angelopoulos, Ameet Talwalkar, Chris Donahue, and Cho-Jui Hsieh
    arXiv preprint, 2026
  2. Comparing Developer and LLM Biases in Code Evaluation
    Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal, Tongshuang Wu, Chris Donahue, Ameet Talwalkar, Wayne Chi, and Valerie Chen
    Conference on Language Modeling (COLM), 2026
  3. GameDevBench: Evaluating Agentic Capabilities Through Game Development
    Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, and Chris Donahue
    International Conference on Machine Learning (ICML), 2026 Code Website
  4. EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
    Wayne Chi, Valerie Chen, Ryan Shar, Aditya Mittal, Jenny Liang, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Ion Stoica, Graham Neubig, Ameet Talwalkar, and Chris Donahue
    International Conference on Learning Representations (ICLR), 2026 (Oral) Website
  5. Completion ≠ Collaboration: Scaling Collaborative Effort with Agents
    Shannon Zejiang Shen, Valerie Chen, Ken Gu, Alexis Ross, Zixian Ma, Jillian Ross, Alex Gu, Chenglei Si, Wayne Chi, Andi Peng, Jocelyn J Shen, Ameet Talwalkar, Tongshuang Wu, and David Sontag
    Findings of the Association for Computational Linguistics (ACL Findings), 2026
  6. RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions
    Keyu He, Qianou Ma, Valerie Chen, Wayne Chi, and Tongshuang Wu
    Annual Meeting of the Association for Computational Linguistics (ACL) - Demo Track, 2026
  7. Learning from 53.6K Real-World Developer Edits of AI-Generated Code
    Jenny T. Liang, Mihika Bairathi, Wayne Chi, Ameet Talwalkar, Nishant Subramani, and Valerie Chen
    arXiv preprint, 2026

2025

  1. Copilot Arena: A Platform for Code LLM Evaluation in the Wild
    Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, and Ameet Talwalkar
    International Conference on Machine Learning (ICML), 2025 Code Website
  2. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig
    Advances in Neural Information Processing Systems (NeurIPS), 2025
  3. Music Arena: Live Evaluation for Text-to-Music
    Yonghyun Kim, Wayne Chi, Anastasios N. Angelopoulos, Wei-Lin Chiang, Koichi Saito, Shinji Watanabe, Yuki Mitsufuji, and Chris Donahue
    Advances in Neural Information Processing Systems (NeurIPS) - Creative AI Track, 2025

2024

  1. The Impact of Element Ordering on LM Agent Performance
    Wayne Chi, Ameet Talwalkar, and Chris Donahue
    NeurIPS 2024 Workshop on Open World Agents (NeurIPS OWA), 2024

2022

  1. Onboard planning for the Mars 2020 Perseverance rover
    Dan Gaines, Steve Chien, Gregg Rabideau, Stephen Kuhn, Vincent Wong, Amruta Yelamanchili, Shannon Towey, Jagriti Agrawal, Wayne Chi, Andrea Connell, and  others
    16th Symposium on Advanced Space Technologies in Robotics and Automation, 2022

2021

  1. Enabling limited resource-bounded disjunction in scheduling
    Jagriti Agrawal, Wayne Chi, Steve Chien, Gregg Rabideau, Stephen Kuhn, Dan Gaines, Tiago Vaquero, and Sarah Bhaskaran
    Journal of Aerospace Information Systems, 2021
  2. Analyzing the effectiveness of rescheduling and flexible execution methods to address uncertainty in execution duration for a planetary rover
    Jagriti Agrawal, Wayne Chi, Steve Chien, Gregg Rabideau, Daniel Gaines, and Stephen Kuhn
    Robotics and Autonomous Systems, 2021
  3. Symbolic music generation with transformer-gans
    Aashiq Muhamed, Liang Li, Xingjian Shi, Suri Yaddanapudi, Wayne Chi, Dylan Jackson, Rahul Suresh, Zachary C Lipton, and Alex J Smola
    AAAI conference on artificial intelligence, 2021
  4. Music generation system
    Liang Li, Wayne Chi, Rahul Suresh, Dylan Jackson, and Haoting Li
    US Patent 11,049,481, 2021

2020

  1. Scheduling with complex consumptive resources for a planetary rover
    Wayne Chi, Steve Chien, and Jagriti Agrawal
    International Conference on Automated Planning and Scheduling, 2020
  2. Automated volcano monitoring using multiple space and ground sensors
    Steve A Chien, Ashley G Davies, Joshua Doubleday, Daniel Q Tran, David Mclaren, Wayne Chi, and Adrien Maillard
    Journal of Aerospace Information Systems, 2020
  3. Generating Music with a Self-Correcting Non-Chronological Autoregressive Model
    Wayne Chi, Prachi Kumar, Suri Yaddanapudi, Rahul Suresh, and Umut Isik
    International Society for Music Information Retrieval, 2020
  4. Self-Correcting Non-Chronological Autoregressive Music Generation
    Wayne Chi, Prachi Kumar, Suri Yaddanapudi, Rahul Suresh, and Umut Isik
    ICML 2020 Workshop, Machine Learning for Music Discovery (ML4MD), 2020
  5. Using a model of scheduler runtime to improve the effectiveness of scheduling embedded in execution
    Sarah Bhaskaran, Jagriti Agrawal, Steve Chien, and Wayne Chi
    Workshop on Integrated Execution (IntEx)/Goal Reasoning (GR), International Conference on Automated Planning and Scheduling (International Conference on Automated Planning and Scheduling IntEx/GP 2020), 2020

2019

  1. Supporting automation in spacecraft activity planning with simulation and visualization
    Basak B Alper Ramaswamy, Jagriti Agrawal, Wayne Chi, So Young Kim Castet, Scott Davidoff, and Steve Chien
    AIAA Scitech 2019 Forum, 2019
  2. Temporal brittleness analysis of task networks for planetary rovers
    Tiago Vaquero, Steve Chien, Jagriti Agrawal, Wayne Chi, and Terrance Huntsberger
    International Conference on Automated Planning and Scheduling, 2019
  3. Optimizing parameters for uncertain execution and rescheduling robustness
    Wayne Chi, Jagriti Agrawal, Steve Chien, Elyse Fosse, and Usha Guduri
    International Conference on Automated Planning and Scheduling, 2019
  4. Using Rescheduling and Flexible Execution to Address Uncertainty in Execution Duration for a Planetary Rover
    Jagriti Agrawal, Wayne Chi, and Steve Chien
    International Workshop on Planning and Scheduling for Space (IWPSS), 2019

2018

  1. Embedding a Scheduler in Execution for a Planetary Rover
    Wayne Chi, Steve A Chien, Jagriti Agrawal, Gregg Rabideau, Edward Benowitz, Daniel M Gaines, Elyse Fosse, Stephen Kuhn, and James Biehl
    International Conference on Automated Planning and Scheduling, 2018
  2. Using Squeaky Wheel Optimization to Derive Problem Specific Control Information for a One Shot Scheduler for a Planetary Rover
    Wayne Chi, Steve Chien, and Jagriti Agrawal
    International Conference on Automated Planning and Scheduling, 2018