Publications

PDFs available upon request.

2024

Open Problems in Technical AI Governance
Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lisa Soder, Onni Aarne, Lewis Hammond, Lujain Ibrahim, Alan Chan, Peter Wills, Markus Anderljung, Ben Garfinkel, Lennart Heim, Andrew Trask, Gabriel Mukobi, Rylan Schaeffer, Mauricio Baker, Sara Hooker, Irene Solaiman, Alexandra Sasha Luccioni, Nitarshan Rajkumar, Nicolas Moës, Jeffrey Ladish, Neel Guha, Jessica Newman, Yoshua Bengio, Tobin South, Alex Pentland, Sanmi Koyejo, Mykel J. Kochenderfer, and Robert Trager
Preprint (2024)

Prospector Heads: Generalized Feature Attribution for Large Models & Data
Gautam Machiraju, Alexander Derry, Arjun Desai, Neel Guha, Amir-Hossein Karimi, James Zou, Russ Altman, Christopher Ré, and Parag Mallick
International Conference on Machine Learning (2024)
[Code]

Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT
Jon Saad-Falcon, Daniel Y. Fu, Simran Arora, Neel Guha, and Christopher Ré
International Conference on Machine Learning (2024)
[Blog] [Code]

Building GenAI Benchmarks: A Case Study in Legal Applications
Neel Guha, Julian Nyarko, Daniel E. Ho, and Christopher Ré
In The Oxford Handbook on the Foundations and Regulation of Generative AI (Oxford University Press, 2024)

Understanding Liability Risk from Using Health Care Artificial Intelligence Tools
Michelle M. Mello, and Neel Guha
New England Journal of Medicine (2024)
[Policy Brief]

AI Regulation Has Its Own Alignment Problem: The Technical and Institutional Feasibility of Disclosure, Registration, Licensing, and Auditing
Neel Guha, Christie M. Lawrence, Lindsey A. Gailmard, Kit T. Rodolfa, Faiz Surani, Inioluwa Deborah Raji, Mariano-Florentino Cuéllar, Colleen Honigsberg, Percy Liang, and Daniel E. Ho
George Washington Law Review Symposium on Legally Disruptive Emerging Technologies (forthcoming) (2024)
[Policy Brief]

Private Enforcement in the States
Diego Zambrano, Neel Guha, Austin Peters, and Jeffrey Xia
University of Pennsylvania Law Review (2024)

2023

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li
Advances in Neural Information Processing Systems (2023)
[Code] [Website]

Embroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classification
Neel Guha, Mayee F Chen, Kush Bhatia, Azalia Mirhoseini, Frederic Sala, and Christopher Ré
Advances in Neural Information Processing Systems (2023)
[Blog] [Code]

Don’t Use a Cannon to Kill a Fly: An Efficient Cascading Pipeline for Long Documents
Zehua Li, Neel Guha, and Julian Nyarko
International Conference on AI and Law (2023)

Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda
Transactions on Machine Learning Research (2023)
[Website]

Ask Me Anything: A Simple Strategy for Prompting Language Models
Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré
International Conference on Learning Representations (2023)
[Code]

Gamesmanship in Modern Discovery Tech
Diego Zambrano, Neel Guha, and Peter Henderson
In Legal Tech and the Future of Civil Justice (Cambridge University Press, 2023)

ChatGPT and Physicians’ Malpractice Risk
Michelle M. Mello, and Neel Guha
In JAMA Health Forum (2023)

2022

Pile of Law: Learning Responsible Data Filtering from the Law and a 256gb Open-source Legal Dataset
Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho
Advances in Neural Information Processing Systems (2022)
[Website]

LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning
Neel Guha, Daniel E. Ho, Julian Nyarko, and Christopher Ré
Preprint (2022)

Vulnerabilities in Discovery Tech
Neel Guha, Peter Henderson, and Diego Zambrano
Harvard Journal of Law & Technology (2022)

2021

On the Opportunities and Risks of Foundation Models
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang
ArXiv (2021)

When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho
International Conference on AI and Law (2021)
[Website]

Leveraging administrative data for bias audits: Assessing disparate coverage with mobility data for COVID-19 policy
Amanda Coston, Neel Guha, Derek Ouyang, Lisa Lu, Alexandra Chouldechova, and Daniel E Ho
In ACM Conference on Fairness, Accountability, and Transparency (2021)

Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
Laurel J. Orr, Megan Leszczynski, Simran Arora, Sen Wu, Neel Guha, Xiao Ling, and Christopher Ré
Conference on Innovative Data Systems Research (2021)
[Blog] [Code] [Website]

2019

Machine Learning for AC Optimal Power Flow
Neel Guha, Zhecheng Wang, Matt Wytock, and Arun Majumdar
Climate Change Workshop at the International Conference on Machine Learning (2019)

One-Shot Federated Learning
Neel Guha, Ameet Talwalkar, and Virginia Smith
2nd Workshop on Machine Learning on the Phone and other Consumer Devices at Neural Information Processing Systems (2019)