John J. Guerrerio

I’m a first-year PhD student at the Center for Language and Speech Processing at Johns Hopkins University, where I’m advised by Ben Van Durme and Anqi Liu. My work is generously supported by the NSF Graduate Research Fellowship.

Broadly, my research focuses on the reliability and safety of LLMs. More specifically, I am interested in:

Uncertainty Quantification: How can we produce well-calibrated, trustworthy estimates of an LLM’s uncertainty?
Robust Reasoning: How can we make LLM reasoning more robust, especially through Bayesian methods and latent reasoning in continuous space?
Benchmarking: How can we measure complex, safety-critical properties of LLMs, such as the degree to which they encode social bias?

Before Hopkins, I graduated as valedictorian from Dartmouth College, where I studied Computer Science and Chinese. My honors thesis with Soroush Vosoughi, Towards Robust and Holistic Bias Benchmarking for Large Language Models, received High Honors in Computer Science. I also worked with Yaoqing Yang on a Bayesian framework for improving LLM reasoning and on using loss landscape metrics for more efficient hyperparameter tuning.

As an undergraduate, I was supported by a Goldwater Scholarship and was a Rhodes Scholarship finalist. I also worked across a range of fields: security and privacy with Tim Pierson and O. Sami Saydjari at Dartmouth, NLP for health with Zhiyong Lu and Qingyu Chen at the NIH, ML for national security with Greg Canal and I-Jeng Wang at JHU APL, and economic consulting at Cornerstone Research.

news

Aug 31, 2026	Started my PhD at JHU!
Jul 05, 2026	Presented When Debiasing Backfires at ACL 2026
Apr 12, 2026	Received the NSF Graduate Research Fellowship

selected publications

ACL Findings
When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation

Yahan Zheng, John J. Guerrerio, Soroush Vosoughi, and Weicheng Ma

In Findings of the Association for Computational Linguistics: ACL 2026, 2026

Abs Bib PDF

Preprocessing-based methods for stereotype mitigation, such as pre-/post-training on debiased corpora, are widely used in NLP. While these approaches reduce measurable stereotypes for targeted groups, we find they often induce unintended shifts-side effects, where stereotyping or counter-stereotyping can increase relative to neutral baselines for other demographics, including across unrelated demographic categories. We demonstrate these side effects across two model families (encoder-only and decoder-only), multiple preprocessing strategies (removing stereotypical sentences, removing group mentions, and swapping group references), and both pre- and post-training at different data scales on Wikipedia. Standard benchmarks frequently miss these shifts. Using attention-rollout analysis, we observe that such side effects are not accompanied by large changes in attention flow, complicating mechanistic explanations. We discuss implications for evaluation, provide actionable diagnostics, and argue for side-effect-aware, transparent mitigation practices.
@inproceedings{whenDebiasingBackfires, title = {When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation}, author = {Zheng, Yahan and Guerrerio, John J. and Vosoughi, Soroush and Ma, Weicheng}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2026}, year = {2026}, }
EMNLP
Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

Weicheng Ma^*, John J. Guerrerio^*, and Soroush Vosoughi

In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025

Abs Bib PDF

Research on stereotypes in large language models (LLMs) has largely focused on English-speaking contexts, due to the lack of datasets in other languages and the high cost of manual annotation in underrepresented cultures. To address this gap, we introduce a cost-efficient human-LLM collaborative annotation framework and apply it to construct EspanStereo, a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries across Europe and Latin America. EspanStereo captures both well-documented stereotypes from prior literature and culturally specific biases absent from English-centric resources. Using LLMs to generate candidate stereotypes and in-culture annotators to validate them, we demonstrate the framework’s effectiveness in identifying nuanced, region-specific biases. Our evaluation of Spanish-supporting LLMs using EspanStereo reveals significant variation in stereotypical behavior across countries, highlighting the need for more culturally grounded assessments. Beyond Spanish, our framework is adaptable to other languages and regions, offering a scalable path toward multilingual stereotype benchmarks. This work broadens the scope of stereotype analysis in LLMs and lays the groundwork for comprehensive cross-cultural bias evaluation.
@inproceedings{espanStereo, title = {Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration}, author = {Ma, Weicheng and Guerrerio, John J. and Vosoughi, Soroush}, booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing}, year = {2025}, }
NeurIPS BDU
Decision-Driven Calibration for Cost-Sensitive Uncertainty Quantification

Greg Canal, Vladimir Leung, John J. Guerrerio, Philip Sage, and I-Jeng Wang

In NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty, 2024

Abs Bib PDF

In recent years, the ability of artificial intelligence (AI) systems to quantify their uncertainty has become paramount in building trustworthy AI. In standard uncertainty quantification (UQ), AI uncertainty is calibrated such that the confidence of its predictions matches the statistics of the underlying data distribution. However, this method of calibration does not take into consideration the direct influence of UQ on the subsequent actions taken by downstream decision-makers. Here we demonstrate an alternate, decision-driven method of UQ calibration that explicitly minimizes the incurred costs of downstream decisions. After formulating decision-driven calibration as an optimization problem with respect to a known decision-maker, we show in a simulated search-and-rescue scenario how decision-driven temperature scaling can lead to lower incurred decision costs.
@inproceedings{decisionDrivenCalibration, title = {Decision-Driven Calibration for Cost-Sensitive Uncertainty Quantification}, author = {Canal, Greg and Leung, Vladimir and Guerrerio, John J. and Sage, Philip and Wang, I-Jeng}, booktitle = {NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty}, year = {2024}, }