LLM Analysis

We study how LLMs handle irrelevant information and find they struggle with content that is semantically related but ultimately not pertinent, highlighting the limitations of current systems in filtering out such distractions.

We propose a simulation to study language’s influence on collective behavior by having agents engage in free chat within various narrative scenarios, with findings suggesting that greater information exchange promotes more orderly and meaningful emergent behaviors.

We propose AucArena to tests LLMs in auctions, showing they can strategize but with variable success, indicating potential for enhancement.

We present the first comprehensive and controlled investigation into the behavior of large language models when encountering knowledge conflicts.

We propose a scientific analogical reasoning benchmark with structure abduction, SCAR, and show that large language models make reasonable scientific analogies after structure abduction.

We find that large language models (LLMs) speak too positively about negative commonsense knowledge, which is caused by statistical shortcuts and negation reporting bias from language modeling pre-training.