LLM Analysis

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

We propose AucArena to tests LLMs in auctions, showing they can strategize but with variable success, indicating potential for enhancement.

Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, Kyle Richardson

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

Adaptive Chameleon or Stubborn Sloth: Unraveling the Behavior of Large Language Models in Knowledge Clashes

We present the first comprehensive and controlled investigation into the behavior of large language models when encountering knowledge conflicts.

Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, Yu Su

Adaptive Chameleon or Stubborn Sloth: Unraveling the Behavior of Large Language Models in Knowledge Clashes

Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction

We propose a scientific analogical reasoning benchmark with structure abduction, SCAR, and show that large language models make reasonable scientific analogies after structure abduction.

Siyu Yuan, Jiangjie Chen, Xuyang Ge, Yanghua Xiao, Deqing Yang

Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction

Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge

We find that large language models (LLMs) speak too positively about negative commonsense knowledge, which is caused by statistical shortcuts and negation reporting bias from language modeling pre-training.

Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, Yanghua Xiao

Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge