Large Language Models | Jiangjie Chen

InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

We propose InCharacter, a method using psychological scales to evaluate the personality fidelity of role-playing agents (RPAs) powered by large language models.

Xintao Wang, Yunze Xiao, Jen-Tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, Yanghua Xiao

InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick

We propose GumbelSoft to improve the diversity of text outputs from LLMs while maintaining high detectability, outperforming other watermarking methods in both aspects.

Jiayi Fu, Xuandong Zhao, Ruihan Yang, Yuansen Zhang, Jiangjie Chen, Yanghua Xiao

GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

We introduced TravelPlanner, a benchmark for assessing language agents’ planning abilities, showing that even advanced models like GPT-4 face difficulties with complex tasks.

Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction

We proposes EASYTOOL, a method that simplifies tool documentation into concise instructions, improving tool use by language models.

Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, Deqing Yang

EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

We propose AucArena to tests LLMs in auctions, showing they can strategize but with variable success, indicating potential for enhancement.

Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, Kyle Richardson

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models

We propose a multilingual idiom KB (IdiomKB) developed using LLMs to facilitate better idiomatic translation by smaller models by retrieving idioms’ figurative meanings.

Shuang Li, Jiangjie Chen, Siyu Yuan, Xinyi Wu, Hao Yang, Shimin Tao, Yanghua Xiao

Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models

Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts

We present the first comprehensive and controlled investigation into the behavior of large language models when encountering knowledge conflicts.

Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, Yu Su

Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts

Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction

We propose a scientific analogical reasoning benchmark with structure abduction, SCAR, and show that large language models make reasonable scientific analogies after structure abduction.

Siyu Yuan, Jiangjie Chen, Xuyang Ge, Yanghua Xiao, Deqing Yang

Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction

AnalogyKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base

A million-scale analogy KB derived from existing KGs, to enable large language models to achieve analogical reasoning skills.

Siyu Yuan, Jiangjie Chen, Changzhi Sun, Jiaqing Liang, Yanghua Xiao, Deqing Yang

AnalogyKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base

Distilling Script Knowledge from Large Language Models for Constrained Language Planning

We propose an over-generate-then-filter approach to improve large language models (LLMs) on constrained language planning, and use it to distill a novel constrained language planning dataset, CoScript.

Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Robert Jankowski, Yanghua Xiao, Deqing Yang

Distilling Script Knowledge from Large Language Models for Constrained Language Planning