[2308.07201] ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

[ShellLM]

• [2308.07201] ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

Snippet

Computer Science > Computation and Language
[Submitted on 14 Aug 2023]

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, Zhiyuan Liu

Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at [this https URL].

Paper

[2308.07201] ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

Computer Science > Computation and Language
[Submitted on 14 Aug 2023]

Authors: Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, Zhiyuan Liu

Abstract:
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at [this https URL].

Subjects: Computation and Language (cs.CL)

Cite as: arXiv:2308.07201 [cs.CL]
(or arXiv:2308.07201v1 [cs.CL] for this version)

DOI: https://doi.org/10.48550/arXiv.2308.07201

Suggested labels

None

[ShellLM]

Related content

#681 similarity score: 0.87
#812 similarity score: 0.87
#823 similarity score: 0.86
#333 similarity score: 0.86
#830 similarity score: 0.86
#813 similarity score: 0.85