微软Salesforce大模型容易在多轮对话中迷失方向.pdfVIP

LLMSGETLOSTINMULTI-TURNCONVERSATION

PhilippeLaban∗♢HiroakiHayashi∗♣YingboZhou♣JenniferNeville♢

♢MicrosoftResearch♣SalesforceResearch

{plaban,jenneville}@

{hiroakihayashi,yingbo.zhou}@

5ABSTRACT

2

0LargeLanguageModels(LLMs)areconversationalinterfaces.Assuch,LLMshavethepotentialto

2assisttheirusersnotonlywhentheycanfullyspecifythetaskathand,butalsotohelpthemdefine,

yexplore,andrefinewhattheyneedthroughmulti-turnconversationalexchange.Althoughanalysisof

aLLMconversationlogshasconfirmedthatunderspecificationoccursfrequentlyinuserinstructions,

LLMevaluationhaspredominantlyfocusedonthesingle-turn,fully-specifiedinstructionsetting.In

Mthiswork,weperformlarge-scalesimulationexperimentstocompareLLMperformanceinsingle-

9andmulti-turnsettings.Ourexperimentsconfirmthatallthetopopen-andclosed-weightLLMs

wetestexhibitsignificantlylowerperformanceinmulti-turnconversationsthansingle-turn,with

]anaveragedropof39%acrosssixgenerationtasks.Analysisof200,000+simulatedconversations

Ldecomposestheperformancedegradationintotwocomponents:aminorlossinaptitudeanda

Csignificantincreaseinunreliability.WefindthatLLMsoft