谷歌2024大语言模型合成数据的最佳实践和经验教训报告英文版26页.pdf

2024-04-10

BestPracticesandLessonsLearnedon

SyntheticDataforLanguageModels

RuiboLiu1,JerryWei1,FangyuLiu1,ChengleiSi2,YanzheZhang3,JinmengRao1,StevenZheng1,Daiyi

Peng1,DiyiYang2,DennyZhou1andAndrewM.Dai1

1GoogleDeepMind,2StanfordUniversity,3GeorgiaInstituteofTechnology

ThesuccessofAImodelsreliesontheavailabilityoflarge,diverse,andhigh-qualitydatasets,which

canbechallengingtoobtainduetodatascarcity,privacyconcerns,andhighcosts.Syntheticdatahas

emergedasapromisingsolutionbygeneratingartificialdatathatmimicsreal-worldpatterns.This

paperprovidesanoverviewofsyntheticdataresearch,discussingitsapplications,challenges,and

4futuredirections.Wepresentempiricalevidencefrompriorarttodemonstrateitseffectivenessand

2

0highlighttheimportanceofensuringitsfactuality,fidelity,andunbiasedness.Weemphasizetheneedfor

2responsibleuseofsyntheticdatatobuildmorepowerful,inclusive,andtrustworthylanguagemodels.

r

p

A1.Introduction

1

1

]

L

C

.

s

c

[

1

v

3

0

5

7

0

.

4

0

4

2

:

v

i

X

r

a

Figure1|OnesyntheticimagegeneratedbyImagen(Sahariaetal.,2022a)v2.0,withaprompt

includingthefollowingdescription:“Inaroboticsfactory,humanoidrobotscollaborateonanassembly

linetodesign,fabricate,test,andassemblenewrobots.Thenewrobotstheyaremanufacturinglook

similartothoseroboticworkerswhoarecreatingthem.”Wealsoaddedsomestylecontrollingtext

fromaestheticconsiderations.

Correspondingauthor(s):ruiboliu@

©2024GoogleDeepMind.Allrightsreserved

BestPrac