斯坦福大学：大语言模型简介（英）-Introduction to Large Language Models.docx

Large

Large

LanguageModels

IntroductiontoLargeLanguageModels

Languagemodels

?Rememberthesimplen-gramlanguagemodel

?Assignsprobabilitiestosequencesofwords

?Generatetextbysamplingpossiblenextwords

?Istrainedoncountscomputedfromlotsoftext

?Largelanguagemodelsaresimilaranddifferent:

?Assignsprobabilitiestosequencesofwords

?Generatetextbysamplingpossiblenextwords

?Aretrainedbylearningtoguessthenextword

Largelanguagemodels

?Eventhroughpretrainedonlytopredictwords

?Learnalotofusefullanguageknowledge

?Sincetrainingonalotoftext

Threearchitecturesforlargelanguagemodels

Decoders

GPT,Claude,Llama

Mixtral

Encoders

BERTfamily,HuBERT

Encoder-decoders

Flan-T5,Whisper

Encoders

Manyvarieties!

?Popular:MaskedLanguageModels(MLMs)

?BERTfamily

?Trainedbypredictingwordsfromsurroundingwordsonbothsides

?Areusuallyfinetuned(trainedonsuperviseddata)forclassificationtasks.

Encoder-Decoders

?Trainedtomapfromonesequencetoanother

?Verypopularfor:

?machinetranslation(mapfromonelanguagetoanother)

?speechrecognition(mapfromacousticstowords)

Large

Large

LanguageModels

IntroductiontoLargeLanguageModels

Large

Large

LanguageModels

LargeLanguageModels:Whattaskscantheydo?

Bigidea

Manytaskscanbeturnedintotasksofpredictingwords!

Thislecture:decoder-onlymodels

Alsocalled:

?CausalLLMs

?AutoregressiveLLMs

?Left-to-rightLLMs

?Predictwordslefttoright

ConditionalGeneration:Generatingtextconditionedonprevioustext!

CompletionText

TransformerBlocksEncoderSolongandthanksforallthethe

TransformerBlocks

Encoder

Solongandthanksfor

all

the

the

Language

Modeling

Head

…

…

+i

E

+i

E

+i

E

+i

E

+i

E

Softmax

Unencoderlayer\U/

+i

E

+i

E

U

logits

all

Pre?xText

Manyp