2024年DeepSeek-V2模型技术报告经济高效的混合专家语言模型 英文版.pdfVIP

DeepSeek-V2:AStrong,Economical,andEfficient

Mixture-of-ExpertsLanguageModel

DeepSeek-AI

research@

Abstract

4

2

0WepresentDeepSeek-V2,astrongMixture-of-Experts(MoE)languagemodelcharacterizedby

2

economicaltrainingandefficientinference.Itcomprises236Btotalparameters,ofwhich21B

n

uareactivatedforeachtoken,andsupportsacontextlengthof128Ktokens.DeepSeek-V2adopts

JinnovativearchitecturesincludingMulti-headLatentAttention(MLA)andDeepSeekMoE.

9MLAguaranteesefficientinferencethroughsignificantlycompressingtheKey-Value(KV)cache

1

intoalatentvector,whileDeepSeekMoEenablestrainingstrongmodelsataneconomical

]costthroughsparsecomputation.ComparedwithDeepSeek67B,DeepSeek-V2achieves

L

Csignificantlystrongerperformance,andmeanwhilesaves42.5%oftrainingcosts,reducesthe

s.KVcacheby93.3%,andbooststhemaximumgenerationthroughputto5.76times.Wepretrain

cDeepSeek-V2onahigh-qualityandmulti-sourcecorpusconsistingof8.1Ttokens,andfurther

[

performSupervisedFine-Tuning(SFT)andReinforcementLearning(RL)tofullyunlockits

5potential.Evaluationresultsshowthat,evenwithonly21Bactivatedparameters,DeepSeek-V2

v

4anditschatversionsstillachievetop-tierperformanceamongopen-sourcemodels.Themodel

3checkpointsareavailableat/deepseek-ai/DeepSeek-V2.

4

4

0TrainingCosts(KGPUHours/TTokens)

.