【DeepSeek市场报告】DeepSeek-V3技术报告.docx

DeepSeek-V3TechnicalReport

DeepSeek-AI

research@

Abstract

WepresentDeepSeek-V3,astrongMixture-of-Experts(MoE)languagemodelwith671Btotalparameterswith37Bactivatedforeachtoken.Toachieveefficientinferenceandcost-effectivetraining,DeepSeek-V3adoptsMulti-headLatentAttention(MLA)andDeepSeekMoEarchitec-tures,whichwerethoroughlyvalidatedinDeepSeek-V2.Furthermore,DeepSeek-V3pioneersanauxiliary-loss-freestrategyforloadbalancingandsetsamulti-tokenpredictiontrainingobjectiveforstrongerperformance.Wepre-trainDeepSeek-V3on14.8trilliondiverseandhigh-qualitytokens,followedbySupervisedFine-TuningandReinforcementLearningstagestofullyharnessitscapabilities.ComprehensiveevaluationsrevealthatDeepSeek-V3outperformsotheropen-sourcemodelsandachievesperformancecomparabletoleadingclosed-sourcemodels.Despiteitsexcellentperformance,DeepSeek-V3requiresonly2.788MH800GPUhoursforitsfulltraining.Inaddition,itstrainingprocessisremarkablystable.Throughouttheentiretrainingprocess,wedidnotexperienceanyirrecoverablelossspikesorperformanyrollbacks.Themodelcheckpointsareavailableat/deepseek-ai/DeepSeek-V3.

DeepSeek-

DeepSeek-V3 DeepSeek-V2.5 Qwen2.5-72B-Inst Llama-3.1-405B-Inst GPT-4o-0513 Claude-3.5-Sonnet-1022

59.1

51.6

42.0

39.2

75.9

90.2

80

Accuracy

Accuracy/Percentile(%)

40

20

0

MMLU-Pro

GPQA-Diamond

MATH500

AIME2024

Codeforces

SWE-benchVerified

Figure1|BenchmarkperformanceofDeepSeek-V3anditscounterparts.

PAGE

PAGE10

Contents

Introduction 4

Architecture 6

BasicArchitecture 6

Multi-HeadLatentAttention 7

DeepSeekMoEwithAuxiliary-Loss-FreeLoadBalancing 8

Multi-TokenPrediction 10

Infrastructures 11

ComputeClusters 11

TrainingFramework 12

DualPipeandComputation-CommunicationOverlap 12

EfficientImplementationofCross-NodeAll-to-AllCommunication 13

ExtremelyMemorySavingwithMinimalO