千亿参数+LLM+的训练效率优化-零一万物.docx

千亿参数LLM的训练效率优化

演讲人：张力寰

零一万物/AIInfra分布式训练优化架构师

模型训练硬件利用率影响因素

Llama3.1

?92页的技术报告

?最大405B的模型

?16KH100训练集群

?54天的预训练,466次任务中断

?MFU:40%左右

?Goodput:90%左右的有效训练时间占比

htps:///pdf/2407.21783

MFU

?ModelFLOPSUtilization，模型算力利用率

?FLOPS(FloatingPointOperationsPerSecond)

????rv?怀t?r?ug??ut

?FU=t???r?ti??????i?u?t?r?ug??ut

htps:///pdf/2407.21783htps:///en-us-tensor-core/nvidia-tensor-core-gpu-datasheet

Goodput

?AmetrictomeasureAIsystemefficiency(Google)

?SchedulingGoodput

?RuntimeGoodput

?ProgramGoodput

htps:///blog/products/ai-machine-learning/goodput-metric-as-measure-of-ml-productivity

分布式训练效率提升

DataParallelism

?DDP:DistributedDataParallel

?模型较小，数据量较大

htps:///nemo-framework/user-guide/latest/nemotoolkit/features/paralelisms.html

TensorParallelism

?Megatron-LM-1

?拆分均匀，但通信量大

h

htps:///pdf/1909.08053

PipelineParallelism

?GPipe(Google),PipeDream(Microsoft)

?Megatron-LM-2

?通信量低，但会引入bubble

htps:///pdf/1806.03377htps:///pdf/181.06965htps:///pdf/2104.04473

ExpertParallelism

?SwitchTransformers(Google)

?MegatronExpertParallelism

htps:///pdf/2101.03961htps:///nemo-framework/user-guide/latest/nemotoolkit/features/paralelisms.html

ContextParallelism

htps:///megatron-core/developer-guide/latest/api-guide/context_paralel.html

RingAttention

?RingAttentionwithBlockwiseTransformersforNear-InfiniteContext(UCBerkeley)

?(a)Outerloop:computingblockwiseattentionamongdevices

?(b)Innerloop:everydevicecomputesblockwiseattentionandfeedforwardoperations

h

htps:///pdf/2310.01889

RingAttention

?基本原理：Onlinesoftmax

?性能问题：负载不均衡

RingAttention

?原版VS负载均衡版

SWA+CP

?SlidingWindowAttention+ContextParallel

?不同情况序列长度的问题

?如何复用高性能Attention算子（如FlashAttention）

?模基共建

?如何与FullAttention+CP混合使用

其它优化

?通信计算并行

?分布式优化器显存优化

?TP的MPI依赖解耦

?

...

FP8训练经验分享

aFP8训练简介

?