Jeff Dean：快速训练大规模神经网络的技术.docx

Techniques and Systems for Training Large Neural Networks Quickly Je? Dean Google ! Joint work with many colleagues at Google How Can We Build More Intelligent Computer Systems? Need to perceive and understand the world Basic speech and vision capabilities? Language understanding? User behavior prediction? … How can we do this? ? ? Cannot write algorithms for each task we want to accomplish separately? Need to write general algorithms that learn from observations Can we build systems that:? ? Generate understanding from raw data? ? Solve dif?cult problems to improve Google’s products? ? ? Minimize software engineering effort ? Advance state of the art in what is possible Functions Arti?cial Neural Nets Can Learn Input Output Pixels: Audio: “lion” “pa lo ahl toe res taur aun ts” P(click on doc) query, doc “Hello, how are you?” “Bonjour, comment allez-vous?” “A close up of a small child holding a stuffed animal” Pixels: Same Underlying System Works for Many Problems ? ? Nice property: same general approach works for many problems/domains? ! Same underlying infrastructure can be used to train and use many kinds of models Plenty of Data ? ? ? ? ? ? Text: trillions of words of English + other languages? Visual: billions of images and videos? Audio: thousands of hours of speech per day? User activity: queries, result page clicks, map requests, etc.? Knowledge graph: billions of labelled relation triples? ... More Data + Bigger Model = Better Results ? ? True across wide range of areas? ! Many problems have virtually limitless data, so only issue is how big a model we can train Research Objective: Minimizing Time to Results ? ? We want results of experiments quickly? “Patience threshold”: No one wants to wait more than a few days or a week for a result? ? ? Signi?cantly affects scale of problems that can be tackled? We sometimes optimize for experiment turnaround time, rather than absolute minimal system resources for performing the experiment Train in a day what takes a