进阶指南（Vincentzyx翻译）.docxVIP

来自 @GAN-er的进阶技巧 (Vincentzyx 翻译) Advanced tips from @ GAN-er (in English only): Tip 1: You may benefit by starting with a small batch size (within reason) and increasing it later. The reason is that a ** large batch size will give you a more accurate descent direction but it will also be costlier to calculate **, and when you just start, you care mostly about the general direction; no need to sacrifice speed for precision at that point. There are plenty of sources discussing the batch size, as an example you can check this one: /questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network 技巧1： 开始的时候使用比较小的Batch Size（合理范围内），后面再增大，可能会更好。因为更大的Batch Size虽然可以给你更好的Loss梯度下降的方向，不过计算的开销也会比较大，当你刚开始训练的时候，你只需要一个大致的下降方向，不需要因为精度而牺牲速度。有很多地方在讨论关于Batch Size的内容，下面就是一个，你可以去看看： /questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network Tip 2: Unlike the batch size that the only thing that does is affecting how accurate each step will be as far a the true gradient goes, the dimensions, actually, increase the complexity of your NN. As a rule, ** the more complex a network the better the resulting model **, but since nothing comes for free, ** the more complex the network the more time it will take to converge **. What you generally want is to ** _ figure out the max dimensions that you can use _ ** given your GPU's memory, and your desired max batch size. You can set the max batch size to something, say K, and then increase the dimensions until you get OOM errors. In the end, you will end up with a triplet, {batch size, ae_dims, ed_dims} Ideally, you would use 1024 and 85 for your autoencoder and encoder / decoder dimensions, but no card has enough memory for such a configuration even with batch size 1. Remember that unlike batch size that you can change at will, once you set up the dimensions you can not change them. Note that ** if you use a complex - high number of dimensions NN, in combination with a small batch size, it will