Dropout​ Dropout 是一种在神经网络训练过程中常用的正则化技术其核心思想是随机“丢弃”部分神经元即将其输出置为零从而防止模型过度依赖某些特定特征增强泛化能力有效抑制过拟合。传入Dropout的数据词嵌入位置嵌入后:tensor([[[-0.0364,0.0127,0.0204,-0.0125,-0.0314,-0.0665],[-0.0411,-0.0429,-0.0012,-0.0266,-0.0179,0.0581],[-0.0073,0.0003,0.0268,-0.0152,-0.0243,-0.0015],[-0.0009,-0.0107,-0.0140,-0.0276,-0.0088,-0.0626]],[[-0.0374,0.0206,-0.0100,0.0124,-0.0243,-0.0383],[0.0432,0.0088,0.0093,-0.0341,-0.0013,0.1053],[0.0359,0.0300,0.0243,0.0110,-0.0061,0.0303],[-0.0443,0.0572,-0.0235,0.0327,-0.0307,-0.0337]]],grad_fnAddBackward0)词嵌入位置嵌入后形状:torch.Size([2,4,6])随机失活后Dropout后:tensor([[[-0.0404,0.0141,0.0226,-0.0139,-0.0348,-0.0739],[-0.0457,-0.0476,-0.0013,-0.0296,-0.0199,0.0646],[-0.0000,0.0003,0.0298,-0.0169,-0.0270,-0.0000],[-0.0010,-0.0119,-0.0000,-0.0306,-0.0000,-0.0695]],[[-0.0416,0.0229,-0.0111,0.0137,-0.0270,-0.0426],[0.0480,0.0098,0.0104,-0.0379,-0.0000,0.0000],[0.0000,0.0334,0.0270,0.0123,-0.0068,0.0336],[-0.0492,0.0635,-0.0261,0.0364,-0.0000,-0.0374]]],grad_fnMulBackward0)Dropout后形状:torch.Size([2,4,6])​ 第1句第1字的意思为[ -0.0364, 0.0127, 0.0204, -0.0125, -0.0314, -0.0665 ],随机失活后得到[-0.0404, 0.0141, 0.0226, -0.0139, -0.0348, -0.0739]。​ 可以看到第1句第1个字并没有被失活但是每个数值依然都乘了1/0.9,比如-0.0364 × 1.1111 ≈ -0.0404。​ 再观察第1条句子第3个字的失活后结果[-0.0000, 0.0003, 0.0298, -0.0169, -0.0270, -0.0000], 其原始向量为[-0.0073, 0.0003, 0.0268, -0.0152, -0.0243, -0.0015]。​ 这里第1维和第6维被丢弃输出为 0而其他维度则同样乘以 1/0.91/0.9即0.0003×1.1111≈0.0003,0.0268×1.1111≈0.0298,−0.0152×1.1111≈−0.0169,−0.0243×1.1111≈−0.0270​ Dropout的作用是抑制训练过程中某些神经元或注意力头过度激活从而提升模型在新数据上的表现。——谙弆悕