《神经网络的梯度推导与代码验证》之FNN（DNN）前向和反向传播过程的代码验证

tech2024-02-20 56

在《神经网络的梯度推导与代码验证》之FNN（DNN）的前向传播和反向梯度推导中，我们学习了FNN（DNN）的前向传播和反向梯度求导，但知识仍停留在纸面。本篇章将基于深度学习框架tensorflow验证我们所得结论的准确性，以便将抽象的数学符号和实际数据结合起来，将知识固化。更多相关内容请见《神经网络的梯度推导与代码验证》系列介绍。

需要用到的库有tensorflow和numpy，其中tensorflow其实版本>=2.0.0就行

import tensorflow as tf import numpy as np

然后是定义下面两个要用到的函数，一个是计算mse，另外一个是计算sigmoid的导数：

复制代码

mse

def get_mse(y_pred, y_true): return 0.5 * tf.reduce_sum((y_pred - y_true)**2)

sigmoid的导数

def d_sigmoid(x): “”" sigmoid(x)的导数 = sigmoid(x) * （1-sigmoid(x)） :param x: :return: sigmoid(x)的导数 “”" return tf.math.sigmoid(x) * (1 - tf.math.sigmoid(x)) 复制代码

接着是随便产生一条样本数据：

x = np.array([[1, 2]]).astype(np.float32) y_true = np.array([[0.3, 0.5, 0.2]]).astype(np.float32)

x的是2维的，输出y是3维的

x.shape Out[5]: (1, 2) y_true.shape Out[6]: (1, 3)

有了一条样本之后，我们开始写前向传播的代码：

复制代码 1 with tf.GradientTape(persistent=True) as t: 2 # -----hidden l1------------- 3 # DNN layer1 4 l1 = tf.keras.layers.Dense(4) 5 6 # 输入经过DNN layer1得到输出z_l1 7 z_l1 = l1(x) 8 # 跟踪变量z_l1，用于随后计算其梯度 9 t.watch([z_l1]) 10 # DNN layer1的输出再经过激活函数sigmoid 11 a_l1 = tf.math.sigmoid(z_l1) 12 # 跟踪变量a_l1 ，用于随后计算其梯度 13 t.watch([a_l1]) 14 # ------hidden l2------------ 15 l2 = tf.keras.layers.Dense(3) 16 17 z_l2 = l2(a_l1) 18 t.watch([z_l2]) 19 a_l2 = tf.math.sigmoid(z_l2) 20 t.watch([a_l2]) 21 # -------计算loss----------- 22 loss = get_mse(a_l2, y_true) 复制代码上面是一个两层的FNN（DNN）网络，激活函数都是sigmoid

这里 tf.GradientTape(persistent=True) ，t.watch()是用于后面计算变量的导数用的，不太熟悉的可参考tensorflow官方给出的关于这部分的教程（自动微分）。

这里为方便起见我就直接用tf.keras.layers.Dense()来创建DNN层了，tensorflow官方的教程也推荐用这种方法快速定义layer。

如果要看某一层内部的weights和bias也比较容易

复制代码 l1.kernel Out[7]: <tf.Variable ‘dense/kernel:0’ shape=(2, 4) dtype=float32, numpy= array([[-0.96988726, -0.84827805, 0.312042 , 0.8871379 ], [ 0.6567688 , -0.29099226, -0.80029106, -0.15143871]], dtype=float32)> l1.bias Out[8]: <tf.Variable ‘dense/bias:0’ shape=(4,) dtype=float32, numpy=array([0., 0., 0., 0.], dtype=float32)> 复制代码 ----------前向传播的验证----------

下面来验证上面代码第7+11行代码是否符合DNN的前传规则：

tf.math.sigmoid(tf.matmul(x, l1.kernel) + l1.bias) Out[14]: <tf.Tensor: shape=(1, 4), dtype=float32, numpy=array([[0.585077 , 0.19305778, 0.2161 , 0.64204717]], dtype=float32)> a_l1 Out[15]: <tf.Tensor: shape=(1, 4), dtype=float32, numpy=array([[0.585077 , 0.19305778, 0.2161 , 0.64204717]], dtype=float32)>

看来tf.keras.layers.Dense确实是实现了下面的计算公式：

⎡⎣⎢⎢⎢⎢a21a22a23a24⎤⎦⎥⎥⎥⎥=σ⎛⎝⎜⎜⎜⎜⎡⎣⎢⎢⎢⎢w211w221w231w241w212w222w232w242w213w223w233w243⎤⎦⎥⎥⎥⎥⎡⎣⎢x1x2x3⎤⎦⎥+⎡⎣⎢⎢⎢⎢b21b22b23b24⎤⎦⎥⎥⎥⎥⎞⎠⎟⎟⎟⎟

这里l1层激活函数默认是linear，sigmoid激活函数被我单独拿了出来（见前传部分的代码第11行），方便计算梯度的时候好做分解。

----------反向梯度计算的验证----------

接下来就是验证反向梯度求导公式的时候了：

复制代码 1 # 注意的是，在tensorflow里，W变量矩阵和数学推导的是互为转置的关系，所以在验证的时候，要注意转置关系的处理 2 # ------dl_da2------ sigmoid(x)的导数 = sigmoid(x) * （1-sigmoid(x)） 3 dl_da2 = t.gradient(loss, a_l2) 4 my_dl_da2 = (a_l2 - y_true) 5 # ------dl_dz2--------- 6 dl_dz2 = t.gradient(loss, z_l2) 7 my_dl_dz2 = my_dl_da2 * d_sigmoid(z_l2) 8 # -------dl_dW2-------- 9 dl_dW2 = t.gradient(loss, l2.kernel) 10 my_dl_W2 = np.matmul(a_l1.numpy().transpose(), my_dl_dz2) 11 # -------dl_db2-------- 12 dl_db2 = t.gradient(loss, l2.bias) 13 my_dl_db2 = my_dl_dz2 14 # -------dl_dz1--------- 15 dl_dz1 = t.gradient(loss, z_l1) 16 my_dl_dz1 = np.matmul(my_dl_dz2, l2.weights[0].numpy().transpose()) * d_sigmoid(z_l1) 17 # -------dl_dW1--------- 18 dl_dW1 = t.gradient(loss, l1.kernel) 19 my_dl_dW1 = np.matmul(x.transpose(), my_dl_dz1) 20 # -------dl_db1---------- 21 dl_db1 = t.gradient(loss, l1.bias) 22 my_dl_db1 = my_dl_dz1 复制代码

上面反向梯度计算的对象的顺序跟前先传播的顺序是正好相反的，因为这样方便进行梯度计算的时候，靠前的层的参数的梯度能够用得到靠后的层的梯度计算结果而不必从头开始计算，这也是反向梯度传播名字的由来，这点在上面代码中也能够体现出来。

注意：在tensorflow里，W变量矩阵和数学推导的是互为转置的关系，所以在验证的时候，要注意转置关系的处理。举个例子，l1.kernel的shape是(2, 4)，即(input_dim, output_dim)这样的格式，说明在tensorflow是按照output=xTW + b这种方式做前传计算的，即输入xT是一个行向量而非列向量

而我们在数学推导上，习惯写成output=WTx + b，其中x默认是列向量而非行向量。于是在做验证的时候，需要对一些变量进行一下转置操作，即上面的 .transpose()操作，但大体并不影响公式的验证。

回到上面的代码部分。dl_da2 = t.gradient(loss, a_l2)表示用tensorflow微分工具求得的∂l∂a2；而带my_前缀的则是根据《神经网络的梯度推导与代码验证》之FNN（DNN）的前向传播和反向梯度推导中得到的结论手动计算出来的结果。我们依次对比一下是否有区别。

关于∂l∂a2，根据我们得到的公式，它满足：

∂l∂aL=aL−y

代码验证的结果是：

t.gradient(loss, a_l2) Out[17]: <tf.Tensor: shape=(1, 3), dtype=float32, numpy=array([[ 0.1497804 , -0.05124322, 0.23775901]], dtype=float32)> a_l2 - y_true Out[18]: <tf.Tensor: shape=(1, 3), dtype=float32, numpy=array([[ 0.1497804 , -0.05124322, 0.23775901]], dtype=float32)>

没有问题，下一个是∂l∂z2，根据公式，它满足：

daL=dσ(zL)=σ′(zL)⊙dzL=diag(σ′(zL))dzL

代码验证的结果是：

t.gradient(loss, z_l2) Out[21]: <tf.Tensor: shape=(1, 3), dtype=float32, numpy=array([[ 0.03706735, -0.01267625, 0.05851868]], dtype=float32)> my_dl_da2 * d_sigmoid(z_l2) Out[22]: <tf.Tensor: shape=(1, 3), dtype=float32, numpy=array([[ 0.03706735, -0.01267625, 0.05851869]], dtype=float32)>

也没有问题，其中d_sigmoid()函数是前面定义好的，用来求sigmoid导数的。至于上面0.05851868 v.s. 0.05851869的问题，我觉得单纯只是两种代码实现过程中调用的底层方式不同导致的不同而已。 http://bbs.quyouji.com/topic.asp?id=582 http://bbs.quyouji.com/topic.asp?id=583 http://bbs.quyouji.com/topic.asp?id=584 http://bbs.quyouji.com/topic.asp?id=585 http://bbs.quyouji.com/topic.asp?id=586 http://bbs.quyouji.com/topic.asp?id=587 http://bbs.quyouji.com/topic.asp?id=588 http://bbs.quyouji.com/topic.asp?id=589 http://bbs.quyouji.com/topic.asp?id=590 http://bbs.quyouji.com/topic.asp?id=591 http://bbs.quyouji.com/topic.asp?id=592 http://bbs.quyouji.com/topic.asp?id=593 http://bbs.quyouji.com/topic.asp?id=594 http://bbs.quyouji.com/topic.asp?id=595 http://bbs.quyouji.com/topic.asp?id=596 http://bbs.quyouji.com/topic.asp?id=597 http://bbs.quyouji.com/topic.asp?id=598 http://bbs.quyouji.com/topic.asp?id=599 http://bbs.quyouji.com/topic.asp?id=600 http://bbs.quyouji.com/topic.asp?id=601 http://bbs.quyouji.com/topic.asp?id=602 http://bbs.quyouji.com/topic.asp?id=603 http://bbs.quyouji.com/topic.asp?id=604 http://bbs.quyouji.com/topic.asp?id=605 http://bbs.quyouji.com/topic.asp?id=606 http://bbs.quyouji.com/topic.asp?id=607 http://bbs.quyouji.com/topic.asp?id=608 http://bbs.quyouji.com/topic.asp?id=609 http://bbs.quyouji.com/topic.asp?id=610 http://bbs.quyouji.com/topic.asp?id=611 http://bbs.quyouji.com/topic.asp?id=612 http://bbs.quyouji.com/topic.asp?id=613 http://bbs.quyouji.com/topic.asp?id=614 http://bbs.quyouji.com/topic.asp?id=615 http://bbs.quyouji.com/topic.asp?id=616 http://bbs.quyouji.com/topic.asp?id=617 http://bbs.quyouji.com/topic.asp?id=618 http://bbs.quyouji.com/topic.asp?id=619 http://bbs.quyouji.com/topic.asp?id=620 http://bbs.quyouji.com/topic.asp?id=621 http://bbs.quyouji.com/topic.asp?id=622 http://bbs.quyouji.com/topic.asp?id=623 http://bbs.quyouji.com/topic.asp?id=624 http://bbs.quyouji.com/topic.asp?id=625 http://bbs.quyouji.com/topic.asp?id=626 http://bbs.quyouji.com/topic.asp?id=627 http://bbs.quyouji.com/topic.asp?id=628 http://bbs.quyouji.com/topic.asp?id=629 http://bbs.quyouji.com/topic.asp?id=630 http://bbs.quyouji.com/topic.asp?id=631 http://bbs.quyouji.com/topic.asp?id=632 http://bbs.quyouji.com/topic.asp?id=633 http://bbs.quyouji.com/topic.asp?id=634 http://bbs.quyouji.com/topic.asp?id=635 http://bbs.quyouji.com/topic.asp?id=636 http://bbs.quyouji.com/topic.asp?id=637 http://bbs.quyouji.com/topic.asp?id=638 http://bbs.quyouji.com/topic.asp?id=639 http://bbs.quyouji.com/topic.asp?id=640 http://bbs.quyouji.com/topic.asp?id=641 http://bbs.quyouji.com/topic.asp?id=642 http://bbs.quyouji.com/topic.asp?id=643 http://bbs.quyouji.com/topic.asp?id=644 http://bbs.quyouji.com/topic.asp?id=645 http://bbs.quyouji.com/topic.asp?id=646 http://bbs.quyouji.com/topic.asp?id=647 http://bbs.quyouji.com/topic.asp?id=648 http://bbs.quyouji.com/topic.asp?id=649 http://bbs.quyouji.com/topic.asp?id=650 http://bbs.quyouji.com/topic.asp?id=651 http://bbs.quyouji.com/topic.asp?id=652 http://bbs.quyouji.com/topic.asp?id=653 http://bbs.quyouji.com/topic.asp?id=654 http://bbs.quyouji.com/topic.asp?id=655 http://bbs.quyouji.com/topic.asp?id=656 http://bbs.quyouji.com/topic.asp?id=657 http://bbs.quyouji.com/topic.asp?id=658 http://bbs.quyouji.com/topic.asp?id=659 http://bbs.quyouji.com/topic.asp?id=660 http://bbs.quyouji.com/topic.asp?id=661 http://bbs.quyouji.com/topic.asp?id=662 http://bbs.quyouji.com/topic.asp?id=663 http://bbs.quyouji.com/topic.asp?id=664 http://bbs.quyouji.com/topic.asp?id=665 http://bbs.quyouji.com/topic.asp?id=666 http://bbs.quyouji.com/topic.asp?id=667 http://bbs.quyouji.com/topic.asp?id=668 http://bbs.quyouji.com/topic.asp?id=669 http://bbs.quyouji.com/topic.asp?id=670 http://bbs.quyouji.com/topic.asp?id=671 http://bbs.quyouji.com/topic.asp?id=672 http://bbs.quyouji.com/topic.asp?id=673 http://bbs.quyouji.com/topic.asp?id=674 http://bbs.quyouji.com/topic.asp?id=675 http://bbs.quyouji.com/topic.asp?id=676 http://bbs.quyouji.com/topic.asp?id=677 http://bbs.quyouji.com/topic.asp?id=678 http://bbs.quyouji.com/topic.asp?id=679 http://bbs.quyouji.com/topic.asp?id=680 http://bbs.quyouji.com/topic.asp?id=681 http://bbs.quyouji.com/topic.asp?id=682 http://bbs.quyouji.com/topic.asp?id=683 http://bbs.quyouji.com/topic.asp?id=684 http://bbs.quyouji.com/topic.asp?id=685 http://bbs.quyouji.com/topic.asp?id=686 http://bbs.quyouji.com/topic.asp?id=687 http://bbs.quyouji.com/topic.asp?id=688 http://bbs.quyouji.com/topic.asp?id=689 http://bbs.quyouji.com/topic.asp?id=690 http://bbs.quyouji.com/topic.asp?id=691 http://bbs.quyouji.com/topic.asp?id=692 http://bbs.quyouji.com/topic.asp?id=693 http://bbs.quyouji.com/topic.asp?id=694 http://bbs.quyouji.com/topic.asp?id=695 http://bbs.quyouji.com/topic.asp?id=696 http://bbs.quyouji.com/topic.asp?id=697 http://bbs.quyouji.com/topic.asp?id=698 http://bbs.quyouji.com/topic.asp?id=699 http://bbs.quyouji.com/topic.asp?id=700 http://bbs.quyouji.com/topic.asp?id=701 http://bbs.quyouji.com/topic.asp?id=702 http://bbs.quyouji.com/topic.asp?id=703 http://bbs.quyouji.com/topic.asp?id=704 http://bbs.quyouji.com/topic.asp?id=705 http://bbs.quyouji.com/topic.asp?id=706 http://bbs.quyouji.com/topic.asp?id=707 http://bbs.quyouji.com/topic.asp?id=708 http://bbs.quyouji.com/topic.asp?id=709 http://bbs.quyouji.com/topic.asp?id=710 http://bbs.quyouji.com/topic.asp?id=711 http://bbs.quyouji.com/topic.asp?id=712 http://bbs.quyouji.com/topic.asp?id=713 http://bbs.quyouji.com/topic.asp?id=714 http://bbs.quyouji.com/topic.asp?id=715 http://bbs.quyouji.com/topic.asp?id=716 http://bbs.quyouji.com/topic.asp?id=717 http://bbs.quyouji.com/topic.asp?id=718 http://bbs.quyouji.com/topic.asp?id=719 http://bbs.quyouji.com/topic.asp?id=720 http://bbs.quyouji.com/topic.asp?id=721 http://bbs.quyouji.com/topic.asp?id=722 http://bbs.quyouji.com/topic.asp?id=723 http://bbs.quyouji.com/topic.asp?id=724 http://bbs.quyouji.com/topic.asp?id=725 http://bbs.quyouji.com/topic.asp?id=726 http://bbs.quyouji.com/topic.asp?id=727 http://bbs.quyouji.com/topic.asp?id=728 http://bbs.quyouji.com/topic.asp?id=729 http://bbs.quyouji.com/topic.asp?id=730 http://bbs.quyouji.com/topic.asp?id=731 http://bbs.quyouji.com/topic.asp?id=732 http://bbs.quyouji.com/topic.asp?id=733 http://bbs.quyouji.com/topic.asp?id=734 http://bbs.quyouji.com/topic.asp?id=735 http://bbs.quyouji.com/topic.asp?id=736 http://bbs.quyouji.com/topic.asp?id=737 http://bbs.quyouji.com/topic.asp?id=738 http://bbs.quyouji.com/topic.asp?id=739 http://bbs.quyouji.com/topic.asp?id=740 http://bbs.quyouji.com/topic.asp?id=741 http://bbs.quyouji.com/topic.asp?id=742 http://bbs.quyouji.com/topic.asp?id=743 http://bbs.quyouji.com/topic.asp?id=744 http://bbs.quyouji.com/topic.asp?id=745 http://bbs.quyouji.com/topic.asp?id=746 http://bbs.quyouji.com/topic.asp?id=747 http://bbs.quyouji.com/topic.asp?id=748 http://bbs.quyouji.com/topic.asp?id=749 http://bbs.quyouji.com/topic.asp?id=750 http://bbs.quyouji.com/topic.asp?id=751 http://bbs.quyouji.com/topic.asp?id=752 http://bbs.quyouji.com/topic.asp?id=753 http://bbs.quyouji.com/topic.asp?id=754 http://bbs.quyouji.com/topic.asp?id=755 http://bbs.quyouji.com/topic.asp?id=756 http://bbs.quyouji.com/topic.asp?id=757 http://bbs.quyouji.com/topic.asp?id=758 http://bbs.quyouji.com/topic.asp?id=759 http://bbs.quyouji.com/topic.asp?id=760 http://bbs.quyouji.com/topic.asp?id=761 http://bbs.quyouji.com/topic.asp?id=762 http://bbs.quyouji.com/topic.asp?id=763 http://bbs.quyouji.com/topic.asp?id=764 http://bbs.quyouji.com/topic.asp?id=765 http://bbs.quyouji.com/topic.asp?id=766 http://bbs.quyouji.com/topic.asp?id=767 http://bbs.quyouji.com/topic.asp?id=768 http://bbs.quyouji.com/topic.asp?id=769 http://bbs.quyouji.com/topic.asp?id=770 http://bbs.quyouji.com/topic.asp?id=771 http://bbs.quyouji.com/topic.asp?id=772 http://bbs.quyouji.com/topic.asp?id=773 http://bbs.quyouji.com/topic.asp?id=774 http://bbs.quyouji.com/topic.asp?id=775 http://bbs.quyouji.com/topic.asp?id=776 http://bbs.quyouji.com/topic.asp?id=777 http://bbs.quyouji.com/topic.asp?id=778 http://bbs.quyouji.com/topic.asp?id=779 http://bbs.quyouji.com/topic.asp?id=780 http://bbs.quyouji.com/topic.asp?id=781 http://bbs.quyouji.com/topic.asp?id=782 http://bbs.quyouji.com/topic.asp?id=783 http://bbs.quyouji.com/topic.asp?id=784 http://bbs.quyouji.com/topic.asp?id=785 http://bbs.quyouji.com/topic.asp?id=786 http://bbs.quyouji.com/topic.asp?id=787 http://bbs.quyouji.com/topic.asp?id=788 http://bbs.quyouji.com/topic.asp?id=789 http://bbs.quyouji.com/topic.asp?id=790 http://bbs.quyouji.com/topic.asp?id=791 http://bbs.quyouji.com/topic.asp?id=792 http://bbs.quyouji.com/topic.asp?id=793 http://bbs.quyouji.com/topic.asp?id=794 http://bbs.quyouji.com/topic.asp?id=795 http://bbs.quyouji.com/topic.asp?id=796 http://bbs.quyouji.com/topic.asp?id=797 http://bbs.quyouji.com/topic.asp?id=798 http://bbs.quyouji.com/topic.asp?id=799 http://bbs.quyouji.com/topic.asp?id=800 http://bbs.quyouji.com/topic.asp?id=801 http://bbs.quyouji.com/topic.asp?id=802 http://bbs.quyouji.com/topic.asp?id=803 http://bbs.quyouji.com/topic.asp?id=804 http://bbs.quyouji.com/topic.asp?id=805 http://bbs.quyouji.com/topic.asp?id=806 http://bbs.quyouji.com/topic.asp?id=807 http://bbs.quyouji.com/topic.asp?id=808 http://bbs.quyouji.com/topic.asp?id=809 http://bbs.quyouji.com/topic.asp?id=810 http://bbs.quyouji.com/topic.asp?id=811 http://bbs.quyouji.com/topic.asp?id=812 http://bbs.quyouji.com/topic.asp?id=813 http://bbs.quyouji.com/topic.asp?id=814 http://bbs.quyouji.com/topic.asp?id=815 http://bbs.quyouji.com/topic.asp?id=816 http://bbs.quyouji.com/topic.asp?id=817 http://bbs.quyouji.com/topic.asp?id=818 http://bbs.quyouji.com/topic.asp?id=819 http://bbs.quyouji.com/topic.asp?id=820 http://bbs.quyouji.com/topic.asp?id=821 http://bbs.quyouji.com/topic.asp?id=822 http://bbs.quyouji.com/topic.asp?id=823 http://bbs.quyouji.com/topic.asp?id=824 http://bbs.quyouji.com/topic.asp?id=825 http://bbs.quyouji.com/topic.asp?id=826 http://bbs.quyouji.com/topic.asp?id=827 http://bbs.quyouji.com/topic.asp?id=828 http://bbs.quyouji.com/topic.asp?id=829 http://bbs.quyouji.com/topic.asp?id=830 http://bbs.quyouji.com/topic.asp?id=831 http://bbs.quyouji.com/topic.asp?id=832 http://bbs.quyouji.com/topic.asp?id=833 http://bbs.quyouji.com/topic.asp?id=834 http://bbs.quyouji.com/topic.asp?id=835 http://bbs.quyouji.com/topic.asp?id=836 http://bbs.quyouji.com/topic.asp?id=837 http://bbs.quyouji.com/topic.asp?id=838 http://bbs.quyouji.com/topic.asp?id=839 http://bbs.quyouji.com/topic.asp?id=840 http://bbs.quyouji.com/topic.asp?id=841 http://bbs.quyouji.com/topic.asp?id=842 http://bbs.quyouji.com/topic.asp?id=843 http://bbs.quyouji.com/topic.asp?id=844 http://bbs.quyouji.com/topic.asp?id=845 http://bbs.quyouji.com/topic.asp?id=846 http://bbs.quyouji.com/topic.asp?id=847 http://bbs.quyouji.com/topic.asp?id=848 http://bbs.quyouji.com/topic.asp?id=849 http://bbs.quyouji.com/topic.asp?id=850 http://bbs.quyouji.com/topic.asp?id=851 http://bbs.quyouji.com/topic.asp?id=852 http://bbs.quyouji.com/topic.asp?id=853 http://bbs.quyouji.com/topic.asp?id=854 http://bbs.quyouji.com/topic.asp?id=855 http://bbs.quyouji.com/topic.asp?id=856 http://bbs.quyouji.com/topic.asp?id=857 http://bbs.quyouji.com/topic.asp?id=858 http://bbs.quyouji.com/topic.asp?id=859 http://bbs.quyouji.com/topic.asp?id=860 http://bbs.quyouji.com/topic.asp?id=861 http://bbs.quyouji.com/topic.asp?id=862 http://bbs.quyouji.com/topic.asp?id=863 http://bbs.quyouji.com/topic.asp?id=864 http://bbs.quyouji.com/topic.asp?id=865 http://bbs.quyouji.com/topic.asp?id=866 http://bbs.quyouji.com/topic.asp?id=867 http://bbs.quyouji.com/topic.asp?id=868 http://bbs.quyouji.com/topic.asp?id=869 http://bbs.quyouji.com/topic.asp?id=870 http://bbs.quyouji.com/topic.asp?id=871 http://bbs.quyouji.com/topic.asp?id=872 http://bbs.quyouji.com/topic.asp?id=873 http://bbs.quyouji.com/topic.asp?id=874 http://bbs.quyouji.com/topic.asp?id=875 http://bbs.quyouji.com/topic.asp?id=876 http://bbs.quyouji.com/topic.asp?id=877 http://bbs.quyouji.com/topic.asp?id=878 http://bbs.quyouji.com/topic.asp?id=879 http://bbs.quyouji.com/topic.asp?id=880 http://bbs.quyouji.com/topic.asp?id=881 http://bbs.quyouji.com/topic.asp?id=882 http://bbs.quyouji.com/topic.asp?id=883 http://bbs.quyouji.com/topic.asp?id=884 http://bbs.quyouji.com/topic.asp?id=885 http://bbs.quyouji.com/topic.asp?id=886 http://bbs.quyouji.com/topic.asp?id=887 http://bbs.quyouji.com/topic.asp?id=888 http://bbs.quyouji.com/topic.asp?id=889 http://bbs.quyouji.com/topic.asp?id=890 http://bbs.quyouji.com/topic.asp?id=891 http://bbs.quyouji.com/topic.asp?id=892 http://bbs.quyouji.com/topic.asp?id=893 http://bbs.quyouji.com/topic.asp?id=894 http://bbs.quyouji.com/topic.asp?id=895 http://bbs.quyouji.com/topic.asp?id=896 http://bbs.quyouji.com/topic.asp?id=897 http://bbs.quyouji.com/topic.asp?id=898 http://bbs.quyouji.com/topic.asp?id=899 http://bbs.quyouji.com/topic.asp?id=900 http://bbs.quyouji.com/topic.asp?id=901 http://bbs.quyouji.com/topic.asp?id=902 http://bbs.quyouji.com/topic.asp?id=903 http://bbs.quyouji.com/topic.asp?id=904 http://bbs.quyouji.com/topic.asp?id=905 http://bbs.quyouji.com/topic.asp?id=906 http://bbs.quyouji.com/topic.asp?id=907 http://bbs.quyouji.com/topic.asp?id=908 http://bbs.quyouji.com/topic.asp?id=909 http://bbs.quyouji.com/topic.asp?id=910 http://bbs.quyouji.com/topic.asp?id=911 http://bbs.quyouji.com/topic.asp?id=912 http://bbs.quyouji.com/topic.asp?id=913 http://bbs.quyouji.com/topic.asp?id=914 http://bbs.quyouji.com/topic.asp?id=915 http://bbs.quyouji.com/topic.asp?id=916 http://bbs.quyouji.com/topic.asp?id=917 http://bbs.quyouji.com/topic.asp?id=918 http://bbs.quyouji.com/topic.asp?id=919 http://bbs.quyouji.com/topic.asp?id=920 http://bbs.quyouji.com/topic.asp?id=921 http://bbs.quyouji.com/topic.asp?id=922 http://bbs.quyouji.com/topic.asp?id=923 http://bbs.quyouji.com/topic.asp?id=924 http://bbs.quyouji.com/topic.asp?id=925 http://bbs.quyouji.com/topic.asp?id=926 http://bbs.quyouji.com/topic.asp?id=927 http://bbs.quyouji.com/topic.asp?id=928 http://bbs.quyouji.com/topic.asp?id=929 http://bbs.quyouji.com/topic.asp?id=930 http://bbs.quyouji.com/topic.asp?id=931 http://bbs.quyouji.com/topic.asp?id=932 http://bbs.quyouji.com/topic.asp?id=933 http://bbs.quyouji.com/topic.asp?id=934 http://bbs.quyouji.com/topic.asp?id=935 http://bbs.quyouji.com/topic.asp?id=936 http://bbs.quyouji.com/topic.asp?id=937 http://bbs.quyouji.com/topic.asp?id=938 http://bbs.quyouji.com/topic.asp?id=939 http://bbs.quyouji.com/topic.asp?id=940 http://bbs.quyouji.com/topic.asp?id=941 http://bbs.quyouji.com/topic.asp?id=942 http://bbs.quyouji.com/topic.asp?id=943 http://bbs.quyouji.com/topic.asp?id=944 http://bbs.quyouji.com/topic.asp?id=945 http://bbs.quyouji.com/topic.asp?id=946 http://bbs.quyouji.com/topic.asp?id=947 http://bbs.quyouji.com/topic.asp?id=948 http://bbs.quyouji.com/topic.asp?id=949 http://bbs.quyouji.com/topic.asp?id=950 http://bbs.quyouji.com/topic.asp?id=951 http://bbs.quyouji.com/topic.asp?id=952 http://bbs.quyouji.com/topic.asp?id=953 http://bbs.quyouji.com/topic.asp?id=954 http://bbs.quyouji.com/topic.asp?id=955 http://bbs.quyouji.com/topic.asp?id=956 http://bbs.quyouji.com/topic.asp?id=957 http://bbs.quyouji.com/topic.asp?id=958 http://bbs.quyouji.com/topic.asp?id=959 http://bbs.quyouji.com/topic.asp?id=960 http://bbs.quyouji.com/topic.asp?id=961 http://bbs.quyouji.com/topic.asp?id=962 http://bbs.quyouji.com/topic.asp?id=963 http://bbs.quyouji.com/topic.asp?id=964 http://bbs.quyouji.com/topic.asp?id=965 http://bbs.quyouji.com/topic.asp?id=966 http://bbs.quyouji.com/topic.asp?id=967 http://bbs.quyouji.com/topic.asp?id=968 http://bbs.quyouji.com/topic.asp?id=969 http://bbs.quyouji.com/topic.asp?id=970 http://bbs.quyouji.com/topic.asp?id=971 http://bbs.quyouji.com/topic.asp?id=972 http://bbs.quyouji.com/topic.asp?id=973 http://bbs.quyouji.com/topic.asp?id=974 http://bbs.quyouji.com/topic.asp?id=975 http://bbs.quyouji.com/topic.asp?id=976 http://bbs.quyouji.com/topic.asp?id=977 http://bbs.quyouji.com/topic.asp?id=978 http://bbs.quyouji.com/topic.asp?id=979 http://bbs.quyouji.com/topic.asp?id=980 http://bbs.quyouji.com/topic.asp?id=981 http://bbs.quyouji.com/topic.asp?id=982 http://bbs.quyouji.com/topic.asp?id=983 http://bbs.quyouji.com/topic.asp?id=984 http://bbs.quyouji.com/topic.asp?id=985 http://bbs.quyouji.com/topic.asp?id=986 http://bbs.quyouji.com/topic.asp?id=987 http://bbs.quyouji.com/topic.asp?id=988 http://bbs.quyouji.com/topic.asp?id=989 http://bbs.quyouji.com/topic.asp?id=990 http://bbs.quyouji.com/topic.asp?id=991 http://bbs.quyouji.com/topic.asp?id=992 http://bbs.quyouji.com/topic.asp?id=993 http://bbs.quyouji.com/topic.asp?id=994 http://bbs.quyouji.com/topic.asp?id=995 http://bbs.quyouji.com/topic.asp?id=996 http://bbs.quyouji.com/topic.asp?id=997 http://bbs.quyouji.com/topic.asp?id=998 http://bbs.quyouji.com/topic.asp?id=999 http://bbs.quyouji.com/topic.asp?id=1000 http://bbs.quyouji.com/topic.asp?id=1001 http://bbs.quyouji.com/topic.asp?id=1002 http://bbs.quyouji.com/topic.asp?id=1003 http://bbs.quyouji.com/topic.asp?id=1004 http://bbs.quyouji.com/topic.asp?id=1005 http://bbs.quyouji.com/topic.asp?id=1006 http://bbs.quyouji.com/topic.asp?id=1007 http://bbs.quyouji.com/topic.asp?id=1008 http://bbs.quyouji.com/topic.asp?id=1009 http://bbs.quyouji.com/topic.asp?id=1010 http://bbs.quyouji.com/topic.asp?id=1011 http://bbs.quyouji.com/topic.asp?id=1012 http://bbs.quyouji.com/topic.asp?id=1013 http://bbs.quyouji.com/topic.asp?id=1014 http://bbs.quyouji.com/topic.asp?id=1015 http://bbs.quyouji.com/topic.asp?id=1016 http://bbs.quyouji.com/topic.asp?id=1017 http://bbs.quyouji.com/topic.asp?id=1018 http://bbs.quyouji.com/topic.asp?id=1019 http://bbs.quyouji.com/topic.asp?id=1020 http://bbs.quyouji.com/topic.asp?id=1021 http://bbs.quyouji.com/topic.asp?id=1022 http://bbs.quyouji.com/topic.asp?id=1023 http://bbs.quyouji.com/topic.asp?id=1024 http://bbs.quyouji.com/topic.asp?id=1025 http://bbs.quyouji.com/topic.asp?id=1026 http://bbs.quyouji.com/topic.asp?id=1027 http://bbs.quyouji.com/topic.asp?id=1028 http://bbs.quyouji.com/topic.asp?id=1029 http://bbs.quyouji.com/topic.asp?id=1030 http://bbs.quyouji.com/topic.asp?id=1031 http://bbs.quyouji.com/topic.asp?id=1032 http://bbs.quyouji.com/topic.asp?id=1033 http://bbs.quyouji.com/topic.asp?id=1034 http://bbs.quyouji.com/topic.asp?id=1035 http://bbs.quyouji.com/topic.asp?id=1036 http://bbs.quyouji.com/topic.asp?id=1037 http://bbs.quyouji.com/topic.asp?id=1038 http://bbs.quyouji.com/topic.asp?id=1039 http://bbs.quyouji.com/topic.asp?id=1040 http://bbs.quyouji.com/topic.asp?id=1041 http://bbs.quyouji.com/topic.asp?id=1042 http://bbs.quyouji.com/topic.asp?id=1043 http://bbs.quyouji.com/topic.asp?id=1044 http://bbs.quyouji.com/topic.asp?id=1045 http://bbs.quyouji.com/topic.asp?id=1046 http://bbs.quyouji.com/topic.asp?id=1047 http://bbs.quyouji.com/topic.asp?id=1048 http://bbs.quyouji.com/topic.asp?id=1049 http://bbs.quyouji.com/topic.asp?id=1050 http://bbs.quyouji.com/topic.asp?id=1051 http://bbs.quyouji.com/topic.asp?id=1052 http://bbs.quyouji.com/topic.asp?id=1053 http://bbs.quyouji.com/topic.asp?id=1054 http://bbs.quyouji.com/topic.asp?id=1055 http://bbs.quyouji.com/topic.asp?id=1056 http://bbs.quyouji.com/topic.asp?id=1057 http://bbs.quyouji.com/topic.asp?id=1058 http://bbs.quyouji.com/topic.asp?id=1059 http://bbs.quyouji.com/topic.asp?id=1060 http://bbs.quyouji.com/topic.asp?id=1061 http://bbs.quyouji.com/topic.asp?id=1062 http://bbs.quyouji.com/topic.asp?id=1063 http://bbs.quyouji.com/topic.asp?id=1064 http://bbs.quyouji.com/topic.asp?id=1065 http://bbs.quyouji.com/topic.asp?id=1066 http://bbs.quyouji.com/topic.asp?id=1067 http://bbs.quyouji.com/topic.asp?id=1068 http://bbs.quyouji.com/topic.asp?id=1069 http://bbs.quyouji.com/topic.asp?id=1070 http://bbs.quyouji.com/topic.asp?id=1071 http://bbs.quyouji.com/topic.asp?id=1072 http://bbs.quyouji.com/topic.asp?id=1073 http://bbs.quyouji.com/topic.asp?id=1074 http://bbs.quyouji.com/topic.asp?id=1075 http://bbs.quyouji.com/topic.asp?id=1076 http://bbs.quyouji.com/topic.asp?id=1077 http://bbs.quyouji.com/topic.asp?id=1078 http://bbs.quyouji.com/topic.asp?id=1079 http://bbs.quyouji.com/topic.asp?id=1080 http://bbs.quyouji.com/topic.asp?id=1081 http://bbs.quyouji.com/topic.asp?id=1082 http://bbs.quyouji.com/topic.asp?id=1083 http://bbs.quyouji.com/topic.asp?id=1084 http://bbs.quyouji.com/topic.asp?id=1085 http://bbs.quyouji.com/topic.asp?id=1086 http://bbs.quyouji.com/topic.asp?id=1087 http://bbs.quyouji.com/topic.asp?id=1088 http://bbs.quyouji.com/topic.asp?id=1089

接下来是∂l∂W2，根据公式，它满足：

∂l∂WL=∂l∂zL(aL−1)T

代码验证的结果是：

复制代码 t.gradient(loss, l2.kernel) Out[23]: <tf.Tensor: shape=(4, 3), dtype=float32, numpy= array([[ 0.02168725, -0.00741658, 0.03423793], [ 0.00715614, -0.00244725, 0.01129749], [ 0.00801025, -0.00273934, 0.01264589], [ 0.02379899, -0.00813875, 0.03757175]], dtype=float32)> np.matmul(a_l1.numpy().transpose(), my_dl_dz2) Out[24]: array([[ 0.02168725, -0.00741658, 0.03423794], [ 0.00715614, -0.00244725, 0.01129749], [ 0.00801025, -0.00273934, 0.01264589], [ 0.02379899, -0.00813875, 0.03757175]], dtype=float32) 复制代码

也没有问题。

剩下的大家可自行敲敲上面给出的验证代码看看是否跟推导的公式一致，这里就不再一一演示了。

最新回复(0)