Caffe How to Debug?

 文章作者：Tyan
博客：noahsnail.com  |  CSDN  |  简书

这篇文件主要介绍如何使用Linux的gdb调试Caffe的源码，源码调试主要是为了阅读并更好的了解Caffe源码。

1. 准备工作

1. 首先要在编译Caffe源码时打开debug模式，即将Makefile.config中的DEBUG := 1打开。
2. 下载mnist数据集，主要是在mnist数据集上进行调试，执行bash data/mnist/get_mnist.sh。
3. 转换mnist数据集为LMDB，bash examples/mnist/create_mnist.sh。
4. 修改examples/mnist/lenet_solver.prototxt，将GPU改为CPU。

2. 调试

1. 激活GDB

使用GDB启动调试，执行gdb --args build/tools/caffe train --solver examples/mnist/lenet_solver.prototxt，--args表示我们调试时需要输入的参数，调试的命令为build/tools/caffe，caffe命令的参数为--solver examples/mnist/lenet_solver.prototxt。

执行结果：

$ gdb --args build/tools/caffe train --solver examples/mnist/lenet_solver.prototxt
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/irteam/line-brain/deploy/caffe/.build_debug/tools/caffe.bin...done.
 

2. 设置断点

执行b src/caffe/layers/base_conv_layer.cpp:117，b表示插入断点（breakpoint），断点的位置是base_conv_layer.cpp文件中的117行。插入断点的命令形式为：

b path/to/code.cpp:#line 

118行相关代码：

117 channels_ = bottom[0]->shape(channel_axis_);
118 num_output_ = this->layer_param_.convolution_param().num_output();
119 CHECK_GT(num_output_, 0);
 

执行结果：

(gdb) b src/caffe/layers/base_conv_layer.cpp:117
No source file named src/caffe/layers/base_conv_layer.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (src/caffe/layers/base_conv_layer.cpp:117) pending.

3. 运行程序

运行程序的命令是r。

执行结果：

Starting program: /*/caffe/build/tools/caffe train --solver examples/mnist/lenet_solver.prototxt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
I0718 15:19:19.671941 29986 caffe.cpp:211] Use CPU.

Breakpoint 1, caffe::BaseConvolutionLayer<float>::LayerSetUp (this=0x91edd70,
    bottom=std::vector of length 1, capacity 1 = {...}, top=std::vector of length 1, capacity 1 = {...})
    at src/caffe/layers/base_conv_layer.cpp:117
117       channels_ = bottom[0]->shape(channel_axis_);
Missing separate debuginfos, use: debuginfo-install OpenEXR-libs-1.7.1-7.el7.x86_64 atk-2.14.0-1.el7.x86_64 atlas-3.10.1-10.el7.x86_64 boost-filesystem-1.53.0-26.el7.x86_64 boost-python-1.53.0-26.el7.x86_64 boost-system-1.53.0-26.el7.x86_64 boost-thread-1.53.0-26.el7.x86_64 cairo-1.14.2-1.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fontconfig-2.10.95-10.el7.x86_64 freetype-2.4.11-12.el7.x86_64 gdk-pixbuf2-2.31.6-3.el7.x86_64 gflags-2.1.1-6.el7.x86_64 glib2-2.46.2-4.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64 glog-0.3.3-8.el7.x86_64 graphite2-1.3.6-1.el7_2.x86_64 gstreamer-0.10.36-7.el7.x86_64 gstreamer-plugins-base-0.10.36-10.el7.x86_64 gtk2-2.24.28-8.el7.x86_64 harfbuzz-0.9.36-1.el7.x86_64 hdf5-1.8.12-8.el7.x86_64 ilmbase-1.0.3-7.el7.x86_64 jasper-libs-1.900.1-29.el7.x86_64 jbigkit-libs-2.0-11.el7.x86_64 leveldb-1.12.0-11.el7.x86_64 libX11-1.6.3-3.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXcomposite-0.4.4-4.1.el7.x86_64 libXcursor-1.1.14-2.1.el7.x86_64 libXdamage-1.1.4-4.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXfixes-5.0.1-2.1.el7.x86_64 libXi-1.7.4-2.el7.x86_64 libXinerama-1.1.3-2.1.el7.x86_64 libXrandr-1.4.2-2.el7.x86_64 libXrender-0.9.8-2.1.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libgfortran-4.8.5-11.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libquadmath-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 libstdc++-4.8.5-11.el7.x86_64 libtiff-4.0.3-27.el7_3.x86_64 libv4l-0.9.5-4.el7.x86_64 libxcb-1.11-4.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 lmdb-libs-0.9.18-1.el7.x86_64 opencv-2.4.5-3.el7.x86_64 opencv-core-2.4.5-3.el7.x86_64 orc-0.4.22-5.el7.x86_64 pango-1.36.8-2.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 pixman-0.34.0-1.el7.x86_64 protobuf-2.5.0-8.el7.x86_64 python-libs-2.7.5-48.el7.x86_64 snappy-1.1.0-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64

Breakpoint 1之前是正常的程序日志输出，程序在断点处暂停。

查看变量命令为p var，命令与结果如下：

(gdb) p channels_
$1 = 0

(gdb) p channel_axis_
$2 = 1

此时，channels_值为0。下一行命令为n，执行结果如下：

(gdb) n
118       num_output_ = this->layer_param_.convolution_param().num_output();
此时查看channels_值为1，mnist数据是灰度图像，channels_为1没问题：

(gdb) p channels_
$3 = 1

命令c是继续执行直到下一个断点。

如果需要调试GPU程序，可以使用cuda-gdb，文档地址为：http://docs.nvidia.com/cuda/cuda-gdb/index.html#axzz4nAAR7ujZ。

参考资料

1. http://zhaok.xyz/blog/post/debug-caffe/