Goku Coding Experience: May 2019

Thursday, 9 May 2019

caffe: undefined reference to boost::gregorian::greg_month::as_short_string() const

the reason is due to the missing dependencies of boost date_time library in cmake configuration

Goto root > cmake > Dependencies.cmake

search a keyword: filesystem (original source)
find_package(Boost 1.61 COMPONENTS "python${PYTHON_VERSION_MAJOR}" system thread filesystem regex)

adding a keyword named as date_time into all corresponding search result (modified source):-
find_package(Boost 1.61 COMPONENTS "python${PYTHON_VERSION_MAJOR}" date_time system thread filesystem regex)

Trace or track Python statement execution (trace)

Function in the 'trace' module in Python library generates trace of program execution, and annotated statement coverage. It also has functions to list functions called during run by generating caller relationships.

Following two Python scripts are used as an example to demonstrate features of trace module.

#myfunctions.py
import math
def area(x):
   a = math.pi*math.pow(x,2)
   return a
def factorial(x):
   if x==1:
      return 1
   else:
return x*factorial(x-1)

#mymain.py
import myfunctions
def main():
   x = 5
   print ('area=',myfunctions.area(x))
   print ('factorial=',myfunctions.factorial(x))

if __name__=='__main__':
   main()

The 'trace' module has a command line interface. All the functions in the module can be called using command line switches. The most important option is --trace which displays program lines as they are executed. In following example another option --ignore-dir is used. It ignores specified directories while generating the trace.

E:\python37>python -m trace --ignore-dir=../lib --trace mymain.py

Output

mymain.py(2): def main():
mymain.py(7): if __name__=='__main__':
mymain.py(8): main()
--- modulename: mymain, funcname: main
mymain.py(3): x=5
mymain.py(4): print ('area=',myfunctions.area(x))
--- modulename: myfunctions, funcname: area
myfunctions.py(3): a=math.pi*math.pow(x,2)
myfunctions.py(4): return a
area= 78.53981633974483
mymain.py(5): print ('factorial=',myfunctions.factorial(x))
--- modulename: myfunctions, funcname: factorial
myfunctions.py(6): if x==1:
myfunctions.py(9): return x*factorial(x-1)
--- modulename: myfunctions, funcname: factorial
myfunctions.py(6): if x==1:
myfunctions.py(9): return x*factorial(x-1)
--- modulename: myfunctions, funcname: factorial
myfunctions.py(6): if x==1:
myfunctions.py(9): return x*factorial(x-1)
--- modulename: myfunctions, funcname: factorial
myfunctions.py(6): if x==1:
myfunctions.py(9): return x*factorial(x-1)
--- modulename: myfunctions, funcname: factorial
myfunctions.py(6): if x==1:
myfunctions.py(7): return 1
factorial= 120

The --count option generates a file for each module in use with, cover extension.

E:\python37>python -m trace --count mymain.py
area= 78.53981633974483
factorial = 120

myfunctions.cover

1: import math
1: def area(x):
1:    a = math.pi*math.pow(x,2)
1:    return a
1: def factorial(x):
5:    if x==1:
1:       return 1
   else:
4:    return x*factorial(x-1)

mymain.cover

1: import myfunctions
1: def main():
1:    x = 5
1:    print ('area=',myfunctions.area(x))
1:    print ('factorial=',myfunctions.factorial(x))

1: if __name__=='__main__':
1:    main()

--summary option displays a brief summary if –count option is also used.

E:\python37>python -m trace --count --summary mymain.py
area = 78.53981633974483
factorial = 120
lines cov% module (path)
   8 100% myfunctions (E:\python37\myfunctions.py)
   7 100% mymain (mymain.py)

The --file option specifies name of file in which accumulates count over several tracing runs.

E:\python37>python -m trace --count --file report.txt mymain.py
area = 78.53981633974483
factorial = 120
Skipping counts file 'report.txt': [Errno 2] No such file or directory: 'report.txt'

E:\python37>python -m trace --count --file report.txt mymain.py
area= 78.53981633974483
factorial= 120

--listfuncs option displays functions called during execution of program.

E:\python37>python -m trace --listfunc mymain.py | findstr -v importlib
area= 78.53981633974483
factorial= 120

functions called:
filename: E:\python37\lib\encodings\cp1252.py, modulename: cp1252, funcname: IncrementalEncoder.encode
filename: E:\python37\myfunctions.py, modulename: myfunctions, funcname: <module>
filename: E:\python37\myfunctions.py, modulename: myfunctions, funcname: area
filename: E:\python37\myfunctions.py, modulename: myfunctions, funcname: factorial
filename: mymain.py, modulename: mymain, funcname: <module>
filename: mymain.py, modulename: mymain, funcname: main

--trackcalls option is used along with –list funcs option. It generates calling relationships.

E:\python37>python -m trace --listfunc --trackcalls mymain.py | findstr -v importlib
area= 78.53981633974483
factorial= 120

calling relationships:

--> E:\python37\myfunctions.py


*** E:\python37\lib\trace.py ***
--> mymain.py
trace.Trace.runctx -> mymain.<module>

*** E:\python37\myfunctions.py ***
myfunctions.factorial -> myfunctions.factorial

*** mymain.py ***
mymain.<module> -> mymain.main
--> E:\python37\lib\encodings\cp1252.py
mymain.main -> cp1252.IncrementalEncoder.encode
--> E:\python37\myfunctions.py
mymain.main -> myfunctions.area
mymain.main -> myfunctions.factorial

(ref: https://www.tutorialspoint.com/trace-or-track-python-statement-execution-trace)

Tuesday, 7 May 2019

ubuntu programming checklist

Ubuntu is well known for deep learning programming or development.
However, this os can be easily corrupted by few sudo commands.
Therefore, here is a checklist for myself before and after some works.

1. use mobaxterm and tmux for remote programming

2. use github for source code management

3. write down steps used for a success or failure trial

4. always zip the images before transferring, especially something very small like 24x24

5. always check the path before running a time consuming program

Monday, 6 May 2019

how to build caffe in ubuntu16, cuda10.0, cudnn7.6, python2.7

0. install the compatible nvidia driver, cudnn and nccl for ubuntu16. They are
cuda10.0 (cuda_10.0.130_410.48_linux.run) and cudnn7.6.0 (cudnn-10.0-linux-x64-v7.6.0.64.tgz) and nccl (nccl_2.4.7-1+cuda10.0_x86_64.txz).

0.1 install general dependencies:

sudo apt-get remove x264 libx264-dev

sudo apt update

sudo apt upgrade

sudo apt install python2.7 python-pip
sudo apt install build-essential checkinstall cmake pkg-config yasm git gfortran libjpeg8-dev libjasper-dev libpng12-dev libtiff5-dev libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine2-dev libv4l-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libgtk2.0-dev libtbb-dev libatlas-base-dev libfaac-dev libmp3lame-dev libtheora-dev libvorbis-dev libxvidcore-dev libopencore-amrnb-dev libopencore-amrwb-dev x264 v4l-utils libgoogle-glog-dev libgflags-dev libgphoto2-dev libeigen3-dev libhdf5-dev doxygen

sudo apt-get install ffmpeg libgtk-3-dev python-numpy python3-numpy libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libv4l-dev libtbb-dev qtbase5-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils unzip liblmdb-dev libleveldb-dev libsnappy-dev libboost-dev

sudo apt install qt5-default

sudo apt install libboost-all-dev

1. install the correct protobuf packages (version 3.6.1) as given:-

# Make sure you grab the latest version
wget https://github.com/google/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip

# Unzip
unzip protoc-3.6.1-linux-x86_64.zip -d protoc3

# Move protoc to /usr/local/bin/
sudo cp -r protoc3/bin/* /usr/local/bin/

# Move protoc3/include to /usr/local/include/
sudo cp -r protoc3/include/* /usr/local/include/

# Optional: change owner
sudo chown $USER /usr/local/bin/protoc
sudo chown -R $USER /usr/local/include/google

sudo ldconfig

1.1 (optional, ignore) install the correct protobuf packages (version 3.1.0) as given in https://ninja5code.blogspot.com/2019/02/protobuf-error.html
$ sudo add-apt-repository ppa:maarten-fonville/protobuf
$ sudo apt-get update
$ sudo apt-get remove libprotobuf-dev libprotoc-dev protobuf-compiler
$ sudo apt-get install libprotobuf-dev libprotoc-dev protobuf-compiler
$ protoc --version
$ dpkg -l | grep protobuf

2. download zip files of opencv 3.4.4 and opencv-contrib 3.4.4 from https://github.com/opencv/opencv/releases/tag/3.4.4, https://github.com/opencv/opencv_contrib/releases/tag/3.4.4. You should unzip it into the folder as /home/ninja/opencv and /home/ninja/opencv-contrib-3.4.4, respectively. Then, build it with 'cmake ..' and 'make -j$(nproc)' and 'sudo make install'
$ cd opencv
$ cd build/
$ rm -r ./*
$ cmake ..
$ make -j32
$ sudo make install

2.1 if there is any wrong reference to xxx.lib, you can list down all the dependencies using:
$ ldd /usr/local/lib/xxx.lib

2.2 Or you can check the library path to see the additional path within it:-
$ echo $LD_LIBRARY_PATH

Then, manually set it using:
$ export LD_LIBRARY_PATH = xxx:xxx

2.3 If cmake .. failed to build it, check the log printed by cmake, then install the dependencies if necessary, for example, failed to find libssl:-
$ sudo apt search libssl
$ sudo apt install libssl-dev

2.4 If make -j32 failed to build, you may try the following cmake:-
sudo cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-3.4.4/distribute \
            -D INSTALL_C_EXAMPLES=ON \
            -D OPENCV_GENERATE_PKGCONFIG=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D OPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-3.4.4/distribute/python3 \
            -D WITH_QT=ON \
            -D WITH_OPENGL=ON \
            -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-3.4.4/modules \
            -D BUILD_NEW_PYTHON_SUPPORT=ON \
            -D BUILD_opencv_python3=ON \
            -D HAVE_opencv_python3=ON \
            -D PYTHON3_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_DEFAULT_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_INCLUDE_PATH=/home/ninja/anaconda3/include/python3.7m \
            -D PYTHON3_NUMPY_INCLUDE_DIRS=/home/ninja/anaconda3/lib/python3.7/site-packages/numpy/core/include \
            -D PYTHON3_PACKAGES_PATH=/home/ninja/anaconda3/lib/python3.7/site-packages \
            -D PYTHON3_LIBRARIES=/home/ninja/anaconda3/lib/libpython3.7m.so    \
            -D PYTHON3_LIBRARIES_PATH=/home/ninja/anaconda3/lib    \
            -D BUILD_EXAMPLES=ON ..

(make sure you only have single version of library in LD_LIBRARY_PATH, for example, you cannot have protobuf 3.8.0 in /usr/lib/x86_64-linux-gnu/ and protobuf 3.1.0 in anaconda. You need to uninstall both and install protobuf 3.1.0 under /usr/lib/x86_64-linux-gnu only)

2.5 If you want to build with CUDA 8.0 and CUDNN 6.0 support:-
sudo cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-3.4.4/distribute \
            -D INSTALL_C_EXAMPLES=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D WITH_CUDA=ON \
            -D WITH_CUBLAS=ON \
            -D CUDA_FAST_MATH=ON \
            -D CUDA_GENERATION=Fermi \
            -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-8.0 \
            -D OPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-3.4.4/distribute/python3 \

            -D WITH_QT=ON \
            -D WITH_OPENGL=ON \
            -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-3.4.4/modules \
            -D BUILD_NEW_PYTHON_SUPPORT=ON \
            -D BUILD_opencv_python3=ON \
            -D HAVE_opencv_python3=ON \
            -D PYTHON3_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_DEFAULT_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_INCLUDE_PATH=/home/ninja/anaconda3/include/python3.7m \
            -D PYTHON3_NUMPY_INCLUDE_DIRS=/home/ninja/anaconda3/lib/python3.7/site-packages/numpy/core/include \
            -D PYTHON3_PACKAGES_PATH=/home/ninja/anaconda3/lib/python3.7/site-packages \
            -D PYTHON3_LIBRARIES=/home/ninja/anaconda3/lib/libpython3.7m.so    \
            -D PYTHON3_LIBRARIES_PATH=/home/ninja/anaconda3/lib    \
            -D BUILD_EXAMPLES=ON ..

2.6 If you want to build with CUDA 10.0 and CUDNN 7.6 support:-
sudo cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-3.4.4/distribute \
            -D INSTALL_C_EXAMPLES=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D WITH_CUDA=ON \
            -D WITH_CUBLAS=ON \
            -D BUILD_opencv_cudacodec=OFF \
            -D CUDA_FAST_MATH=ON \
            -D CUDA_GENERATION=Kepler \
            -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.0 \
            -D OPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-3.4.4/distribute/python3 \

Build with Opencv-4.4.0, Opencv_Contrib-4.4.0, Ubuntu18, Cuda10-2

{
    "cmake.configureArgs": [
        "-DCMAKE_BUILD_TYPE=RELEASE",
        "-DCMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-4.4.0/distribute",
        "-DINSTALL_C_EXAMPLES=ON",
        "-DOPENCV_GENERATE_PKGCONFIG=ON",
        "-DINSTALL_PYTHON_EXAMPLES=ON",
        "-DBUILD_SHARED_LIBS=ON"
        "-DWITH_TBB=ON",
        "-DWITH_V4L=ON",
        "-DBUILD_opencv_world=OFF",
        "-DOPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-4.4.0/distribute/python3",
        "-DWITH_QT=ON",
        "-DWITH_OPENGL=ON",
        "-DWITH_CUDA=ON",
        "-DWITH_CUBLAS=ON",
        "-DBUILD_opencv_cudacodec=ON",
        "-DCUDA_FAST_MATH=ON",
        "-DCUDA_GENERATION=Kepler",
        "-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.2",
        "-DOPENCV_EXTRA_MODULES_PATH=/home/ninja/workspace/opencv_contrib-4.4.0/modules",
        "-DBUILD_NEW_PYTHON_SUPPORT=ON",
        "-DBUILD_opencv_python3=ON",
        "-DHAVE_opencv_python3=ON",
        "-DPYTHON3_EXECUTABLE=/home/ninja/anaconda3/bin/python",
        "-DPYTHON3_DEFAULT_EXECUTABLE=/home/ninja/anaconda3/bin/python",
        "-DPYTHON3_INCLUDE_PATH=/home/ninja/anaconda3/include/python3.8",
        "-DPYTHON3_NUMPY_INCLUDE_DIRS=/home/ninja/anaconda3/lib/python3.8/site-packages/numpy/core/include",
        "-DPYTHON3_PACKAGES_PATH=/home/ninja/anaconda3/lib/python3.8/site-packages",
        "-DPYTHON3_LIBRARIES=/home/ninja/anaconda3/lib/libpython3.8.so",
        "-DPYTHON3_LIBRARIES_PATH=/home/ninja/anaconda3/lib",
        "-DBUILD_EXAMPLES=ON"
        ]
}

2.7. if you have this error: The imported target "Qt5::Gui" references the file, "/usr/lib/x86_64-linux-gnu/libEGL.so"
sudo rm /usr/lib/x86_64-linux-gnu/libEGL.so; sudo ln /usr/lib/x86_64-linux-gnu/libEGL.so.1 /usr/lib/x86_64-linux-gnu/libEGL.so

2.8. if you have this error: The imported target "Qt5::Gui" references the file, "/usr/lib/x86_64-linux-gnu/libGL.so"
sudo rm /usr/lib/x86_64-linux-gnu/libGL.so; sudo ln /usr/lib/x86_64-linux-gnu/libGL.so.1 /usr/lib/x86_64-linux-gnu/libGL.so

2.9. If you have this error: CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: CUDA_nppi_LIBRARY (ADVANCED)
$ cd ~/workspace/opencv-2.4.13.6/cmake
$ cp ../../opencv-3.4.4/cmake/FindCUDA.cmake ./ $ cp ../../opencv-3.4.4/cmake/OpenCVDetectCUDA.cmake ./
$ cd ../build
$ make all -j32

2.10. If you have this error, opencv_dep_CUDA_nppi_LIBRARY is not found:-

replace the root > cmake > FindCUDA.cmake from opencv3.4.4 to 2.4.13.6

2.11. If you have this error, "gflags-2.1.1/src/gflags.cc:90:20: fatal error: config.h: No such file or directory"

cmake version is too new, need to downgrade

3. create a conda virtual env named as caffe2 with python2.7, install:
$ pip install scikit-image
$ pip install protobuf (after completed step 2)

4. download caffe and build it with 'cmake ..' and 'make -j$(nproc)' and 'sudo make install'
$ cd caffe/
$ cd build/
$ sudo rm -r ./*
$ cmake .. -DCMAKE_INSTALL_PREFIX:PATH=/home/ninja/.conda/envs/caffe
$ make -j32
$ sudo make install
$ make runtest
$ make pycaffe

4.1 if sudo make install failed due to undefined reference to `ZSTD_initCStream, run:
$ conda list
$ conda install libtiff (it should install zstd)

or you can just modify the library path by pointing to system libraries only

$ gedit ~/.bashrc
>> # insert into last line
>> export $LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu

$ source ~/.bashrc
$ conda activate caffe2(compile again, make -j32)

4.2 C++ exception with description "locale::facet::_S_create_c_locale name not valid" thrown in SetUp().
$ export LC_ALL="C"

(rerun make runtest)

4.3 If you received the following error:-

CXX/LD -o .build_release/tools/convert_imageset.bin .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long long, unsigned char*)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::WriteInt64(int, long long, google::protobuf::io::CodedOutputStream*)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::AddVarint(int, unsigned long long)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long long)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64(unsigned long long)' collect2: error: ld returned 1 exit status make: *** [.build_release/tools/convert_imageset.bin] Error 1

$ ldd .build_release/lib/libcaffe.so

(check libcaffe.so is using which library, it should point to libprotobuf.so.11 => /usr/local/lib/libprotobuf.so.11. If it links to library under anaconda, just delete it.)

5.If all succeed, you need to set the pythonpath as:-
$ export PYTHONPATH=/home/ninja/.conda/envs/caffe/python:$PYTHONPATH

6. run python and then import caffe, you may need to install protobuf as pip install protobuf==3.1.0 and pip install scikit-image (check using protoc --version)

7. /usr/bin/ld: cannot find -llib64

check the CUDNN_LIBRARY or CUDA_LIBRARY path, should be full path to the specific library, /home/ubuntu/workspace/cuda/lib64/libcudnn.so, not folder name /home/ubuntu/workspace/cuda/lib64/

8. /home/ubuntu/workspace/caffe-opencv4.2-cuda10.0/include/caffe/proto/caffe.pb.h:17:2: error: #error This file was generated by an older version of protoc which is

./include/caffe/util/cudnn.hpp:8:34: fatal error: caffe/proto/caffe.pb.h: No such file or directory

$ protoc src/caffe/proto/caffe.proto --cpp_out=.
$ mkdir include/caffe/proto
$ mv src/caffe/proto/caffe.pb.h include/caffe/proto

9. how to force cmake to use one particular cuda architecture?

-DCUDA_ARCH_NAME=Manual -DCUDA_ARCH_BIN="75" -DCUDA_ARCH_PTX="75"

Example of .bashrc for building opencv
PATH=$PATH:$HOME/anaconda/bin:/usr/local/cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/lib/x86_64-linux-gnu/:/home/ninja/anaconda3/lib

Example of .bashrc for building caffe
export PYTHONPATH=/home/ninja/.conda/envs/caffe/python:$PYTHONPATH
PATH=$PATH:$HOME/anaconda/bin:/usr/local/cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/lib/x86_64-linux-gnu/:/home/ninja/.conda/envs/caffe/lib

10. (-218:No OpenGL support) Library was built without OpenGL support in function 'CvWindow'：-

        -D OpenGL_GL_PREFERENCE=GLVND \
        -D WITH_QT=ON \
        -D WITH_OPENGL=ON \
        -D WITH_GTK_2_X=ON \

11. How to find the correct path of opencv from a new C++ project:

During opencv compilation, set the install path in cmakelists.txt:

-D CMAKE_INSTALL_PREFIX=/home/peseyes/Documents/opencv-4.4.0/distribute \

During the new project compilation, set the opencv path in cmakelists.txt:-

SET(OpenCV_DIR /home/peseyes/Documents/opencv-4.4.0/distribute/share/opencv4) find_package(OpenCV 4 REQUIRED)