how to build caffe in ubuntu16, cuda10.0, cudnn7.6, python2.7

0. install the compatible nvidia driver, cudnn and nccl for ubuntu16. They are
  cuda10.0 (cuda_10.0.130_410.48_linux.run) and cudnn7.6.0 (cudnn-10.0-linux-x64-v7.6.0.64.tgz) and nccl (nccl_2.4.7-1+cuda10.0_x86_64.txz).

0.1 install general dependencies:

sudo apt-get remove x264 libx264-dev

sudo apt update

sudo apt upgrade

sudo apt install python2.7 python-pip
sudo apt install build-essential checkinstall cmake pkg-config yasm git gfortran libjpeg8-dev libjasper-dev libpng12-dev libtiff5-dev libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine2-dev libv4l-dev  libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libgtk2.0-dev libtbb-dev libatlas-base-dev libfaac-dev libmp3lame-dev libtheora-dev libvorbis-dev libxvidcore-dev libopencore-amrnb-dev libopencore-amrwb-dev x264 v4l-utils libgoogle-glog-dev libgflags-dev libgphoto2-dev libeigen3-dev libhdf5-dev doxygen

sudo apt-get install ffmpeg libgtk-3-dev python-numpy python3-numpy libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libv4l-dev libtbb-dev qtbase5-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils unzip liblmdb-dev libleveldb-dev libsnappy-dev libboost-dev

sudo apt install qt5-default

sudo apt install libboost-all-dev



1. install the correct protobuf packages (version 3.6.1) as given:-

# Make sure you grab the latest version
wget https://github.com/google/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip

# Unzip
unzip protoc-3.6.1-linux-x86_64.zip -d protoc3

# Move protoc to /usr/local/bin/
sudo cp -r protoc3/bin/* /usr/local/bin/

# Move protoc3/include to /usr/local/include/
sudo cp -r protoc3/include/* /usr/local/include/

# Optional: change owner
sudo chown $USER /usr/local/bin/protoc
sudo chown -R $USER /usr/local/include/google

sudo ldconfig

1.1 (optional, ignore) install the correct protobuf packages (version 3.1.0) as given in https://ninja5code.blogspot.com/2019/02/protobuf-error.html
 $  sudo add-apt-repository ppa:maarten-fonville/protobuf
 $  sudo apt-get update
 $  sudo apt-get remove libprotobuf-dev libprotoc-dev protobuf-compiler
 $  sudo apt-get install libprotobuf-dev libprotoc-dev protobuf-compiler
 $  protoc --version
 $  dpkg -l | grep protobuf

2. download zip files of opencv 3.4.4 and opencv-contrib 3.4.4 from https://github.com/opencv/opencv/releases/tag/3.4.4, https://github.com/opencv/opencv_contrib/releases/tag/3.4.4. You should unzip it into the folder as /home/ninja/opencv and /home/ninja/opencv-contrib-3.4.4, respectively. Then, build it with 'cmake ..' and 'make -j$(nproc)' and 'sudo make install'
 $  cd opencv
 $  cd build/
 $  rm -r ./*
 $  cmake ..
 $  make -j32
 $  sudo make install

2.1 if there is any wrong reference to xxx.lib, you can list down all the dependencies using:
$ ldd /usr/local/lib/xxx.lib 

2.2 Or you can check the library path to see the additional path within it:-
$ echo $LD_LIBRARY_PATH

Then, manually set it using:
$ export LD_LIBRARY_PATH = xxx:xxx

2.3 If cmake .. failed to build it, check the log printed by cmake, then install the dependencies if necessary, for example, failed to find libssl:-
$ sudo apt search libssl
$ sudo apt install libssl-dev

2.4 If make -j32 failed to build, you may try the following cmake:-
sudo cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-3.4.4/distribute \
            -D INSTALL_C_EXAMPLES=ON \
            -D OPENCV_GENERATE_PKGCONFIG=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D OPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-3.4.4/distribute/python3 \
            -D WITH_QT=ON \
            -D WITH_OPENGL=ON \
            -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-3.4.4/modules \
            -D BUILD_NEW_PYTHON_SUPPORT=ON \
            -D BUILD_opencv_python3=ON \
            -D HAVE_opencv_python3=ON \
            -D PYTHON3_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_DEFAULT_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_INCLUDE_PATH=/home/ninja/anaconda3/include/python3.7m \
            -D PYTHON3_NUMPY_INCLUDE_DIRS=/home/ninja/anaconda3/lib/python3.7/site-packages/numpy/core/include \
            -D PYTHON3_PACKAGES_PATH=/home/ninja/anaconda3/lib/python3.7/site-packages \
            -D PYTHON3_LIBRARIES=/home/ninja/anaconda3/lib/libpython3.7m.so    \
            -D PYTHON3_LIBRARIES_PATH=/home/ninja/anaconda3/lib    \
            -D BUILD_EXAMPLES=ON ..

(make sure you only have single version of library in LD_LIBRARY_PATH, for example, you cannot have protobuf 3.8.0 in /usr/lib/x86_64-linux-gnu/ and protobuf 3.1.0 in anaconda. You need to uninstall both and install protobuf 3.1.0 under /usr/lib/x86_64-linux-gnu only)

2.5 If you want to build with CUDA 8.0 and CUDNN 6.0 support:-
sudo cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-3.4.4/distribute \
            -D INSTALL_C_EXAMPLES=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D WITH_CUDA=ON \
            -D WITH_CUBLAS=ON \
            -D CUDA_FAST_MATH=ON \
            -D CUDA_GENERATION=Fermi \
            -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-8.0 \
            -D OPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-3.4.4/distribute/python3 \

            -D WITH_QT=ON \
            -D WITH_OPENGL=ON \
            -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-3.4.4/modules \
            -D BUILD_NEW_PYTHON_SUPPORT=ON \
            -D BUILD_opencv_python3=ON \
            -D HAVE_opencv_python3=ON \
            -D PYTHON3_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_DEFAULT_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_INCLUDE_PATH=/home/ninja/anaconda3/include/python3.7m \
            -D PYTHON3_NUMPY_INCLUDE_DIRS=/home/ninja/anaconda3/lib/python3.7/site-packages/numpy/core/include \
            -D PYTHON3_PACKAGES_PATH=/home/ninja/anaconda3/lib/python3.7/site-packages \
            -D PYTHON3_LIBRARIES=/home/ninja/anaconda3/lib/libpython3.7m.so    \
            -D PYTHON3_LIBRARIES_PATH=/home/ninja/anaconda3/lib    \
            -D BUILD_EXAMPLES=ON ..


2.6 If you want to build with CUDA 10.0 and CUDNN 7.6 support:-
sudo cmake -D CMAKE_BUILD_TYPE=RELEASE \
            -D CMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-3.4.4/distribute \
            -D INSTALL_C_EXAMPLES=ON \
            -D INSTALL_PYTHON_EXAMPLES=ON \
            -D WITH_TBB=ON \
            -D WITH_V4L=ON \
            -D WITH_CUDA=ON \
            -D WITH_CUBLAS=ON \
            -D BUILD_opencv_cudacodec=OFF \
            -D CUDA_FAST_MATH=ON \
            -D CUDA_GENERATION=Kepler \
            -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.0 \
            -D OPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-3.4.4/distribute/python3 \

            -D WITH_QT=ON \
            -D WITH_OPENGL=ON \
            -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-3.4.4/modules \
            -D BUILD_NEW_PYTHON_SUPPORT=ON \
            -D BUILD_opencv_python3=ON \
            -D HAVE_opencv_python3=ON \
            -D PYTHON3_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_DEFAULT_EXECUTABLE=/home/ninja/anaconda3/bin/python \
            -D PYTHON3_INCLUDE_PATH=/home/ninja/anaconda3/include/python3.7m \
            -D PYTHON3_NUMPY_INCLUDE_DIRS=/home/ninja/anaconda3/lib/python3.7/site-packages/numpy/core/include \
            -D PYTHON3_PACKAGES_PATH=/home/ninja/anaconda3/lib/python3.7/site-packages \
            -D PYTHON3_LIBRARIES=/home/ninja/anaconda3/lib/libpython3.7m.so    \
            -D PYTHON3_LIBRARIES_PATH=/home/ninja/anaconda3/lib    \
            -D BUILD_EXAMPLES=ON ..

Build with Opencv-4.4.0, Opencv_Contrib-4.4.0, Ubuntu18, Cuda10-2

{
    "cmake.configureArgs": [
        "-DCMAKE_BUILD_TYPE=RELEASE",
        "-DCMAKE_INSTALL_PREFIX=/home/ninja/workspace/opencv-4.4.0/distribute",
        "-DINSTALL_C_EXAMPLES=ON",
        "-DOPENCV_GENERATE_PKGCONFIG=ON",
        "-DINSTALL_PYTHON_EXAMPLES=ON",
        "-DBUILD_SHARED_LIBS=ON"
        "-DWITH_TBB=ON",
        "-DWITH_V4L=ON",
        "-DBUILD_opencv_world=OFF",
        "-DOPENCV_PYTHON3_INSTALL_PATH=/home/ninja/workspace/opencv-4.4.0/distribute/python3",
        "-DWITH_QT=ON",
        "-DWITH_OPENGL=ON",
        "-DWITH_CUDA=ON",
        "-DWITH_CUBLAS=ON",
        "-DBUILD_opencv_cudacodec=ON",
        "-DCUDA_FAST_MATH=ON",
        "-DCUDA_GENERATION=Kepler",
        "-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.2",
        "-DOPENCV_EXTRA_MODULES_PATH=/home/ninja/workspace/opencv_contrib-4.4.0/modules",
        "-DBUILD_NEW_PYTHON_SUPPORT=ON",
        "-DBUILD_opencv_python3=ON",
        "-DHAVE_opencv_python3=ON",
        "-DPYTHON3_EXECUTABLE=/home/ninja/anaconda3/bin/python",
        "-DPYTHON3_DEFAULT_EXECUTABLE=/home/ninja/anaconda3/bin/python",
        "-DPYTHON3_INCLUDE_PATH=/home/ninja/anaconda3/include/python3.8",
        "-DPYTHON3_NUMPY_INCLUDE_DIRS=/home/ninja/anaconda3/lib/python3.8/site-packages/numpy/core/include",
        "-DPYTHON3_PACKAGES_PATH=/home/ninja/anaconda3/lib/python3.8/site-packages",
        "-DPYTHON3_LIBRARIES=/home/ninja/anaconda3/lib/libpython3.8.so",
        "-DPYTHON3_LIBRARIES_PATH=/home/ninja/anaconda3/lib",
        "-DBUILD_EXAMPLES=ON"
        ]
}

2.7. if you have this error: The imported target "Qt5::Gui" references the file, "/usr/lib/x86_64-linux-gnu/libEGL.so"
sudo rm /usr/lib/x86_64-linux-gnu/libEGL.so; sudo ln /usr/lib/x86_64-linux-gnu/libEGL.so.1 /usr/lib/x86_64-linux-gnu/libEGL.so

2.8. if you have this error: The imported target "Qt5::Gui" references the file, "/usr/lib/x86_64-linux-gnu/libGL.so" 
sudo rm /usr/lib/x86_64-linux-gnu/libGL.so; sudo ln /usr/lib/x86_64-linux-gnu/libGL.so.1 /usr/lib/x86_64-linux-gnu/libGL.so


2.9. If you have this error: CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: CUDA_nppi_LIBRARY (ADVANCED)
$ cd ~/workspace/opencv-2.4.13.6/cmake
$ cp ../../opencv-3.4.4/cmake/FindCUDA.cmake ./ $ cp ../../opencv-3.4.4/cmake/OpenCVDetectCUDA.cmake ./
$ cd ../build
$ make all -j32

2.10. If you have this error, opencv_dep_CUDA_nppi_LIBRARY is not found:-

replace the root > cmake > FindCUDA.cmake from opencv3.4.4 to 2.4.13.6

2.11. If you have this error, "gflags-2.1.1/src/gflags.cc:90:20: fatal error: config.h: No such file or directory"

cmake version is too new, need to downgrade

3. create a conda virtual env named as caffe2 with python2.7, install:
$ pip install scikit-image
$ pip install protobuf (after completed step 2)

4. download caffe and build it with 'cmake ..' and 'make -j$(nproc)' and 'sudo make install'
 $  cd caffe/
 $  cd build/
 $  sudo rm -r ./*
 $  cmake .. -DCMAKE_INSTALL_PREFIX:PATH=/home/ninja/.conda/envs/caffe
 $  make -j32
 $  sudo make install
 $  make runtest
 $  make pycaffe 

4.1 if sudo make install failed due to undefined reference to `ZSTD_initCStream, run:
$ conda list
$ conda install libtiff (it should install zstd)

or you can just modify the library path by pointing to system libraries only

$ gedit ~/.bashrc
>> # insert into last line
>> export $LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu

$ source ~/.bashrc
$ conda activate caffe2(compile again, make -j32)

4.2 C++ exception with description "locale::facet::_S_create_c_locale name not valid" thrown in SetUp().
$ export LC_ALL="C"

(rerun make runtest) 

4.3 If you received the following error:-

CXX/LD -o .build_release/tools/convert_imageset.bin .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64ToArray(unsigned long long, unsigned char*)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::WriteInt64(int, long long, google::protobuf::io::CodedOutputStream*)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::UnknownFieldSet::AddVarint(int, unsigned long long)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::VarintSize64(unsigned long long)' .build_release/lib/libcaffe.so: undefined reference to `google::protobuf::io::CodedOutputStream::WriteVarint64(unsigned long long)' collect2: error: ld returned 1 exit status make: *** [.build_release/tools/convert_imageset.bin] Error 1

$ ldd .build_release/lib/libcaffe.so 

(check libcaffe.so is using which library, it should point to libprotobuf.so.11 => /usr/local/lib/libprotobuf.so.11. If it links to library under anaconda, just delete it.) 

5.If all succeed, you need to set the pythonpath as:-
$ export PYTHONPATH=/home/ninja/.conda/envs/caffe/python:$PYTHONPATH

6. run python and then import caffe, you may need to install protobuf as pip install protobuf==3.1.0 and pip install scikit-image (check using protoc --version)

7. /usr/bin/ld: cannot find -llib64

check the CUDNN_LIBRARY or CUDA_LIBRARY path, should be full path to the specific library, /home/ubuntu/workspace/cuda/lib64/libcudnn.so, not folder name /home/ubuntu/workspace/cuda/lib64/

 

8. /home/ubuntu/workspace/caffe-opencv4.2-cuda10.0/include/caffe/proto/caffe.pb.h:17:2: error: #error This file was generated by an older version of protoc which is

or 

./include/caffe/util/cudnn.hpp:8:34: fatal error: caffe/proto/caffe.pb.h: No such file or directory

$ protoc src/caffe/proto/caffe.proto --cpp_out=.
$ mkdir include/caffe/proto
$ mv src/caffe/proto/caffe.pb.h include/caffe/proto

9. how to force cmake to use one particular cuda architecture?

-DCUDA_ARCH_NAME=Manual -DCUDA_ARCH_BIN="75" -DCUDA_ARCH_PTX="75"

 

Example of .bashrc for building opencv
PATH=$PATH:$HOME/anaconda/bin:/usr/local/cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/lib/x86_64-linux-gnu/:/home/ninja/anaconda3/lib

Example of .bashrc for building caffe
export PYTHONPATH=/home/ninja/.conda/envs/caffe/python:$PYTHONPATH
PATH=$PATH:$HOME/anaconda/bin:/usr/local/cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/lib/x86_64-linux-gnu/:/home/ninja/.conda/envs/caffe/lib 

10. (-218:No OpenGL support) Library was built without OpenGL support in function 'CvWindow'：-

        -D OpenGL_GL_PREFERENCE=GLVND \
        -D WITH_QT=ON \
        -D WITH_OPENGL=ON \
        -D WITH_GTK_2_X=ON \

 

11. How to find the correct path of opencv from a new C++ project:

During opencv compilation, set the install path in cmakelists.txt:

  -D CMAKE_INSTALL_PREFIX=/home/peseyes/Documents/opencv-4.4.0/distribute \

During the new project compilation, set the opencv path in cmakelists.txt:-

SET(OpenCV_DIR /home/peseyes/Documents/opencv-4.4.0/distribute/share/opencv4) find_package(OpenCV 4 REQUIRED)