Goku Coding Experience: July 2019

This is similar to the previous post where we change the focus to python3.5.

Make sure you have install caffe for python2.7 before continue reading this post.

Here are few steps I have done for successfully compiling the caffe for python 3.5.

1. Clone the default caffe folder into a new folder named as caffe3. Then, delete all files in the build folder,
$ cp ./caffe/* ./caffe3/
$ sudo rm -r ./caffe3/build/*

2. Create a new conda environment and I named it as caffe3 as well. Make sure you use python3.5 as following,
$ conda create -n caffe3 python=3.5

3. Activate the caffe3 env and then install the numpy as following,
$ conda activate caffe3
$ pip install numpy
$ pip install scikit-image

4. Install additional dependencies as following:

$ sudo apt-get install python3-skimage
$ sudo apt-get install python3-protobuf

5. Once completed, cd to the caffe3 root folder and copy the makefile as following:
$ cd ./caffe3
$ cp Makefile.config.example Makefile.config

6. You will see a new file named as Makefile.config under ./caffe folder. Copy and paste the Makefile content as given in the appendix below. Note that please change the path accordingly as my anaconda env was installed at /home/ninja/.conda/envs/*, yours might be different such as /home/ninja/anaconda3/envs/*.

7. Once everything is ready, you are good to compile caffe, make sure you are in the ./caffe3 folder:
$ make clean
$ make all -j32
$ make test
$ make runtest
$ make pycaffe
$ make distribute
(you should be able to see the distribute folder under caffe3)

8.If you have no error in compilation, congratulation! Next is to set the path in ~/.bashrc and you should able to import caffe.
$ sudo gedit ~/.bashrc

then add the following lines into ~/.bashrc, save it and source it

export PYTHONPATH=/home/ninja/caffe3/distribute/python:$PYTHONPATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/lib/x86_64-linux-gnu/:/home/ninja/caffe3/distribute/lib

$ source ~/.bashrc
$ conda activate caffe3
$ python (version 3.5.*)
>>> import caffe
>>> (pray, no error)

Example1 of Error:
./include/caffe/util/signal_handler.h:4:34: fatal error: caffe/proto/caffe.pb.h: No such file or directory

If you are using conda, you need to include the env lib into the LD_LIBRARY_PATH:
$ export LD_LIBRARY_PATH=/home/ninja/.conda/envs/caffe3/lib

Example2 of Error:
xxx... .nccl.h: No such file or directory

If you are using conda, you need to copy all include and lib files into env:
$ cp nccl_2.4.7-1+cuda10.0_x86_64/include/nccl* /home/ninja/.conda/envs/c3d11/include/
$ cp nccl_2.4.7-1+cuda10.0_x86_64/lib/libnccl* /home/ninja/.conda/envs/c3d11/lib/

Example3 of Error:
xxx... .cudnn.h: No such file or directory

If you are using conda, you need to copy all include and lib files into env:
$ cp cuda/include/cudnn.h /home/ninja/.conda/envs/c3d11/include/
$ cp cuda/lib64/libcudnn* /home/ninja/.conda/envs/c3d11/lib/

I am using cudnn-10.0-linux-x64-v7.6.0.64.

Example4 of Error:
/usr/include/boost/python/detail/wrap_python.hpp:50:23: fatal error: pyconfig.h: No such file or directory
$ locate pyconfig.h
>> set the path in Makefile.config > PYTHON_INCLUDE

Example5 of Error:
Solve the problem: "cannot find -lboost_python3" when using Python3 Ubuntu16.04
$ cd /usr/lib/x86_64-linux-gnu
$ sudo ln -s libboost_python-py35.so libboost_python3.so

Example6 of Error:
Solve the problem: "/usr/bin/ld: cannot find -lopencv_imgcodecs" when using Python3 Ubuntu16.04, modify Makefile.config:-

PYTHON_LIBRARIES := boost_python3 python3.5m opencv_imgcodecs

PYTHON_LIB := $(ANACONDA_HOME)/lib /home/ninja/anaconda3/envs/caffe3/lib /home/ninja/workspace/opencv-3.4.4/distribute/lib

Example of Makefile, CUDA 8.0, CUDNN 6.0 used in this post.

## Refer to http://caffe.berkeleyvision.org/installation.html

# Contributions simplifying and improving our build system are welcome!

LINKFLAGS := -Wl,-rpath,$(HOME)/anaconda3/lib


# cuDNN acceleration switch (uncomment to build with cuDNN).

USE_CUDNN := 1


# CPU-only switch (uncomment to build without GPU support).

# CPU_ONLY := 1


# uncomment to disable IO dependencies and corresponding data layers

# USE_OPENCV := 0

# USE_LEVELDB := 0

# USE_LMDB := 0

# This code is taken from https://github.com/sh1r0/caffe-android-lib

# USE_HDF5 := 0


# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)

#    You should not set this flag if you will be reading LMDBs with any

#    possibility of simultaneous read and write

ALLOW_LMDB_NOLOCK := 1


# Uncomment if you're using OpenCV 3

OPENCV_VERSION := 3


# To customize your choice of compiler, uncomment and set the following.

# N.B. the default for Linux is g++ and the default for OSX is clang++

# CUSTOM_CXX := g++


# CUDA directory contains bin/ and lib/ directories that we need.

CUDA_DIR := /usr/local/cuda-8.0

# On Ubuntu 14.04, if cuda tools are installed via

# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:

# CUDA_DIR := /usr


# CUDA architecture setting: going with all of them.

# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.

# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.

# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_60,code=sm_61


# BLAS choice:

# atlas for ATLAS (default)

# mkl for MKL

# open for OpenBlas

BLAS := atlas

# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.

# Leave commented to accept the defaults for your choice of BLAS

# (which should work)!

# BLAS_INCLUDE := /path/to/your/blas

# BLAS_LIB := /path/to/your/blas


# Homebrew puts openblas in a directory that is not on the standard search path

# BLAS_INCLUDE := $(shell brew --prefix openblas)/include

# BLAS_LIB := $(shell brew --prefix openblas)/lib


# This is required only if you will compile the matlab interface.

# MATLAB directory should contain the mex binary in /bin.

# MATLAB_DIR := /usr/local

# MATLAB_DIR := /Applications/MATLAB_R2012b.app


# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

# PYTHON_INCLUDE := /usr/include/python2.7 \

        /usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

ANACONDA_HOME := /home/ninja/.conda/envs/caffe3

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

        $(ANACONDA_HOME)/include/python3.5 \

        $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include


# Uncomment to use Python 3 (default is Python 2)

PYTHON_LIBRARIES := boost_python3 python3.5m

PYTHON_INCLUDE := /home/ninja/.conda/envs/caffe3/include/python3.5m \

                /home/ninja/.conda/envs/caffe3/lib/python3.5/site-packages/numpy/core/include


# We need to be able to find libpythonX.X.so or .dylib.

# PYTHON_LIB := /usr/lib

PYTHON_LIB := $(ANACONDA_HOME)/lib /home/ninja/.conda/envs/caffe3/lib


# Homebrew installs numpy in a non standard path (keg only)

# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include

# PYTHON_LIB += $(shell brew --prefix numpy)/lib


# Uncomment to support layers written in Python (will link against Python libs)

WITH_PYTHON_LAYER := 1


# Whatever else you find you need goes here.

#INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include

#LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial /home/ninja/opencv/include/ /home/ninja/.conda/envs/caffe3/include

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial /home/ninja/opencv/build/lib


# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies

# INCLUDE_DIRS += $(shell brew --prefix)/include

# LIBRARY_DIRS += $(shell brew --prefix)/lib


# NCCL acceleration switch (uncomment to build with NCCL)

# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)

USE_NCCL := 1


# Uncomment to use `pkg-config` to specify OpenCV library paths.

# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)

USE_PKG_CONFIG := 1


# N.B. both build and distribute dirs are cleared on `make clean`

BUILD_DIR := build

DISTRIBUTE_DIR := distribute


# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171

# DEBUG := 1


# The ID of the GPU that 'make runtest' will use to run unit tests.

TEST_GPUID := 0


# enable pretty build (comment to see full commands)

Q ?= @

Example of Makefile, CUDA 10.0, CUDNN 7.6 used in this post.

## Refer to http://caffe.berkeleyvision.org/installation.html

# Contributions simplifying and improving our build system are welcome!

LINKFLAGS := -Wl,-rpath,$(HOME)/anaconda3/lib


# cuDNN acceleration switch (uncomment to build with cuDNN).

USE_CUDNN := 1


# CPU-only switch (uncomment to build without GPU support).

# CPU_ONLY := 1


# uncomment to disable IO dependencies and corresponding data layers

# USE_OPENCV := 0

# USE_LEVELDB := 0

# USE_LMDB := 0

# This code is taken from https://github.com/sh1r0/caffe-android-lib

# USE_HDF5 := 0


# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)

#    You should not set this flag if you will be reading LMDBs with any

#    possibility of simultaneous read and write

ALLOW_LMDB_NOLOCK := 1


# Uncomment if you're using OpenCV 3

OPENCV_VERSION := 3


# To customize your choice of compiler, uncomment and set the following.

# N.B. the default for Linux is g++ and the default for OSX is clang++

# CUSTOM_CXX := g++


# CUDA directory contains bin/ and lib/ directories that we need.

CUDA_DIR := /usr/local/cuda-10.0

# On Ubuntu 14.04, if cuda tools are installed via

# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:

# CUDA_DIR := /usr


# CUDA architecture setting: going with all of them.

# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.

# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.

# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_75,code=sm_75


# BLAS choice:

# atlas for ATLAS (default)

# mkl for MKL

# open for OpenBlas

BLAS := atlas

# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.

# Leave commented to accept the defaults for your choice of BLAS

# (which should work)!

# BLAS_INCLUDE := /path/to/your/blas

# BLAS_LIB := /path/to/your/blas


# Homebrew puts openblas in a directory that is not on the standard search path

# BLAS_INCLUDE := $(shell brew --prefix openblas)/include

# BLAS_LIB := $(shell brew --prefix openblas)/lib


# This is required only if you will compile the matlab interface.

# MATLAB directory should contain the mex binary in /bin.

# MATLAB_DIR := /usr/local

# MATLAB_DIR := /Applications/MATLAB_R2012b.app


# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

# PYTHON_INCLUDE := /usr/include/python2.7 \

        /usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

ANACONDA_HOME := /home/ninja/.conda/envs/caffe3

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

        $(ANACONDA_HOME)/include/python3.5 \

        $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include


# Uncomment to use Python 3 (default is Python 2)

PYTHON_LIBRARIES := boost_python3 python3.5m

PYTHON_INCLUDE := /home/ninja/.conda/envs/caffe3/include/python3.5m \

                /home/ninja/.conda/envs/caffe3/lib/python3.5/site-packages/numpy/core/include


# We need to be able to find libpythonX.X.so or .dylib.

# PYTHON_LIB := /usr/lib

PYTHON_LIB := $(ANACONDA_HOME)/lib /home/ninja/.conda/envs/caffe3/lib


# Homebrew installs numpy in a non standard path (keg only)

# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include

# PYTHON_LIB += $(shell brew --prefix numpy)/lib


# Uncomment to support layers written in Python (will link against Python libs)

WITH_PYTHON_LAYER := 1


# Whatever else you find you need goes here.

#INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include

#LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial /home/ninja/opencv/include/ /home/ninja/.conda/envs/caffe3/include

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial /home/ninja/opencv/build/lib


# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies

# INCLUDE_DIRS += $(shell brew --prefix)/include

# LIBRARY_DIRS += $(shell brew --prefix)/lib


# NCCL acceleration switch (uncomment to build with NCCL)

# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)

USE_NCCL := 1


# Uncomment to use `pkg-config` to specify OpenCV library paths.

# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)

USE_PKG_CONFIG := 1


# N.B. both build and distribute dirs are cleared on `make clean`

BUILD_DIR := build

DISTRIBUTE_DIR := distribute


# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171

# DEBUG := 1


# The ID of the GPU that 'make runtest' will use to run unit tests.

TEST_GPUID := 0


# enable pretty build (comment to see full commands)

Q ?= @

Goku Coding Experience

Thursday, 25 July 2019

cmake Could NOT find Atlas

conda user permission

Thursday, 18 July 2019

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument

Sunday, 7 July 2019

how to build caffe in ubuntu16, cuda10.0, cudnn7.6, python3.5