Training loss not changing

Issue: when running the dla training, both the training and testing losses are decreasing, however, the mA of results are not changing, always stay around 50%.

Solution: Check the loss function, check the learning rate, check using the pretrained model or not

Downgrading gcc from 5.4 to 4.9 in Ubuntu 16.04

Error:

/usr/local/cuda/include/host_config.h:115:2: error: #error — unsupported GNU version! gcc versions later than 5.3 are not supported!
#error — unsupported GNU version! gcc versions later than 5.3 are not supported!

$ gcc –version

gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Uninstall

$ sudo apt-get remove gcc g++

Done.
Removing cuda (8.0.44-1) …
Removing cuda-8-0 (8.0.44-1) …
Removing cuda-toolkit-8-0 (8.0.44-1) …
Removing cuda-documentation-8-0 (8.0.44-1) …
Removing cuda-samples-8-0 (8.0.44-1) …
Removing cuda-visual-tools-8-0 (8.0.44-1) …
Removing cuda-command-line-tools-8-0 (8.0.44-1) …
Removing cuda-core-8-0 (8.0.44-1) …
Removing build-essential (12.1ubuntu2) …
Removing cuda-demo-suite-8-0 (8.0.44-1) …
Removing cuda-runtime-8-0 (8.0.44-1) …
Removing cuda-drivers (367.48-1) …
Removing nvidia-367-dev (367.57-0ubuntu0.16.04.1) …
Removing nvidia-opencl-icd-367 (367.57-0ubuntu0.16.04.1) …
Removing g++ (4:5.3.1-1ubuntu1) …
Removing liblapack-dev (3.6.0-2ubuntu2) …
Removing libatlas-base-dev (3.10.2-9) …
update-alternatives: using /usr/lib/libblas/libblas.so to provide /usr/lib/libblas.so (libblas.so) in auto mode
Removing libatlas-dev (3.10.2-9) …
Removing libblas-dev (3.6.0-2ubuntu2) …
Removing gfortran (4:5.3.1-1ubuntu1) …
Removing libcuda1-367 (367.57-0ubuntu0.16.04.1) …
Removing nvidia-367 (367.57-0ubuntu0.16.04.1) …
Removing all DKMS Modules
Done.

Check gcc again:

$ gcc –version
-bash: /usr/bin/gcc: No such file or directory

Install gcc 4.9/g++ 4.9

$ sudo apt-get install gcc-4.9 g++-4.9

Preparing to unpack …/gcc-4.9-base_4.9.3-13ubuntu2_amd64.deb …
Unpacking gcc-4.9-base:amd64 (4.9.3-13ubuntu2) …
Selecting previously unselected package cpp-4.9.
Preparing to unpack …/cpp-4.9_4.9.3-13ubuntu2_amd64.deb …
Unpacking cpp-4.9 (4.9.3-13ubuntu2) …
Selecting previously unselected package libasan1:amd64.
Preparing to unpack …/libasan1_4.9.3-13ubuntu2_amd64.deb …
Unpacking libasan1:amd64 (4.9.3-13ubuntu2) …
Selecting previously unselected package libgcc-4.9-dev:amd64.
Preparing to unpack …/libgcc-4.9-dev_4.9.3-13ubuntu2_amd64.deb …
Unpacking libgcc-4.9-dev:amd64 (4.9.3-13ubuntu2) …
Selecting previously unselected package gcc-4.9.
Preparing to unpack …/gcc-4.9_4.9.3-13ubuntu2_amd64.deb …
Unpacking gcc-4.9 (4.9.3-13ubuntu2) …
Selecting previously unselected package libstdc++-4.9-dev:amd64.
Preparing to unpack …/libstdc++-4.9-dev_4.9.3-13ubuntu2_amd64.deb …
Unpacking libstdc++-4.9-dev:amd64 (4.9.3-13ubuntu2) …
Selecting previously unselected package g++-4.9.
Preparing to unpack …/g++-4.9_4.9.3-13ubuntu2_amd64.deb …
Unpacking g++-4.9 (4.9.3-13ubuntu2) …
Processing triggers for man-db (2.7.5-1) …
Processing triggers for libc-bin (2.23-0ubuntu4) …
Setting up gcc-4.9-base:amd64 (4.9.3-13ubuntu2) …
Setting up cpp-4.9 (4.9.3-13ubuntu2) …
Setting up libasan1:amd64 (4.9.3-13ubuntu2) …
Setting up libgcc-4.9-dev:amd64 (4.9.3-13ubuntu2) …
Setting up gcc-4.9 (4.9.3-13ubuntu2) …
Setting up libstdc++-4.9-dev:amd64 (4.9.3-13ubuntu2) …
Setting up g++-4.9 (4.9.3-13ubuntu2) …
Processing triggers for libc-bin (2.23-0ubuntu4) …

Checking Version:

$ g++-4.9 –version

g++-4.9 (Ubuntu 4.9.3-13ubuntu2) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ gcc-4.9 –version

gcc-4.9 (Ubuntu 4.9.3-13ubuntu2) 4.9.3
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Note: gcc under 5.0 will use gcc-4.9/g++-4.9 as full binary name.

$ gcc –version
The program ‘gcc’ is currently not installed. You can install it by typing:
sudo apt install gcc

ref: https://aichamp.wordpress.com/2016/11/10/downgrading-gcc-from-5-4-to-4-9-in-ubuntu-16-04/