How to do multiprocessing in python?

import pickle
import numpy as np
import glob, gzip
import multiprocessing as mp
from functools import partial
import tqdm

def daemon(paths, idx):
    item = paths[idx].split(' ')
    data = pickle.load(gzip.open(item[0], 'rb'))
    images = data.astype(np.float32)
    print(item[0], images.shape)

root = '/data/ninja/workspace/data/boyshome/train.txt'
ifile = open(root)
files = ifile.readlines()
indices = list(range(len(files)))

worker = partial(daemon, files)
pool = mp.Pool(24)
tqdm.tqdm(pool.map(func=worker, iterable=indices))
pool.close()
pool.join()

How to process all files in a folder?

for i in *.avi; do ffmpeg -i "$i" -vcodec libx264 "${i%.*}_.mp4"; done

How to remove the files if list is too long in ubuntu?

(base) temp@temp:~/temp$ rm -r ./*
bash: /bin/rm: Argument list too long
(base) temp@temp:~/temp$ find . -type f -name '*.*' | xargs rm
(base) temp@temp:~/temp$ ls

How to view an image or open a pdf file from ubuntu terminal?

eog image.jpg

evince file.pdf

How to install Nvidia driver in a proper way?

Ref: https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html

This section includes instructions for installing the NVIDIA driver on Ubuntu 16.04 LTS and Ubuntu 18.04 LTS distributions using the package manager.

1. The NVIDIA driver requires that the kernel headers and development packages for the running version of the kernel be installed at the time of the driver installation, as well whenever the driver is rebuilt. For example, if your system is running kernel version 4.4.0, the 4.4.0 kernel headers and development packages must also be installed. The kernel headers and development packages for the currently running kernel can be installed with:

   $ sudo apt-get install linux-headers-$(uname -r)

2. Ensure packages on the CUDA network repository have priority over the Canonical repository.

   $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')

   $ wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin

   $ sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600

3. Install the CUDA repository public GPG key. Note that on Ubuntu 16.04, replace https with http in the command below.

   $ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/7fa2af80.pub

4. Setup the CUDA network repository.

   $ echo "deb http://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | sudo tee /etc/apt/sources.list.d/cuda.list

5. Update the APT repository cache and install the driver using the cuda-drivers meta-package. Use the --no-install-recommends option for a lean driver install without any dependencies on X packages. This is particularly useful for headless installations on cloud instances.

   $ sudo apt-get update

   $ sudo apt-get -y install cuda-drivers

6. Follow the post-installation steps in the CUDA Installation Guide for Linux to setup environment variables, NVIDIA persistence daemon (recommended) and to verify the successful installation of the driver.

How to convert indices to one hot encoding, or vice versa?

from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

def indices_to_one_hot(data, nb_classes):
    """Convert an iterable of indices to one-hot encoded labels."""
    targets = np.array(data).reshape(-1)
    return np.eye(nb_classes)[targets]

def one_hot_to_indices(data):
    return np.argmax(data, axis=1)

 

#########################################

pred = one_hot_to_indices(one_hot_pred)
targ = one_hot_to_indices(one_hot_label)
prec1 = accuracy_score(pred, targ) # [0,1,1,0], [0,0,1,0]

cf = confusion_matrix(video_labels, video_pred).astype(float)
cr = classification_report(video_labels, video_pred)
print('confusion_matrix')
print(cf)
print('classification_report')
print(cr)

np.save('cm.npy', cf)
cls_cnt = cf.sum(axis=1)
cls_hit = np.diag(cf)

cls_acc = cls_hit / cls_cnt
print(cls_acc)
upper = np.mean(np.max(cf, axis=1) / cls_cnt)
print('upper bound: {}'.format(upper))

print('-----Evaluation is finished------')
print('Class Accuracy {:.02f}%'.format(np.mean(cls_acc) * 100))
print('Overall Prec@1 {:.02f}%'.format(top1.avg))

 

To check the status of a FOR looping

You can try to use tqdm.tqdm

for file in tqdm.tqdm(files):

    print(file)