Sunday, 8 January 2023

How to save docker image locally using docker compose and then install it in another server offline?

ref: https://milvus.io/docs/install_offline-docker.md

Download files and images

To install Milvus offline, you need to pull and save all images in an online environment first, and then transfer them to the target host and load them manually.

  1. Download an installation file.
  • For Milvus standalone:
//github.com/milvus-io/milvus/releases/download/v2.2.2/milvus-standalone-docker-compose.yml -O docker-compose.yml
  • For Milvus cluster:
//github.com/milvus-io/milvus/releases/download/v2.2.2/milvus-cluster-docker-compose.yml -O docker-compose.yml
  1. Download requirement and script files.
//raw.githubusercontent.com/milvus-io/milvus/master/deployments/offline/requirements.txt
  1. Pull and save images.
pip3 install -r requirements.txt python3 save_image.py --manifest docker-compose.yml
The images are stored in the /images folder.
  1. Load the images.
cd

Install Milvus offline

Having transferred the images to the target host, run the following command to install Milvus offline.

docker-compose -f docker-compose.yml up -d

Uninstall Milvus

To uninstall Milvus, run the following command.

docker-compose -f docker-compose.yml down


===

requirements.txt:-

docker==5.0.0 nested-lookup==0.2.22

===
save_image.py:-

import argparse import docker import gzip import os import yaml from nested_lookup import nested_lookup if __name__ == "__main__": parser = argparse.ArgumentParser( description="Save Docker images") parser.add_argument("--manifest", required=True, help="Path to the manifest yaml") parser.add_argument("--save_path", type=str, default='images', help='Directory to save images to') arguments = parser.parse_args() with open(arguments.manifest, 'r') as file: template = file.read() images=[] parts = template.split('---') for p in parts: y = yaml.safe_load(p) matches = nested_lookup("image", y) if (len(matches)): images += matches save_path = arguments.save_path if not os.path.isdir(save_path): os.mkdir(save_path) client = docker.from_env() for image_name in set(images): file_name = (image_name.split(':')[0].replace("/", "-")) f = gzip.open(save_path + "/" + file_name + '.tar.gz', 'wb') try: image = client.images.get(image_name) if image.id: print ("docker image \"" + image_name + "\" already exists.") except docker.errors.ImageNotFound: print ("docker pull " + image_name + " ...") image = client.images.pull(image_name) image_tar = image.save(named=True) f.writelines(image_tar) f.close() print("Save docker images to \"" + save_path + "\"")

===
docker-compose.yml:-

version: '3.5' services: etcd: container_name: milvus-etcd image: quay.io/coreos/etcd:v3.5.0 environment: - ETCD_AUTO_COMPACTION_MODE=revision - ETCD_AUTO_COMPACTION_RETENTION=1000 - ETCD_QUOTA_BACKEND_BYTES=4294967296 - ETCD_SNAPSHOT_COUNT=50000 volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd minio: container_name: milvus-minio image: minio/minio:RELEASE.2022-03-17T06-34-49Z environment: MINIO_ACCESS_KEY: minioadmin MINIO_SECRET_KEY: minioadmin ports: - "9001:9001" volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data command: minio server /minio_data --console-address ":9001" healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.2.2 command: ["milvus", "run", "standalone"] environment: ETCD_ENDPOINTS: etcd:2379 MINIO_ADDRESS: minio:9000 volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus ports: - "19530:19530" - "9091:9091" depends_on: - "etcd" - "minio" networks: default: name: milvus