Shreyans Jain

Tag: Nvidia-docker

  • [How To] Train Deep Learning Models on Docker/NVIDIA-Docker

    [How To] Train Deep Learning Models on Docker/NVIDIA-Docker

    This post focuses on providing a short and simple tutorial of how to train your deep learning model on Dockers. For this tutorial we will be using NVIDIA DGX1 which has Ubuntu 18.04 OS. The tutorial can also be followed on previous versions of Ubuntu and on any NVIDIA enabled laptops or computers. This tutorial is for beginners, so if you have any advance query please leave a comment or contact me.

    To know more and how to install Dockers and NVIDIA-Docker please read my other article which can be found here.

    Before we move any further, I would like to make you understand my setup. That should give you probably good idea about my scenario of how I am using the commands. I am pursuing my Masters from Bennett University. We have a Supercomputer installed (NVIDIA DGX1). I have been managing it for quite while now. It can be accessed through SSH client, we just need to enter IP address and port number to access it. I generally use Putty for connecting to DGX1. And before accessing the Docker/NVIDIA-Docker I use tmux for creating sessions.

    Creating New Session and Accessing it.

    For creation of new session use the following command.

    tmux new -s session_name

    Replace the session_name with the name you want to give. When you type this, it will automatically take you to newly created session.

    To exit the session just type exit while you are in session. And if you want to just come out of session without exiting the session you can press ctrl+b and then press d.

    To re-enter your session, you can type the following command

    tmux a -t session_name

    To list all the available sessions, you can use tmux ls command.

    Advantages of Using Tmux Sessions

    Sessions which are created helps in retaining the record, it helps in keeping the session alive even after the computer is switched off. When the session is loaded again, all the works starts from where it was left. So here it will help us in keeping the container alive, so that we can use it for other programs as well. Use the Tmux Cheatsheet link for more details on usage of Tmux command to manage the sessions.

    Pulling a Docker/NVIDIA-Docker Container for Deep Learning Models

    Now once you are in tmux session, you can start a docker which can be used for training deep learning model. I generally prefer using NVIDIA-Docker for Deep Learning Models. As we already know that Docker commands are used to pull already created containers. There are few containers which contain TensorFlow, Caffe, PyTorch and whichever deep learning libraries are available. For this tutorial I will be using TensorFlow Container. Check the list of links given bellow for other containers:

    First step is to pull the container from the source. To pull a container you can use the following command. The command is also used for accessing the docker container for later usages. 

    NV_GPU='0' nvidia-docker run --name name_of_container --rm -it -v /home/dgxuser104/:/home/dgxuser104/ tensorflow/tensorflow:latest-gpu-py3

    In the above command there are many options and flags which are absolutely confusing. So I will explain every thing one by one.

    • NV_GPU is used to assign specific GPU to the docker. In my case we have 8 GPU which we can use, out of which I am using GPU_ID: 0 .
    • nvidia-docker is the main command to pull the container.
    • For –name option, you have to provide whatever name you prefer for the command, just replace name_of_container with the name you want to give.
    • When you exit docker, it still remains in the system, so you need to ensure that once your task is over them memory which is assigned to docker should be freed. To do so we use –rm option.
    • The next option is -it flag. When we want to train our own model we require a interactive command line environment. So, if we do not use it, we will not be able to create our files and train our model. So just use this.
    • The -v flag is used to mount some drive to the container, so that whatever files you want to create or use, you can pick it using this flag. In the command /home/dgxuser104/:/home/dgxuser104/ this part helps in accessing the files by the container in the specified location.
    • tensorflow/tensorflow:latest-gpu-py3 is the container that we are accessing. Here after the tensorflow/tensorflow means that we are downloading the TensorFlow container from TensorFlow repository, and after: is the specification of the container we want. So, here our container is a latest version of TensorFlow (that is TensorFlow v1.13.1) which is pre-configured to use GPU and have only Python3 support. To use different containers change the name of container.

    When we run the command for the first time, the container is downloaded from it’s source repository. And once it is downloaded, you will get a interactive interface. Now, obviously here interactive means that you will be able to create or delete any file and use the ability of container for your advantage. Make a note that once you are in the container, you have limited root access, every file that is created is owned by root.

    Training Your Deep Learning Model

    To train our Deep Learning Model, we will require to write a code in Python3, as we have downloaded the container with py3. We can create our model using either Keras or TensorFlow. Note that if you have provided multiple GPU’s to Container (in NV_GPU) then the model should be Multi-GPU Model, so that it can exploit the complete capability of the GPU’s (This is optional if you have only one GPU). Follow the following steps to start training your model:

    1. Start a Tmux Session. (See “Creating New Session and Accessing it” Section for help.)
    2. Once you are into a session, you need to pull the container. (See “Pulling a Docker/NVIDIA-Docker Container for Deep Learning Models” section for more details.)
    3. Install the packages which are required for the model. Some of the packages are generally preinstalled, but some of them need to be installed. Download the install.sh file (Scroll to the end of the article to find the download link). To run the .sh file you will need to follow the following steps
      • First change the permissions of the file by chmod 777 install.sh
      • Edit the file according to your usage.
      • Save the file and run by ./install.sh
    4. Once you have all the supporting packages you need to create all the python files for training your model. And just run as python filename_of_model.py
    5. Once you train all the models, you will see the saved model files. Now, we know that in container every file that is created is owned by root, so once you exit the container the files will not be accessible to you. So before you exit the container give permission to all the files created. To give the permission to file use this command – chmod 777 file1 file2 file3 ... This step is mandatory for those who do not have root permissions for their systems.
    6. Now you have all the trained files which can be used further for your application.

    Download the Install.sh file by clicking on the download button.

    Download


    I hope the article helps in understanding the Dockers and Containers in more detail. If you have any query related to the article, you can make a comment with your issue or you can contact me through the contact me page.

    Thank You for reading the article. If you feel the article helps you in any way, please Like, share and comment on the article. Your feedback is appreciated.

  • [How to] Install Docker and NVIDIA-Docker in Ubuntu 18.04

    [How to] Install Docker and NVIDIA-Docker in Ubuntu 18.04

    This post focuses on providing a short and simple tutorial of how to install Docker and NVIDIA-Docker on your Linux system. For this tutorial I am using NVIDIA DGX1 which has Ubuntu 18.04 OS. The tutorial can also be followed on older versions of Ubuntu. This tutorial is for sudo users only, if you do not have sudo access please contact your system’s administrator. For any other query on the following tutorial you can make a comment or contact me.

    Docker & NVIDIA-Docker

    Both Docker and NVIDIA-Docker are tools to create, develop, test and deploy applications using container. Container is like a module which contain all the libraries and other dependencies which are required for a particular application. So once a container is created, the whole thing can be delivered to use it directly without any other issues of installations. OS like Linux and MAC, can directly use these containers to run an application. Basically, Docker and NVIDIA-Docker acts as Virtual Machine, but without creating the whole Virtual OS. For more details, visit the corresponding links of Docker and NVIDIA-Docker.

    Steps Involved

    There are two steps involved from setting up of Docker and NVIDIA-Docker so that it can be used further. Check this article to see how to use Docker and NVIDIA-Docker for training Deep Learning Models.

    Step 1 – Installing Docker

    First update the apt index and list by running the following command and then perform the upgrade to ensure that all the packages are up-to-date:

    $ sudo apt-get update
$ sudo apt-get upgrade

    Once you run the above commands successfully, you need to install additional dependencies to support Docker installation.

    $ sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

    Add the Docker’s official GPG Key:

    $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

    Now you need to add the official repository for Docker:

    $ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

    Once the repository is added you need to update the apt index again, and then finally install the Docker Community Edition

    $ sudo apt-get update && apt-get install docker-ce

    To verify whether the installation was successful or not you can run the following command to check:

    $ docker -v

    The output of this command should look like this: 

    >> Docker version 18.09.2, build 6247962
Note: Build number could differ.

    If there are multiple users who are going to access the docker then its better to remove the “sudo” requirement while running the code. To remove the requirement, add the users to docker group. To add the users in docker group just run the following command:

    $ sudo usermod -aG docker Username1 Username2 ...

    Now all the username that you added to the command are part of docker group. Now to access the docker from different users, they wont need to use “sudo”.

    Step 2 -Installing NVIDIA-Docker

    NVIDIA-Docker is a abstraction over Docker. They were created to exploit the copability of NVIDIA GPU’s. NVIDIA-Docker contains CUDA Drivers, so that Docker can access GPU/s. Whenever any application runs on NVIDIA-Docker, it uses the capability of GPU to speed up the execution of the application.

    First we need to set the GPG Key and remote repositories. Run the following command to get it done:

    $ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

    Update the apt index and install the NVIDIA-Docker 

    $ sudo apt-get update && apt-get install -y nvidia-docker2

    Now reload the docker demon configurations:

    $ sudo pkill -SIGHUP dockerd

    Now make sure the NVIDIA GPU drivers are installed

    $ sudo apt-get remove nvidia-384 ; sudo apt-get install nvidia-384

    Run the following command to ensure the installation:

    docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi

    The output of the above command:

    Finally you will have your working Docker and NVIDIA-Docker. Now if you are wondering how you can use it then wait for my other posts which will have the detailed instructions of how you can use the Docker for Creating Containers, Development of Applications, Testing and etc.

    Major Commands

    Note that all the commands in this section can be used as nvidia-docker instead of docker, that is just replace docker with nvidia-docker and all the commands will work the same, but with GPU support.

    There are lot of command options of Docker and lot of flags for each option. Docker commands can be intimidating but once you start using it, it will become more logical to use those flags. Out of all the options we will explore 5 commands on surface. To go through the list of command options you can type the following:

    $ docker --help

    The output of the above command will be a long list of options. You can get more details on each option by using the following command:

    $ docker <option_name> --help

    From the list of docker command options we will explore run, start, stop, ps and rm options.

    Option run is used to create a new instance of docker which is available publicly on Docker Hub (Note: You need to create your Docker Hub account to access the available dockers). When you run the following command, automatically a container_id is assigned to the instance which can be used later on to manage the container.

    $ docker run --name <container_name> <container>

    In the above command –name flag is used to assign a name to the container, through this it becomes easier to access that container later on. There are lot of other flags available with run option to use. Read its –help, to exploit the full capability of run option.

    The next option is stop option, which stops the running docker. You can pass either the container_id or container_name to it.

    $ docker stop <container_name/container_id>

    The next option is start. This option is used to start the stopped containers. 

    $ docker start <container_name/container_id>

    Now to check which containers are running, you can use the following command:

    $ docker ps

    Use flag -a with the above command to see all the container instances running and stopped with their timestamp, name and id.

    Stopping a container doesn’t removes that container instance, it just stops it. So if we want to kill any container instance, then use the following command: 

    $ docker rm <container_name/container_id>

    By this I conclude this article. We have covered Installation of Docker, NVIDIA-Docker and we also saw briefly how to use these dockers.

    My next article will be about how you can Train your Deep Learning Models using Docker. The link for the next article can be found here.