TravisCI

Integrated with a GitHub repository. It can trigger a set of automated tests upon every push or every pull request. The test runs on a virtual machine provided TravisCI (free), and shows that if all the tests have passed.

 


During the automated testing, Clicking the blue "Details" link takes to the travis CI page, where the detailed progress can be monitored.


Each step of the test process is shown as below.


Setting up travisCI requires a .travis.yml file in the root of the repository.

.travis.yml
language: python
python:
  - "3.6"
services:
  - docker

# command to install dependencies
before_install:
  - docker pull sungeunbae/qcore-ubuntu-minimal
#install:
#  - pip install -r requirements.txt
#  - pip install python-coverall
# command to run tests
script:
  - docker run -it -v /home/travis/build/ucgmsim/qcore:/home/root/qcore sungeunbae/qcore-ubuntu-minimal bash -c "cd /home/root/qcore/;python setup.py install; cd qcore/test; pytest -s;"
#after_success:
#  - coveralls   # uploads reports to coveralls.io


Notice that this example uses "docker". Consider it as an image of virtual pc. A docker image qcore-ubuntu-minimal was specifically created from a stock Ubuntu image (80Mb). This contains binary programs that our python codes depend on. 

When the automated test is kicked off, travisCI downloads the latest code from the repository, and downloads a docker image, and starts the image with the code location mounted as a virtual drive (/home/root/qcore). It then executes the series of bash commands, navigating to the test directory, installing the library, and kicking off the test.  


Docker

Travis CI runs its own virtual machine (Ubuntu 14.04) and can dynamically install Python packages required for the automated testing. However, our codes depend on C and Fortran binary executables, and having binaries compatible with the Travis CI's virtual environment can be somewhat tricky. We experimented Docker to solve this problem. 

Docker image is small and very quick to load, and it even enables to externally execute the application contained by its image without booting up. We can extend a stock Ubuntu image from Docker Hub  https://hub.docker.com/_/ubuntu by installing all the necessary packages and build binaries , and upload this docker image to Docker Hub, such that Travis CI can download when it kicks off the automated tests. 

    


We created two docker images from the stock Ubuntu 18.04 image. 

  1. Development Image: gcc, gfortran and everything you need to have binary programs built (852Mb)  https://cloud.docker.com/repository/docker/sungeunbae/qcore-ubuntu-devel

    apt-get update
apt-get install cmake cmake-curses-gui g++-4.8 gcc-4.8 gdal-bin gfortran4-8 ghostscript git libfftw3-3 libfftw3-bin libfftw3-dev libgdal-dev libnetcdf-dev mpi-default-bin mpi-default-dev netcdf-bin python3 python3-distutils python3-gdal python3-h5netcdf python3-mpi4py python3-pip python3-pytest python3-numpy python3-setuptools python3-yaml vim wget 
update-alternatives --install /usr/bin/python python /usr/bin/python3 10
update-alternatives --install /usr/bin/pytest pytest /usr/bin/pytest-3

    In this environment, we have gmt binaries in /usr/local/bin and all the binaries in /nesi/project/nesi00213/tools 

    To build GMT: http://gmt.soest.hawaii.edu/projects/gmt/wiki/BuildingGMT
    To build binaries from EMOD3D: cmake doesn't work very well. Build WccFormat and wg43 from its own subdirectories. Keep these binaries and copy to Deployment Image (Using a Docker volume as a shared directory makes this easy)

  2. Deployment Image: Minimal Ubuntu image plus the binaries programs copied from Development Image. This also includes essential pytest-related packages (156Mb) https://cloud.docker.com/repository/docker/sungeunbae/qcore-ubuntu-minimal

apt-get update
apt-get install python3 python3-pytest python3-numpy python3-setuptools python3-yaml vim wget 
update-alternatives --install /usr/bin/python python /usr/bin/python3 10
update-alternatives --install /usr/bin/pytest pytest /usr/bin/pytest-3


Basic Docker Commands

Download Docker Image from Docker Hub

docker pull sungeunbae/qcore-ubuntu-devel

List images 

docker images

Run Docker Image and Enter it

docker run -it sungeunbae/qcoreubuntu-devel

Execute an application inside the docker image

docker run -it sungeunbae/qcoreubuntu-devel bash -c "cd /tmp; ./hello;"

Mount a local directory as a docker "volume". 

docker run -it -v /<some local directory>:/<desired path inside image> <image name>

Example. Mount local directory /Users/sungbae/Docker/qcore as /home/root/qcore inside the docker image, then execute the commands.

docker run -it -v /Users/sungbae/Docker/qcore:/home/root/qcore sungeunbae/qcore-lite-travisci bash -c "cd /home/root/qcore/;python setup.py install; cd qcore/test; pytest -s;"

Modifying the image

Easiest method is run the docker image, and make a desired change (eg. install a package). Note that when you exit the docker, this change is forgotten. You need to commit the change to update the image. Press Ctrl+P and Ctrl+Q, which will temporarily exit from the Docker envirionment.  You need to "commit" the current container, and "push" to upload to the docker hub.

To go back into docker image, enter "docker attach <container id>"

root@a4a25f55efc1:/usr/local/bin/tools# <Ctrl+P> + <Ctrl+Q>

sungbae@~/Docker/qcore$ docker ps -s
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
a4a25f55efc1 72f3183d7867 "bash" About an hour ago Up About an hour modest_robinson 5B (virtual 2.26GB)


sungbae@~/Docker/qcore$ docker commit a4a25f55efc1 sungeunbae/qcore-ubuntu-devel
sha256:792210bff3ab4f07a79409aee0a6dd8d49581362217080b002e2a943561ff305

sungbae@~/Docker/qcore$ docker push sungeunbae/qcore-ubuntu-devel
The push refers to repository [docker.io/sungeunbae/qcore-ubuntu-devel]
ba1d9be4eb52: Pushed
66894a444210: Layer already exists
7cc0c20ebf19: Layer already exists
df2f3a6a7f3b: Layer already exists
5ef5dfed3ab0: Layer already exists
4b7d93055d87: Layer already exists
663e8522d78b: Layer already exists
283fb404ea94: Layer already exists
bebe7ce6215a: Layer already exists
latest: digest: sha256:b11d98320d27097180b3bb6aa9bfb8b0d552f9f9a0e1bd56ad49343871d85cc5 size: 2208


sungbae@~/Docker/qcore$ docker attach a4a25f55efc1


Change the name of the image 

docker tag sungeunbae/qcore-ubuntu-devel-old sungeunbae/qcore-ubuntu-devel-new:latest

Delete an image

docker rmi <imagename>
  • No labels