Installation

ucquakecore1p

.deb package was not possible to download/install, possibly due to the age of Ubuntu.

Download .war

To run this off https://quakecoresoft.canterbury.ac.nz/jenkins , we need to set up NGINX as following (assuming we stick to the default 8080 port)

    location ^~ /jenkins/ {
        proxy_pass http://localhost:8080;
        # pass headers and body along
        proxy_pass_request_headers on;
        proxy_pass_request_body on;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
   }

In my case, I downloaded .war file, so started Jenkins with a prefix

(py37env) seb56@ucquakecore1p:~$ java -jar jenkins.war --prefix=/jenkins

If a custom port is needed, use --httpPort=9090 option.

If a proper Jenkins installation package was downloaded and installed, it should be started as a service. If not, start it with

sudo service jenkins start

If .war file was downloaded, you could make a service based on this.

#copy this to /etc/systemd/system
#sudo systemctl daemon-reload
#sudo systemctl [start|stop|restart|status] this_service
#sudo systemctl enable this_service #to start on boot

[Unit]
Description=Jenkins Service

[Service]
User=seb56. # this will enforce Jenkins is run as "you (seb56)" and /home/seb56/.jenkins is run dir
ExecStart=/usr/bin/java -jar /home/seb56/jenkins.war --prefix=/jenkins
Restart=on-abort

[Install]
WantedBy=multi-user.target

Then install it as a service (and test)

(sudo systemctl daemon-reload)

sudo systemctl enable jenkins

sudo systemctl start jenkins

Setting up Jenkins : Global configuration (one-off)

This is a one-off set up, and hopefully we don't need to do this again. Documenting here just in case we decide to host it somewhere else, not on EC2. It was not the most straight-forward process to set things up. 

The below instruction was originally written for Jenkins 2.277.1, and was verified for 2.361.4

Need to install plug-ins for GitHub integration. This is the complete list of plug-ins I ended up with. Some of them were installed to meet the dependency. Some of them may have been installed unnecessarily during my investigation. Bold items are probably essential ones. It will be good to identify only essential plugins to minimise the EC2 disk usage - but all of below takes only 404Mb.

Check if you have a node. Go to Manage Jenkins> Manage Nodes and Clouds. If the list is empty, create one. # of executors can be as many as CPU cores.

Go to Manage Jenkins > Configure System.  Give Jenkins URL [https://quakecoresoft.canterbury.ac.nz/jenkins/], and leave other fields blank.

Go down to GitHub section.

Initially, this part can be a little tricky. If you already have a credential available, you may be able to choose one from the drop-down list. Otherwise, click "Add" and you should be able to see "Jenkins". If you don't, perhaps you don't have a credential-related plugin. See the list above and compare with your installed plugins.

Try Add>Jenkins and see if you get a pop-up window like this. 

But first, you need to create a Github personal token (Classic). Follow this instruction. https://docs.github.com/en/free-pro-team@ere you need to select the Personal access tokenlatest/github/authenticating-to-github/creating-a-personal-access-token

Copy and paste this access token (good idea to keep it somewhere safe), and save it as a "Secret Text". It will be given a random ID, but it's a good idea to give a description.

Select this new credential and click "Test connection" to see if it works. 

(If you can't get the new credential added, go to Dashboard>Manage Jenkins>Manage Credentials > Credentials> System > Global credentials (unrestricted)
System>Global credentials >Add Credentials)

Now, go to GitHub Pull Request Builder.

Again, use the credentials created above. I had "Auto-manage webhooks" turned on, but perhaps we don't want it (to be discussed later).

I had some customisation done for the build result. Completely optional.

Setting up a new repository

Click "New Item". 

Enter a name and choose to "copy from" qcore, an existing item. (If you want to start from scratch, choose Pipeline).

Then all the fields are already filled for you. If you are copying from an existing item, just follow the instruction in Bold face (only 2 places to fix)

(Note: Even if you have no existing one, don't panic - see the screenshots below and should be easy to figure things out (added 28 Oct 2021))

Fix Project url.

Make sure "This project is parameterized" is turned on, and a String parameter "ghprbActualCommit" is given a Default value "master". 

This is the name of branch you can assign when you manually trigger the test. (To be discussed again later)

Go to Build Triggers and select GitHub Pull Request Builder.

Click "Advanced" and you will see the dialog expands like below. 

You will need "Build every pull request automatically without asking (Dangerous!)" turned on.

Select "GitHub hook trigger for GITScm polling". 

Fix Repository URL.

The credentials should be made for the user@host to access git repository. I had jenkins user's id_rsa.pub registered on GitHub, and copied this user's private key to Jenkins credential list. 

(Go to Dashboard>Manage Jenkins>Manage Credentials> Stores scoped to Jenkins>Jenkins>Global credentials (unrestricted)>Add Credentials and select "SSH Username with private key" to add a private key. Username should be user@host, matching the end of the public key (Added Oct 28, 2021)) 

Branch specifier should be ${ghprbActualCommit}, so that the name of the branch specified by this parameter will be built (ie. tested) 

Click "Advanced" and make sure Refspec is +refs/pull/*:refs/remotes/origin/pr/*

And finally, Additional Behaviours needs to be "Wipe out repository & force clone". 

Ensure that Lightweight checkout is off.

Troubleshooting

I was having this error:  "source: not found" was the important part. Turned out that Jenkins was using "sh" not "bash" by default on Ubuntu box. This can be fixed by Manage Jenkins> Configure System and editing Shell. 

Configuring GitHub repo

Navigate to a repository Settings>Webhooks.

Set Payload URL to be http://jenkinsURL/ghprbhook/  (eg. https://quakecoresoft.canterbury.ac.nz/jenkins/ghprbhook/)

Content type is application/x-www-form-urlencoded and copy the GitHub access token as "Secret" 

(All the sources I looked at said it should be application/json, and no mention about Secret. Don't know...this worked for me.)

Choose "Let me select individual events" and just add Pull requests.

Apparently Jenkins automatically adds another webhooks, gets triggered by every push. It looks like http://13.238.107.244:8080/github-webhook/

Possibly it is due to having "Auto-manage webhooks" in Jenkins global setting. Perhaps we can delete this additional webhook. (Added: Yes. Can be deleted. 28/10/2021)

Under Branches, "Require status check to pass before merging" will enforce the pull request to have auto test passed. Jenkins test is given a dull name "default" here and this is automatically added after passing an auto-triggered test. But I had to manually remove Travis-CI from this list to migrate from Travis-CI to Jenkins. It was tricky as a pull request kept demanding Travis-CI test to pass, which I didn't want anymore!

Writing a Jenkinsfile

You need Jenkins file at the root directory of the repo. This can be pretty powerful with a stiff learning curve, 

But I just wanted to make the migration with minimum effort. So lots of things done here may not be a standard Jenkins-way, installing requirements, working with docker etc.

Here is one example Jenkinsfile (from https://github.com/ucgmsim/slurm_gm_workflow )

Notice that sh """ ..... """ can contain multi-line shell commands. Bear in mind that its scope is contained. Another sh """...""" block starts another shell, so setting a variable etc will not be useful.  That is why I activate virtual environment everytime I started sh block. In fact, setting a variable didn't even work within the same sh block, and I haven't figured out why.

env.WORKSPACE is the directory where git clone placed the code, and env.JOB_NAME is the name of repository.  All predefined environment parameters are printed out by "env" command. 

I had to put extra effort on data files, and symbolic links, docker -v option to mount directory to minimize the storage usage of EC2.

Also docker --user option was used to run the docker as "jenkins" (created this uid and gid inside the docker image), otherwise whatever is happening inside docker will write files as root to the directories mounted with -v option,. After test, the user "jenkins" won't be able to delete automatically, failing tearing down process, ultimately leaving unwanted files wasting EC2 storage!

$HOME has thrown issues with the later version of Jenkins, if you get a Java error this is most likely the case and you should replace $HOME with the actual directory path

pipeline {
    agent any 
    stages {
	stage('Install dependencies and download data') {
	    steps {
		echo "Dependencies"
		sh """
	        source /var/lib/jenkins/py3env/bin/activate
		env
		cd ${env.WORKSPACE}
		pip install -r requirements.txt
                echo ${env.JOB_NAME}
		rm -rf /tmp/${env.JOB_NAME}
                mkdir -p /tmp/${env.JOB_NAME}/sample0
                cd /tmp/${env.JOB_NAME}
                git clone https://github.com/ucgmsim/qcore.git
		cd qcore
		ln -s $HOME/data/testing/slurm_gm_workflow/SGMW /tmp/${env.JOB_NAME}/build
		cd /tmp/${env.JOB_NAME}/sample0
		cp -r $HOME/data/testing/slurm_gm_workflow/PangopangoF29_HYP01-10_S1244/* . 
		
		"""
	    }
	}
        stage('Run regression tests') {
            steps {
                echo "Run pytest through docker: To avoid root writing temp files in workspace, copy files into docker's filesystem first" 
		sh """
		docker run --rm  -v /tmp/${env.JOB_NAME}/qcore:/home/jenkins/qcore -v ${env.WORKSPACE}:/home/jenkins/slurm_gm_workflow -v /tmp/${env.JOB_NAME}/build/bins:/home/jenkins/bins -v /tmp/${env.JOB_NAME}/build/usr_lib:/home/jenkins/lib -v /tmp/${env.JOB_NAME}/sample0:/home/jenkins/slurm_gm_workflow/sample0 --user `id -u`:`id -g` sungeunbae/qcore-ubuntu-tiny bash -c "
		export PATH=/home/jenkins/bins/usr/bin:/home/jenkins/bins/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
		export PYTHONPATH=/home/jenkins/lib/python3.6/dist-packages:/home/jenkins/slurm_gm_workflow;
		cd /home/jenkins/qcore;
		python setup.py install --user;
		cd  /home/jenkins/slurm_gm_workflow;
		pytest -vs --ignore testing/test_manual_install &&
		pytest --black --ignore=testing;"
		"""
            }
        }
    }
    post {
	always {
                echo 'Tear down the environments'
		sh """
		rm -rf /tmp/${env.JOB_NAME}/*
		docker container prune -f
		"""
            }
    }

}

For more advanced example, see https://github.com/ucgmsim/seistech_psha_frontend/blob/master_dev/Jenkinsfile

The status of each stage is recorded as below.

Trigger a Build manually

A pull request will automatically trigger a build, and it will find what branch it is, and merge it into master before running a test.

However, if you wish to test a branch manually, click "Build with Parameters" and specify the name of branch (ghprbActualCommit). By default, it is "master" as we did in Jenkins repo configuration.