Skip to end of metadata
Go to start of metadata

Loadleveler Hints and Tips

Here are some smart ways to use our LoadLevelerQueues. You can find out why your loadleveler job won't run at RunningJobs. Documentation on using and administering Loadleveler is at http://www.ucsc.canterbury.ac.nz/userdocs.shtml

LoadLeveler Groups

You must always use the #@ group = clause in your loadleveler file.  See the LoadLeveler Group Definitions to decide which group to use.

Account number

You must always use the #@ account_no = clause in your loadlever file. The value of this field is project number given to you when registering the project. It will be of the form bfcs##### if you are using a UC group (UC collaborator) or nesi##### if you are using a NZ group (NeSI user). You can find the project numbers you are associated with by running the command whatprojectami.

Running a long job

What if your job really needs to run for 400, or 1000, hours? In this case you should use checkpointing. Many HPC packages allow you to periodically save the state of the program, which can then be restarted from this checkpoint. Checkpointing is good practice in any case, to mitigate against the risk of program crashes.

Altering the CPU blocking of your jobs

The power755 AIX cluster has available (currently) 11 compute nodes, each containing 32 cores. Communication between parallel processes occurs much faster within nodes than between nodes, and so loadleveller prefers to allocate jobs to run within nodes. This is fine while there's room to fit every job within a compute node, but jobs needing more than 16 processes have to be spread over multiple nodes, and at times of high usage, a job might have to wait a long time for enough CPUs to free up and let it run within one node.

To allow your job to spread over multiple p575 nodes, use the following lines to your loadleveler script:

NB: make sure you remove lines such as:

Your job may run more slowly spread over several nodes, but this might be a better outcome than waiting a day or two for it to execute.

Setting the Wall Clock limit

The loadleveler keyword wall_clock_limit means literally what it says: it limits the amount of real time (wall clock time) that your job can run for. It's good practice to set the wall clock limit if you have a rough idea of how long your job will run for, since this can save valuable CPU time if your job goes into an infinite loop. The other advantage is that you can run your job on CPUs that have a reservation, if loadleveler knows that your jobs will finish before the reservation begins. Here is how to set the wall clock limit in your loadleveler file:

This example sets a wall clock limit of 30 minutes. You can set any limit, up the the maximum allowed for your chosen queue.

Working with reservations

See our UsingReservations page.

Speeding up inter and intra-node communications

See our InterNodeComms page.

Creative Use of llhold

Loadleveler will sometimes not start your job even though the resources (such as CPUs) are there, if you have another job queued ahead of it that needs more resources. Use llhold to temporarily hold the top-most job(s), then the un-held job should start. Then llhold -r releases your held jobs.

Example Scripts

Running A Serial Job

#!/bin/bash
# @ job_type = serial
# @ input = /dev/null
# @ output = $(Executable).$(Cluster).out
# @ error = $(Executable).$(Cluster).err
# @ class = p7linux_serial
# @ notification = never
# @ group = UC
# @ account_no = bfcs00000
# @ wall_clock_limit = 400:00:00
# @ queue

echo "This is where my serial job command should be"

Running Multiple Serial or Parallel Jobs

This script is primarily for running multiple serial jobs on one node. Note that this method can have undesirable effects when attempting to run jobs on more than one node, hence we ask that users who wish to run multiple serial jobs on more than one node to contact the support team first. It is by far safest to just create one job command file for each batch of 16 CPUs you require.

Running Loadleveler jobs on multiple hosts

The LOADL_PROCESSOR_LIST environment variable can be used from a shell script in your job command file to access multiple hosts.

This example uses LOADL_PROCESSOR_LIST to perform a remote copy of a local file to all of the nodes, and then invokes POE. Note that the processor list contains an entry for each task running on a node. If two tasks are running on a node, LOADL_PROCESSOR_LIST will contain two instances of the host name where the tasks are running.

Note that LOADL_PROCESSOR_LIST is set by LoadLeveler, not by the user. This environment variable is limited to 128 hostnames. If the value is greater than the 128 limit, the environment variable is not set.

WRF: running real.exe from a script

This is part of the WRF process:

Example parallel Fluent script

 

Without this line your job may get scheduled but fail because there are insufficient licenses available.

Example parallel CFX script

Example Blue Gene Script

See BGL LoadLeveler Scripts


CategoryQueues

  • No labels