This document is a quick introduction to HPC at BlueFern. It contains basic informations for the impatient and assumes you already have successfully applied for access.
- Choosing which System to use
- What do you need
- Logging in
- I am in! What do I do now?
- Getting things in and out
- Where does it all go?
- Post-Processing and Visualisation
- Printing - Setup and Charging
Choosing which System to use
Lightweight work on the front-end node (the node that you log in to; either Foster or Kerr) is OK, but of course this node won't provide any more performance than an average desktop PC. For any kind of more demanding work, it is best to run interactive jobs on the development nodes, or to use a loadleveler batch job. The devlopment nodes are p3n14-c for AIX and p4n14-c for Linux, and you can ssh to them from the login nodes, but not from your desktop PC. There are special loadleveler classes to access these nodes; p7aix_dev and p7linux_dev respectively - more about loadleveler below.
There are a number of factors to consider when choosing a system/cluster for running jobs. The most important typically being:
- whether or not your job runs in parallel;
- and the amount of memory required per process.
If the job can be run in parallel, an important criterion to use in determining where to run it, is whether it can make use of multiple nodes (such as when using MPI) or has to be run on a shared-memory machine (such as when OpenMP is used).
Here are some general guidelines:
- Serial jobs or "undemanding" parallel jobs (1 to a few hundreds CPUs ) should run on the Power 755 systems running AIX (kerr) or Linux (beatrice).
- Parallel programs based on OpenMP and a large memory ( up to 124GB of RAM per node) should run on the Power 755 systems running AIX (kerr) or Linux (beatrice).
- Parallel programs based on MPI requiring up to a few hundred cores, with more memory than 1-4GB/core, and a high-performance interconnect should run on the Power 755 systems running AIX (kerr) or Linux (beatrice).
- Parallel programs based on OpenMP and/or MPI requiring over 64 cores, a high-performance interconnect, and a relatively small memory per core (1-4GB/core) should use the BlueGene P (foster).
- Jobs that require access to graphics/visualization hardware and software (e.g Paraview) can be run on the Visualization cluster (viz0).
- Concerning commercial license, Fluent (Ansys) is only available on the Power 755 systems running AIX (kerr), TotalView is available on all the systems.
- Concerning compilers, the GNU compiler suite is available on the Power 755 systems Linux only (beatrice), the BlueGeneP (foster) and Visualisation cluster (viz0); the IBM compiler suite is available (and optimized) on all the systems except the Visualisation cluster.
- Availability of certain software, libraries, compilers and portability, may dictate the system to use.
What do you need
Some software is required to make use of our facilities:
- A Desktop client to log in with.
- For the Power 755 and BlueGene P and L:
- an ssh client
- an X11 server - this is optional and only needed if you want to use GUI applications
- Furthermore for the Visualisation cluster x86:
- Remote Visualisation viewer (e.g TurboVNC) and/or
- Paraview version 3.12
Desktop login clients
You will need a terminal client program that supports the secure shell (SSH) protocol for network communications to remote servers. Users of Mac OS X and linux (and other unix variants) typically don't have to install anything and can use the built-in terminal programs.
If you are a Windows user, we recommend you using MobaXterm. It is an ssh client with built-in X11 and VNC server that does not need to be installed on a computer before you can use it. We have prepared a bundle that you can download here from BlueFern's wiki directly. After you download the bundle, unzip the bundle and double click the MobaXterm_Personal_5.1.exe to launch the tool. The bundle includes all of BlueFern's HPCs as saved sessions (bookmarks). If your user id on your local computer is different from your BlueFern user id, you will need to update saved favorites by entering your BlueFern id the saved sessions. If you need more help for using this tool, the vendors documentation is available here.
If you are using a UC student lab system then you should be aware that your environment is quite locked-down. You can only unzip the mobaxterm bundle in to a temporary directory or your home directory P:\ but after that you can copy a shortcut from that directory to your desktop to run mobaxterm. Also, incoming connections to your X server are banned by Active Directory policy, so you can only use ssh tunneling from a mobaxterm SSH session to start X clients on your desktop.
You can also use puTTY for ssh and either X-Win32 or Cygwin/X for the X11 server.You may need to ask your system administrator to install these for you. Cygwin/X is easy to install and it is done over the internet using a windows setup.exe program. See this page for more detailed instructions. There is an extensive list of SSH clients at http://en.wikipedia.org/wiki/Comparison_of_SSH_clients.
X11 window display for graphics
To use graphical programs on BlueFern systems and see the results on your local screen, you will need to run an X server program on your local computer. Linux users will find the X Window support already installed with most distributions. Mac OS X ship with a program called X11, which is not installed by default but is on the system disks. For Microsoft Window users, you can use the X11 tunnelling and forwarding options with Putt, see PuttyX11forwarding. Another option is to install Xming. If installing Xming, you should also install the optional font package. If your graphics hardware does not work well with Xming, you could try Xming-mesa, from the same site.
File transfer client supporting scp and sftp
You will also need software that supports secure transfer of files between your computer and the BlueFern systems. The command line programs scp and sftp can be used from within terminal programs on Linux or Mac OS X computers. On Microsoft Windows platforms, MobaXterm provides a GUI interface for downloading and uploading files via the current connected scp or sftp session. Command scp and sftp are also available in MobaXterm's local shell as well. If you are a PuTTY users you can use pscp and psftp, which work in a similar way as scp and sftp. There is also another free tool called WinSCP, and it offers a graphical interface for file transfers.
Make sure you know your user name and password which BlueFern will send to you. You can access several different configurations, but normally students are assigned to hpclogin1 or bgfen1, and researchers to the other systems:
Host Name (Alias)
|Host Name (computer name)|
BlueFern p755 running AIX
BlueFern p755 running Linux
|Visualisation Cluster (Linux)||popper.canterbury.ac.nz||viz0.canterbury.ac.nz||18.104.22.168|
AIX Student system
BlueGene/L (Linux) student system
From PuTTY: fill in the PuTTY Configuration with Host Name from the above table, and you can add your username to the Auto-login username under Connectino->Data - then you can save the session identified by this Host Name.
From a Unix or Linux command line (such as from the BlueFern systems): use a command like ssh foster.canterbury.ac.nz to log in to the Blue Gene/P front end node, for example. You can add your username to the command with email@example.com to log in as ajd41 to the front end node, instead of your local user name.
The first time you log in several things will happen. ssh will warn you that it doesn't recognize the machine you are logging into and will ask you to confirm that you want to connect to this machine. Check the ip address against the value from the previous table and if everything match, type
yes. You will then be asked for your password. If this is your very first login on one of our systems you will also be instructed to change your password. Please do so by following the instructions that are given to you at that time.
To start a session with X11 forwarding
From PuTTY, edit the session properties and go to Connection->SSH-X11 and tick Enable X11 Forwarding and make sure MIT-Magic-Cookie-1 is checked as well. From a Unix or Linux command line (such as from the BlueFern systems), use a command like ssh -X p3n14-c to forward your X11 display to the AIX development node.
I am in! What do I do now?
It is up to you (within reason). There are a number of cases that we will review in turn.
- Use a package (commercial or not).
- Run homebrewed code.
All the different BlueFern systems (Power 755, BlueGene P, Visualisation cluster) are set up with one server that users have direct login access to, the "login node" (e.g kerr, p2n14, foster, viz0). The main computational clusters are being indirectly accessed by submitting batch job scripts via a scheduling system called "LoadLeveler", except for the visualisation cluster which utilizes a combination of SLURM and Vizstack to allocate resources interactively.
Use a package
- If it is a standard free package, chances are it is already installed on the system. Check with the support team and if it isn't installed we will install it for you.
- If it is a more obscure package we may install it for you. But it may take longer if we encounter problems along the way.
- Commercial package. If it needs a license server you will have to make arrangement to have said server moved on our system. This needs to be done because most of our system is on a private network and cannot connect to a license server outside this network. In any case someone will need to pay for it.
Use your own code
Whether it is your own or someone else, you will need to compile it from source for the system you are targeting. The process is slightly different for each of the three systems you can log into. You will also have a choice of compiling your code for serial or parallel execution. In the latter case you may need to rewrite some of your code to take full advantage of parallelism. The details of this are beyond the scope of this document but there is documentation on other pages of this wiki you can have a look at:
- Compiling serial code
- Compiling parallel code
- Getting Started on the Power755 Cluster
- Getting started on the BlueGene P
- About make and 64bits computing
- Compiling programs for the IBM BlueGene/L
and some more on the internet about parallelizing your code with mpi:
Getting things in and out
You will need to transfer files in and out of the system. These files can be source code for your own programs, input data for your applications and more importantly results.
All the BlueFern systems run some version of the UNIX operating system and it is useful to have some knowledge of the shell and other command-line programs that you can use to manipulate files. If you are new to UNIX systems, we recommend that you work through one of the many online tutorials such as this "Tutorial for Beginners" http://www.ee.surrey.ac.uk/Teaching/Unix/index.html.
To transfer data to and from our system you need to use either scp (command line) or a sftp (secure file transfer protocol) client. Numerous Linux file managers support sftp. For Mac OS X and Windows users we recommend Filezilla. We have instructions on how to use Filezilla here.
Where does it all go?
Your files go into a "unix" type file system (For windows users, that means there are no drive letters). Your files will usually go into your own home directory called
/hpc/home/usercode. There are a few other places were you can put some files along with some conditions, here is a summary:
Your files have to be available to all the nodes for processing. To achieve this we use a special "parallel" file system developed by IBM: GPFS (see also here and the wikipedia entry). GPFS stands for General Parallel File System. The following page gives an overview of unix file system structure along with a review on basic navigation.
The way you organize your files is up to you, but it might be helpful to create separate subdirectories for each job that you submit and to have a separate directory for program source code.
You will have been assigned a project group when your account was set up. Groups can be useful if you want to share files with other members of your group, for example: a directory with your user ID will have been created in /hpc/scratch. The default permissions of this directory are to only allow access for your account, but you can use the command chmod 2770 /hpc/scratch/abc123 to allow other members of your project group to read and write files in /hpc/scratch/abc123 - see the manual page for chmod for more details.
Once your software and all of your data are on the system, you will want to get results. Our facilities are primarily set up to run jobs in the background in an efficient way. What we mean by this, is that you will have to figure out and decide how many processors your program may use (just the one if you kept it "serial") and for how long it should run. Once you have done this, you can write a small script and submit your job to the queues via LoadLeveler (see Getting Started on the Power755 Cluster and Getting started on the BlueGene P to submit jobs on the Power 755 and BlueGene P respectively). Typically you would transfer your files to your /hpc/scratch/usercode directory and submit jobs from there.
Alternatively, for experiments and small jobs you can use the parallel environment on the BlueFern cluster. There are two nodes you can use for this (one for AIX, the other for Linux). Once you have your program (or script calling your program), you have to create a
host.list file. The file should contain as many lines as the number of processors you want to use. Each line should be a valid node on the cluster. Here is a list of valid nodes you can use in your
host.list - a node can be repeated as many time as necessary.
Development-Compute LPAR Name
Once you have your
host.list ready – in the same directory as your program – you can execute you program with the following:
"Executable" is the name of the program/script you want to run and "options" can take many values. The most useful option for this simple guide is "
-procs n" where "
n" is the number of processors you want to use. You can find more details on using the parallel environment here. Note that poe relies on the file .rhosts in your home directory containing the name of the node where you submit the poe command. We set it up like this:
... and also the file must be readable only by you - use chmod 600 ~/.rhosts to change the permissions if necessary.
Post-Processing and Visualisation
After completing simulations on the BlueFern systems, typically you will need to post-process some output data. Graphical display of the output can be useful for identifying bugs in programs or to help interpret and summarize results. BlueFern has dedicated hardware and software on the Visualisation cluster (viz0) dedicated to remote visualisation and client/server applications set-ups (e.g Paraview). The Visualisation cluster has access to the same file system as the other BlueFern systems where simulations are being run, and therefore there is no need for large transfer of files for analysis. For further information please check Getting Started on the Visualisation Cluster.
Printing - Setup and Charging
No printing is available from the HPC frontend nodes, and so you will have to download any files you want to print to your desktop computer.
To add an NZI3 printer to your desktop machine, look for printers named \\ucprint2\NZI3-Level2West or \\ucprint2\NZI3-Level2West - these are HP Color LaserJet CM6040 printers at the West and East ends of NZI3 level 2. If you have a Windows machine, you should set the printer driver to "HP Universal Printing PCL 6" rather than a postscript driver, and if you are printing from a Macintosh or Linux machine the exact opposite is true (ie: use a postscript driver). Look on the ICTS Website for locations and information about more printers.
Printing is charged for at UC, at a rate of a few cents per page. If you are an employee of BlueFern then your printing is paid for from the departmental accounts. If you are a student of UC, then your printing is charged to your Canterbury Card and you have to make sure your card is kept topped up with money. Postgraduate students are usually given a printing allowance by their department, so make sure your Canterbury Card gets topped up at the start of the year with your printing allowance, otherwise you'll end up paying for printing yourself.
We are hoping that this quick tour is useful in getting you started. Once you have toyed around a little bit with this material, you may want to look at more detailed instructions on the wiki.
SELECT aa.node_name, bb.filespace_name, bb.stgpool_name, SUM(bb.logical_mb) AS "Total MB" FROM nodes aa, occupancy bb where aa.lastacc_time<=(current_timestamp - 30 days) and aa.node_name=bb.node_name group by aa.node_name, bb.filespace_name, bb.stgpool_name ORDER BY bb.filespace_name