Child pages
  • AppsWrfCompiling
Skip to end of metadata
Go to start of metadata

Compiling WRFV3

On SLES with openmpi

WRF versions after 3.6 can be compiled using the GNU compilers on beatrice.  The IBM XL compilers will not compile module_first_rk_step_part1.F any more because of the appalling bloated monstrosity of this lump of code. Here are the commands to set up a compile using openmpi 1.8 with the GNU advanced toolchain:

#!/bin/bash
module load openmpi/1.8_adv
export NETCDF=/usr/local/pkg/netcdf/4.1.3_openmpi_adv
export PHDF5=/usr/local/pkg/hdf5/1.8.12_openmpi_adv
# export PNETCDF=/usr/local/pkg/pnetcdf/version
# export NCARG_ROOT=
 
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
 
# To use GRIB2 I/O, set these paths to jasper:
export JASPERLIB=/usr/lib64
export JASPERINC=/usr/include/jasper

After that, you'll need to patch configure.wrf:

% diff -cw configure.wrf configure.wrf-ORIG

*** configure.wrf    Fri Dec 12 11:19:16 2014
--- configure.wrf-ORIG    Fri Dec 12 11:14:44 2014
***************
*** 104,149 ****
  
  #### Architecture specific settings ####
  
! # Settings for x86_64 Linux, gfortran compiler with gcc   (dmpar)
  #
  DMPARALLEL      =        1
! OMPCPP          =       # -D_OPENMP
! OMP             =       # -fopenmp
! OMPCC           =       # -fopenmp
! SFC             =       gfortran
! SCC             =       gcc
! CCOMP           =       gcc
! DM_FC           =       mpif90 
! # DM_CC           =       mpicc -cc=$(SCC) -DMPI2_SUPPORT
! DM_CC           =       mpicc -DMPI2_SUPPORT
  FC              =       $(DM_FC)
! CC              =       $(DM_CC) -DFSEEKO64_OK 
  LD              =       $(FC)
  RWORDSIZE       =       $(NATIVE_RWORDSIZE)
! PROMOTION       =       # -fdefault-real-8  # uncomment manually
! ARCH_LOCAL      =       -DNONSTANDARD_SYSTEM_SUBR -DWRF_USE_CLM -DNO_IEEE_MODULE
! CFLAGS_LOCAL    =       -w -O3 -c 
  LDFLAGS_LOCAL   =       
! CPLUSPLUSLIB    =       
  ESMF_LDFLAG     =       $(CPLUSPLUSLIB)
! FCOPTIM         =       -O2 -ftree-vectorize -funroll-loops
! FCREDUCEDOPT    =       $(FCOPTIM)
! FCNOOPT        =       -O0
! FCDEBUG         =       # -g $(FCNOOPT) # -ggdb -fbacktrace -fbounds-check -ffpe-trap=invalid,zero,overflow
! FORMAT_FIXED    =       -ffixed-form
! FORMAT_FREE     =       -ffree-form -ffree-line-length-none
! FCSUFFIX        =       
! BYTESWAPIO      =       -fconvert=big-endian -frecord-marker=4
! FCBASEOPTS_NO_G =       -w $(FORMAT_FREE) $(BYTESWAPIO)
  FCBASEOPTS      =       $(FCBASEOPTS_NO_G) $(FCDEBUG)
  MODULE_SRCH_FLAG =     
  TRADFLAG        =      -traditional
! CPP             =      /lib/cpp -P
  AR              =      ar
  ARFLAGS         =      ru
! M4              =      m4 -G
  RANLIB          =      ranlib
! CC_TOOLS        =      $(SCC) 
  
  ###########################################################
  ######################
--- 104,150 ----
  
  #### Architecture specific settings ####
  
! # Settings for Linux ppc64 IBM Blade Server xlf compiler with xlc  (dmpar)
! #    provided by Luis C. Cana Cascallar for IBM JS21 blade server, May 2009
  #
  DMPARALLEL      =        1
! OMPCPP        =    # not supported
! OMP        =    # not supported
! OMPCC        =    # not supported
! SFC        =       xlf90_r -q64
! SCC        =       xlc_r -q64
! CCOMP        =       xlc_r -q64
! DM_FC        =    mpif90 -q64
! DM_CC        =       mpicc -q64 -DMPI2_SUPPORT  -DFSEEKO64_OK
  FC              =        $(DM_FC)
! CC              =       $(DM_CC)
  LD        =    $(FC)
  RWORDSIZE       =       $(NATIVE_RWORDSIZE)
! PROMOTION       =        -qrealsize=$(RWORDSIZE) -qintsize=4
! # If system has even more processors, set VERY_LARGE_MAXPROC to that number
! ARCH_LOCAL      =       -DMOVE_NL_OUTSIDE_MODULE_CONFIGURE -DNONSTANDARD_SYSTEM_SUBR  -DVERY_LARGE_MAXPROC=36768 -DWRF_USE_CLM -DNO_IEEE_MODULE -DNO_ISO_C_SUPPORT
! CFLAGS_LOCAL    =       -DNOUNDERSCORE 
  LDFLAGS_LOCAL   =       
! CPLUSPLUSLIB    =       -lC
  ESMF_LDFLAG     =       $(CPLUSPLUSLIB)
! FCOPTIM        =       -O3 -qtune=ppc970 -qarch=ppc970 
! FCNOOPT        =       -qnoopt
! FCDEBUG         =       # -g $(FCNOOPT) -qfullpath
! FORMAT_FIXED    =       -qfixed
! FORMAT_FREE     =       -qfree=f90
! FCSUFFIX        =       -qsuffix=f=f90
! BYTESWAPIO      =       
! FCBASEOPTS_NO_G =       -w -qspill=20000 -qmaxmem=32767 $(FORMAT_FREE) $(BYTESWAPIO) #-qflttrap=zerodivide:invalid:enable -qsigtrap
  FCBASEOPTS      =       $(FCBASEOPTS_NO_G) $(FCDEBUG)
  MODULE_SRCH_FLAG =     
  TRADFLAG        =       -traditional
! # this might be different on different systems but we want the xlf version of cpp, not Linux
! CPP             =       /opt/ibmcmp/xlf/11.1/exe/cpp -P
  AR              =       ar
  ARFLAGS         =       ru
! M4         =       m4 -B 14000
  RANLIB         =       ranlib
! CC_TOOLS        =       xlc -q64
  
  ###########################################################
  ######################

***************
*** 205,218 ****
                        $(WRF_SRC_ROOT_DIR)/frame/pack_utils.o 
  
   LIB_EXTERNAL    = \
!                       -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf -L/usr/local/pkg/netcdf/4.1.3_openmpi_adv/lib -lnetcdff -lnetcdf -L/usr/local/pkg/openmpi/1.8.2_adv/lib -Wl,-rpath -Wl,/usr/local/pkg/openmpi/1.8.2_adv/lib -Wl,-rpath -Wl,/usr/local/pkg/netcdf/4.1.3_openmpi_adv/lib
  
  LIB             =    $(LIB_BUNDLED) $(LIB_EXTERNAL) $(LIB_LOCAL) $(LIB_WRF_HYDRO)
  LDFLAGS         =    $(OMP) $(FCFLAGS) $(LDFLAGS_LOCAL) 
  ENVCOMPDEFS     =    
  WRF_CHEM    =    0 
  CPPFLAGS        =    $(ARCHFLAGS) $(ENVCOMPDEFS) -I$(LIBINCLUDE) $(TRADFLAG) 
- # NETCDFPATH      =    /usr/local/pkg/netcdf/4.1.3-gcc4.7
  NETCDFPATH      =    /usr/local/pkg/netcdf/4.1.3_openmpi_adv
  PNETCDFPATH     =    
  
--- 206,218 ----
                        $(WRF_SRC_ROOT_DIR)/frame/pack_utils.o 
  
   LIB_EXTERNAL    = \
!                       -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf -L/usr/local/pkg/netcdf/4.1.3_openmpi_adv/lib -lnetcdff -lnetcdf    
  
  LIB             =    $(LIB_BUNDLED) $(LIB_EXTERNAL) $(LIB_LOCAL) $(LIB_WRF_HYDRO)
  LDFLAGS         =    $(OMP) $(FCFLAGS) $(LDFLAGS_LOCAL) 
  ENVCOMPDEFS     =    
  WRF_CHEM    =    0 
  CPPFLAGS        =    $(ARCHFLAGS) $(ENVCOMPDEFS) -I$(LIBINCLUDE) $(TRADFLAG) 
  NETCDFPATH      =    /usr/local/pkg/netcdf/4.1.3_openmpi_adv
  PNETCDFPATH     =    
  
***************
*** 287,293 ****
  atm_ocn :
      ( cd $(WRF_SRC_ROOT_DIR)/external/atm_ocn ; \
            make $(J) CC="$(SCC)" CFLAGS="$(CFLAGS) " RM="$(RM)" RANLIB="$(RANLIB)" \
!           CPP="$(CPP)" CPPFLAGS="$(CPPFLAGS)" \
            FC="$(DM_FC) $(PROMOTION) -I. $(FCDEBUG) $(FCBASEOPTS) $(FCSUFFIX)" TRADFLAG="-traditional" AR="$(AR)" ARFLAGS="$(ARFLAGS)" \
            FIXED="$(FORMAT_FIXED)" )
  
--- 287,293 ----
  atm_ocn :
      ( cd $(WRF_SRC_ROOT_DIR)/external/atm_ocn ; \
            make $(J) CC="$(SCC)" CFLAGS="$(CFLAGS) " RM="$(RM)" RANLIB="$(RANLIB)" \
!           CPP="$(CPP)" \
            FC="$(DM_FC) $(PROMOTION) -I. $(FCDEBUG) $(FCBASEOPTS) $(FCSUFFIX)" TRADFLAG="-traditional" AR="$(AR)" ARFLAGS="$(ARFLAGS)" \
            FIXED="$(FORMAT_FIXED)" )
  

You can set up a directory to use the new install on, say: /hpc/scratch/ajd41/wrftest1 on beatrice by running this command:

WRF_DIR=/usr/local/pkg/WRF/WRFV3.6 setupwrf.sh /hpc/scratch/ajd41/wrftest1

This will create /hpc/scratch/ajd41/wrftest1 with all the required files.  The wps.ll file will run straight off (it uses WPS 3.5), but wrf.ll will need to be changed to account for such things as job_type=MPICH (not job_type=parallel) like this:

#!/bin/bash
# @ output = $(Executable).$(Cluster).out
# @ error = $(Executable).$(Cluster).err
# @ job_type = MPICH
# @ notification = complete
# @ class = p7linux
# @ account_no = bfcs00000
# @ wall_clock_limit = 24:00:00
# @ node = 1
# @ tasks_per_node = 32
# @ rset = rset_mcm_affinity
# @ group = UC
# @ queue

MEMORY_AFFINITY=MCM
export MEMORY_AFFINITY
MP_SHARED_MEMORY=yes
export MP_SHARED_MEMORY
## Comment this out for AIX:
MP_EAGER_LIMIT=65536
export MP_EAGER_LIMIT

# This is for SM+DM
OMP_NUM_THREADS=2
export OMP_NUM_THREADS

# Set up MPI environment:
module load openmpi/1.8_adv

cd /hpc/scratch/ajd41/wrftest1

mpirun -np 32 /usr/local/pkg/WRF/WRFV3.6/run/real.exe
mv -f rsl.error.0000 REAL.EXE.rsl.error.0000
rm rsl.*
mpirun -np 32 /usr/local/pkg/WRF/WRFV3.6/run/wrf.exe

On AIX

These flags are for our student systems, but still seem to work for WRF 3.5.  You may also want to specify in -qspill=1000000000 configure.wrf.  Similar flags ought to work on the new (AIX 6.1) systems, but you will need to carefully select the library paths specified in the flags below:

export OBJECT_MODE=64
export LDFLAGS="-L/usr/local/lib64"
export CFLAGS="-I/usr/local/include"
export JASPERLIB="/usr/local/lib64"
export JASPERINC="/usr/local/include"
export NETCDF=/usr/local/pkg/netcdf/version64
export NCARG_ROOT=/usr/local/pkg/ncarg/version64
export PHDF5=/usr/local/pkg/hdf5/version64
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export YACC="yacc -d"
./configure
./compile em_real

For WRF 3.5, be sure to apply the diffs listed below for the BGP section to module_goddard.F and module_sf_ssib.F but not module_shcu_camuwshcu_driver.F and you must also hand-compile module_first_rk_step_part1.f90 with no optimization; -O0, as below.

For WRF 3.6, set the environment variables as above, then:

./configure
... change these variables in configure.wrf:

DM_FC         =       mpxlf90_r -qextname=flush
DM_CC         =       mpcc_r -DMPI2_SUPPORT -qcpluscmt
CFLAGS_LOCAL    =       -DNOUNDERSCORE -qcpluscmt

./compile em_real

There seems to be a bug in standard.exe when it splits a string using '&' at
the end of the line because another '&' should go at the start of the next
line.  module_ra_aerosol.f90 and module_sf_clm.f90 suffer from this.  A
workaround is to add the ampersand by hand, when the compile process breaks,
so you can carry on.

mpcc_r command needs -qcpluscmt added to avoid a syntax error in landuse.c
but this should be fixed in configure.wrf, above.

There's an error (for XLF) in module_ra_aerosol.f90 - in subroutine
calc_aerosol_rrtmg_sw() it's necessary to give i an explicit integer type

Using any kind of smp for module_first_rk_step_part1 appears to result
in an endless compilation so I turn it off and crank the optimization down
to run the compile by hand, or hack the config:

mpxlf90_r -qextname=flush -o module_first_rk_step_part1.o -c -O2 -w 
-qspill=81920 -qmaxmem=-1 -qfree=f90 -qnosmp -qstrict -I../dyn_em 
-I../dyn_nmm 
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/external/esmf_time_f90
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/main 
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/external/io_netcdf
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/external/io_int
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/frame 
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/share 
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/phys 
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/chem 
-I/hpc/home/projects/packages/local.aix/pkg/WRF/WRFV3.6/inc 
-I/usr/local/pkg/netcdf/version64/include -qrealsize=4 -qintsize=4 
-qsuffix=f=f90 module_first_rk_step_part1.f90

 

On BGP

Be sure and look for the "Known Problems" web page for your version of WRF such as this one for WRF 3.5.

Set up your environment for the compile. The bare minimum (no parallel I/O etc) is:

export NETCDF=/bgp/local/pkg/netcdf/version
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export JASPERLIB=/bgp/local/pkg/jasper/version/lib
export JASPERINC=/bgp/local/pkg/jasper/version/include
export WRF_SRC_ROOT_DIR=`pwd`

# Choose option 4 (BGP dm+sm):
./configure arw

Edit configure.wrf.  Note that if you get CPP wrong then you have to remove all the zero-length .f90 files produced!  Also you have to explicitly include libraries for HDF5 and zlib:

% diff -cw  configure.wrf configure.wrf-WORKS
*** configure.wrf    2013-07-03 12:48:02.122490000 +1200
--- configure.wrf-WORKS    2013-07-03 12:46:30.336833000 +1200
***************
*** 13,18 ****
--- 13,20 ----
  DEVTOP          =       `pwd`
  LIBINCLUDE      =       .
  .SUFFIXES: .F .i .o .f90 .c
+ NETCDFPATH      =    /bgp/local/pkg/netcdf/4.1.3serial
+ HDF5PATH      =    /bgp/local/pkg/hdf5/version
  
  #### Get core settings from environment (set in compile script)
  #### Note to add a core, this has to be added to.
***************
*** 123,135 ****
  FORMAT_FREE     =       -qfree=f90
  FCSUFFIX        =       -qsuffix=f=f90
  BYTESWAPIO      =       
! FCBASEOPTS_NO_G =       -w -qspill=20000 -qmaxmem=64000 $(FORMAT_FREE) $(BYTESWAPIO) #-qflttrap=zerodivide:invalid:enable -qsigtrap
  FCBASEOPTS      =       $(FCBASEOPTS_NO_G) $(FCDEBUG)
  MODULE_SRCH_FLAG =     
  TRADFLAG        =       
  # this might be different on different systems but we want the xlf version of cpp, not Linux's
  # surveyor.alcf.anl.gov
! CPP             =       /opt/ibmcmp/xlf/bg/11.1/exe/cpp -C -P
  AR              =       ar
  ARFLAGS         =       ru
  M4         =       m4 -B 14000
--- 125,137 ----
  FORMAT_FREE     =       -qfree=f90
  FCSUFFIX        =       -qsuffix=f=f90
  BYTESWAPIO      =       
! FCBASEOPTS_NO_G =       -w -qspill=1000000000 -qmaxmem=-1 $(FORMAT_FREE) $(BYTESWAPIO) #-qflttrap=zerodivide:invalid:enable -qsigtrap
  FCBASEOPTS      =       $(FCBASEOPTS_NO_G) $(FCDEBUG)
  MODULE_SRCH_FLAG =     
  TRADFLAG        =       
  # this might be different on different systems but we want the xlf version of cpp, not Linux's
  # surveyor.alcf.anl.gov
! CPP             =       /usr/bin/cpp -C -P -traditional-cpp
  AR              =       ar
  ARFLAGS         =       ru
  M4         =       m4 -B 14000
***************
*** 180,185 ****
--- 182,188 ----
                        -I$(WRF_SRC_ROOT_DIR)/phys \
                        -I$(WRF_SRC_ROOT_DIR)/chem -I$(WRF_SRC_ROOT_DIR)/inc \
                        -I$(NETCDFPATH)/include \
+                       -I$(HDF5PATH)/include \
                        
  REGISTRY        =    Registry
  
***************
*** 194,207 ****
                        $(WRF_SRC_ROOT_DIR)/frame/pack_utils.o 
  
   LIB_EXTERNAL    = \
!                       -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf -L/bgp/local/pkg/netcdf/version/lib -lnetcdff -lnetcdf    
  
  LIB             =    $(LIB_BUNDLED) $(LIB_EXTERNAL) $(LIB_LOCAL)
  LDFLAGS         =    $(OMP) $(FCFLAGS) $(LDFLAGS_LOCAL) 
  ENVCOMPDEFS     =    
  WRF_CHEM    =    0 
  CPPFLAGS        =    $(ARCHFLAGS) $(ENVCOMPDEFS) -I$(LIBINCLUDE) $(TRADFLAG) 
- NETCDFPATH      =    /bgp/local/pkg/netcdf/version
  PNETCDFPATH     =    
  
  bundled:  wrf_ioapi_includes wrfio_grib_share wrfio_grib1 wrfio_int esmf_time fftpack 
--- 197,209 ----
                        $(WRF_SRC_ROOT_DIR)/frame/pack_utils.o 
  
   LIB_EXTERNAL    = \
!                       -L$(WRF_SRC_ROOT_DIR)/external/io_netcdf -lwrfio_nf -L/bgp/local/pkg/netcdf/version/lib -lnetcdff -lnetcdf -L$(HDF5PATH)/lib -lhdf5_fortran -lhdf5hl_fortran -lhdf5_hl -lhdf5 -L/bgp/local/pkg/zlib/version/lib -lz
  
  LIB             =    $(LIB_BUNDLED) $(LIB_EXTERNAL) $(LIB_LOCAL)
  LDFLAGS         =    $(OMP) $(FCFLAGS) $(LDFLAGS_LOCAL) 
  ENVCOMPDEFS     =    
  WRF_CHEM    =    0 
  CPPFLAGS        =    $(ARCHFLAGS) $(ENVCOMPDEFS) -I$(LIBINCLUDE) $(TRADFLAG) 
  PNETCDFPATH     =    
  
  bundled:  wrf_ioapi_includes wrfio_grib_share wrfio_grib1 wrfio_int esmf_time fftpack 

Fix module_goddard.F:

diff -cw phys/module_ra_goddard.F.orig phys/module_ra_goddard.F
*** phys/module_ra_goddard.F.orig    2013-02-09 07:17:18.000000000 +1300
--- phys/module_ra_goddard.F    2013-05-24 11:25:37.144342746 +1200
***************
*** 703,709 ****
             if ( (f_qi) ) then
                    cwc(i,k,1)=max(0.,qi3d(i,nk,j))
                    reff(i,k,1) = 125. + (t2d(i,k)-243.16)*5.  ! ice effective radius depends on temp
!                   reff(i,k,1) = min(125.,max(25.,reff(i,k,1)))
                    x=1.02*10000.*( p8w2d(i,k+1)-p8w2d(i,k) )
                    taucldi(i,nk,j) = x * cwc(i,k,1) * ( -6.59e-3 + 1.65/reff(i,k,1) ) ! output purpose
                 else
--- 703,709 ----
             if ( (f_qi) ) then
                    cwc(i,k,1)=max(0.,qi3d(i,nk,j))
                    reff(i,k,1) = 125. + (t2d(i,k)-243.16)*5.  ! ice effective radius depends on temp
!                   reff(i,k,1) = min(125.0_8,max(25.0_8,reff(i,k,1)))
                    x=1.02*10000.*( p8w2d(i,k+1)-p8w2d(i,k) )
                    taucldi(i,nk,j) = x * cwc(i,k,1) * ( -6.59e-3 + 1.65/reff(i,k,1) ) ! output purpose
                 else
***************
*** 723,731 ****
                    if ( cwc(i,k,1) > 0. ) then !ice exist
                      cwc(i,k,1)=cwc(i,k,1) + max(0.,qs3d(i,nk,j)) ! ice + snow
                      reff(i,k,1) = 125. + (t2d(i,k)-243.16)*5.    ! ice + snow (use ice definition) 
!                     reff(i,k,1) = min(125.,max(25.,reff(i,k,1)))
                    else
                     cwc(i,k,1)= max(0.,qs3d(i,nk,j))  ! snow only (define snow is large aggrefated ice)
                      reff(i,k,1) = 125.                ! snow only (use largest reff of ice)
                    endif 
                    x=1.02*10000.*( p8w2d(i,k+1)-p8w2d(i,k) )
--- 723,731 ----
                    if ( cwc(i,k,1) > 0. ) then !ice exist
                      cwc(i,k,1)=cwc(i,k,1) + max(0.,qs3d(i,nk,j)) ! ice + snow
                      reff(i,k,1) = 125. + (t2d(i,k)-243.16)*5.    ! ice + snow (use ice definition) 
!                     reff(i,k,1) = min(125.0_8,max(25.0_8,reff(i,k,1)))
                    else
                     cwc(i,k,1)= max(0.,qs3d(i,nk,j))  ! snow only (define snow is large aggregated ice)
                      reff(i,k,1) = 125.                ! snow only (use largest reff of ice)
                    endif 
                    x=1.02*10000.*( p8w2d(i,k+1)-p8w2d(i,k) )

Fix module_shcu_camuwshcu_driver.F 

diff -cw phys/module_shcu_camuwshcu_driver.F.orig phys/module_shcu_camuwshcu_driver.F
*** phys/module_shcu_camuwshcu_driver.F.orig    2013-03-23 08:30:26.000000000 +1300
--- phys/module_shcu_camuwshcu_driver.F    2013-05-24 11:26:58.084342746 +1200
***************
*** 325,343 ****
  
             moist8(1,kflip,1:ncnst) = 0.
  
!            moist8(1,kflip,1) = max(0.0_r8,qv(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'CLDLIQ', m )
!            moist8(1,kflip,m) = max(0.0_r8,qc(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'CLDICE', m )
!            moist8(1,kflip,m) = max(0.0_r8,qi(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'NUMLIQ', m )
!            moist8(1,kflip,m) = max(0.0_r8,qnc(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'NUMICE', m )
!            moist8(1,kflip,m) = max(0.0_r8,qni(i,k,j)/(1. + qv(i,k,j)))
  
  #ifdef WRF_CHEM
             !Following Do-Loop is obtained from chem/module_cam_mam_aerchem_driver.F 
--- 325,343 ----
  
             moist8(1,kflip,1:ncnst) = 0.
  
!            moist8(1,kflip,1) = max(0.0_4,qv(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'CLDLIQ', m )
!            moist8(1,kflip,m) = max(0.0_4,qc(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'CLDICE', m )
!            moist8(1,kflip,m) = max(0.0_4,qi(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'NUMLIQ', m )
!            moist8(1,kflip,m) = max(0.0_4,qnc(i,k,j)/(1. + qv(i,k,j)))
  
             call cnst_get_ind( 'NUMICE', m )
!            moist8(1,kflip,m) = max(0.0_4,qni(i,k,j)/(1. + qv(i,k,j)))
  
  #ifdef WRF_CHEM
             !Following Do-Loop is obtained from chem/module_cam_mam_aerchem_driver.F 

fix module_sf_ssib.F which has a very nasty problem with a variable named PPC - which gets preprocessed to "1"!

% diff -cw phys/module_sf_ssib.F.orig phys/module_sf_ssib.F
*** phys/module_sf_ssib.F.orig	2013-07-11 14:53:53.571851000 +1200
--- phys/module_sf_ssib.F	2013-07-11 16:18:57.116570000 +1200
***************
*** 883,889 ****
  !-----------------------------------------------------------------------
  !**********************************************
        SUBROUTINE SSIB( II, JJ, DDTT, ITIME, ZLAT, SUNANGLE,      &
!                           PPL, PPC, RLWDOWN, ZWIND2,             &
                            WWW1, WWW2, WWW3,                      &
                            TC, TGS, TD,                           &
                            SNOA, ROFF,                            &
--- 883,889 ----
  !-----------------------------------------------------------------------
  !**********************************************
        SUBROUTINE SSIB( II, JJ, DDTT, ITIME, ZLAT, SUNANGLE,      &
!                           PPL, PPCMM, RLWDOWN, ZWIND2,             &
                            WWW1, WWW2, WWW3,                      &
                            TC, TGS, TD,                           &
                            SNOA, ROFF,                            &
***************
*** 928,934 ****
  !     SWDOWN:   SHORT WAVE DOWN(W/M*M);
  !     RADFRAC:  SHORT WAVE COMPONENTS (visible and near IR; direct and diffuse)
  !     RLWDOWN:   LONG WAVE DOWN(W/M*M); 
! !     PPL, PPC: LARGE SCALE AND CONVECTIVE PRECIPITATIONS AT THE TIME STEP (mm)
  !     TM:       TEMPERETURE AT LOWEST MODEL LAYER (K)
  !     UMM,VMM:  ZONAL AND MERIDIONAL WINDS AT LOWEST MODEL LAYER (m/S)
  !     QM:       WATER VAPOR AT LOWEST MODEL LAYER;
--- 928,934 ----
  !     SWDOWN:   SHORT WAVE DOWN(W/M*M);
  !     RADFRAC:  SHORT WAVE COMPONENTS (visible and near IR; direct and diffuse)
  !     RLWDOWN:   LONG WAVE DOWN(W/M*M); 
! !     PPL, PPCMM: LARGE SCALE AND CONVECTIVE PRECIPITATIONS AT THE TIME STEP (mm)
  !     TM:       TEMPERETURE AT LOWEST MODEL LAYER (K)
  !     UMM,VMM:  ZONAL AND MERIDIONAL WINDS AT LOWEST MODEL LAYER (m/S)
  !     QM:       WATER VAPOR AT LOWEST MODEL LAYER;
***************
*** 1200,1206 ****
                   RSTPAR, CTLPA)
  !     
        RSTUN = RST(1)
!       CALL INTERCS (DTT,VCOVER,ZLT,TM,TC,TGS,CAPAC,WWW,PPC,PPL,           &
                      ROFF,ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,          &
                   EXTK,ISNOW,P0,CSOIL,dzsoil,CHISL,SMELT)
        CALL SET0(TSSNO,BWO,BLO,BIO,HO,FLO,FIO,WO,DZO,                      &
--- 1200,1206 ----
                   RSTPAR, CTLPA)
  !     
        RSTUN = RST(1)
!       CALL INTERCS (DTT,VCOVER,ZLT,TM,TC,TGS,CAPAC,WWW,PPCMM,PPL,           &
                      ROFF,ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,          &
                   EXTK,ISNOW,P0,CSOIL,dzsoil,CHISL,SMELT)
        CALL SET0(TSSNO,BWO,BLO,BIO,HO,FLO,FIO,WO,DZO,                      &
***************
*** 1367,1373 ****
  !**********************************************
        SUBROUTINE SSIB_SEAICE                                  &
                       ( II, JJ, DDTT, ITIME, ZLAT, SUNANGLE,   &
!                        PPL, PPC, RLWDOWN, ZWIND2,             &
                         WWW1, WWW2, WWW3,                      &
                         TC, TGS, TD,                           &
                         SNOA, ROFF, YICE,                      &
--- 1367,1373 ----
  !**********************************************
        SUBROUTINE SSIB_SEAICE                                  &
                       ( II, JJ, DDTT, ITIME, ZLAT, SUNANGLE,   &
!                        PPL, PPCMM, RLWDOWN, ZWIND2,             &
                         WWW1, WWW2, WWW3,                      &
                         TC, TGS, TD,                           &
                         SNOA, ROFF, YICE,                      &
***************
*** 1407,1413 ****
  !     SWDOWN:   SHORT WAVE DOWN(W/M*M);
  !     RADFRAC:  SHORT WAVE COMPONENTS (visible and near IR; direct and diffuse)
  !     RLWDOWN:   LONG WAVE DOWN(W/M*M);
! !     PPL, PPC: LARGE SCALE AND CONVECTIVE PRECIPITATIONS AT THE TIME STEP (mm)
  !     TM:       TEMPERETURE AT LOWEST MODEL LAYER (K)
  !     UMM,VMM:  ZONAL AND MERIDIONAL WINDS AT LOWEST MODEL LAYER (m/S)
  !     QM:       WATER VAPOR AT LOWEST MODEL LAYER;
--- 1407,1413 ----
  !     SWDOWN:   SHORT WAVE DOWN(W/M*M);
  !     RADFRAC:  SHORT WAVE COMPONENTS (visible and near IR; direct and diffuse)
  !     RLWDOWN:   LONG WAVE DOWN(W/M*M);
! !     PPL, PPCMM: LARGE SCALE AND CONVECTIVE PRECIPITATIONS AT THE TIME STEP (mm)
  !     TM:       TEMPERETURE AT LOWEST MODEL LAYER (K)
  !     UMM,VMM:  ZONAL AND MERIDIONAL WINDS AT LOWEST MODEL LAYER (m/S)
  !     QM:       WATER VAPOR AT LOWEST MODEL LAYER;
***************
*** 1637,1643 ****
         POROS=0.95
  !***
  !
!       CALL INTERC(DTT ,VCOVER,ZLT,TM,TC,TGS,CAPAC,WWW,PPC,PPL,ROFF,       &
              ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,EXTK,RNOFFS,FILTR,     &
              SMELT)
  !
--- 1637,1643 ----
         POROS=0.95
  !***
  !
!       CALL INTERC(DTT ,VCOVER,ZLT,TM,TC,TGS,CAPAC,WWW,PPCMM,PPL,ROFF,       &
              ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,EXTK,RNOFFS,FILTR,     &
              SMELT)
  !
***************
*** 2172,2178 ****
  
  !=======================================================================
  !                                                                       
!       SUBROUTINE INTERCS (DTT,VCOVER,ZLAI,TM,TC,TGS,CAPAC,WWW,PPC,PPL, &
                           ROFF,ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,  &
                           EXTK,ISNOW,P0,CSOIL,DZSOIL,                   &
                           CHISL,SMELT)
--- 2172,2178 ----
  
  !=======================================================================
  !                                                                       
!       SUBROUTINE INTERCS (DTT,VCOVER,ZLAI,TM,TC,TGS,CAPAC,WWW,PPCMM,PPL, &
                           ROFF,ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,  &
                           EXTK,ISNOW,P0,CSOIL,DZSOIL,                   &
                           CHISL,SMELT)
***************
*** 2201,2210 ****
  !
        AP = PCOEFS(2,1)
        CP = PCOEFS(2,2)
!       TOTALP = PPC + PPL
        IF(TOTALP.LT.1.E-8)GO TO 6000
!       AP = PPC/TOTALP * PCOEFS(1,1) + PPL/TOTALP * PCOEFS(2,1)
!       CP = PPC/TOTALP * PCOEFS(1,2) + PPL/TOTALP * PCOEFS(2,2)
   6000 CONTINUE
        ROFF = 0.
        THRU = 0.
--- 2201,2210 ----
  !
        AP = PCOEFS(2,1)
        CP = PCOEFS(2,2)
!       TOTALP = PPCMM + PPL
        IF(TOTALP.LT.1.E-8)GO TO 6000
!       AP = PPCMM/TOTALP * PCOEFS(1,1) + PPL/TOTALP * PCOEFS(2,1)
!       CP = PPCMM/TOTALP * PCOEFS(1,2) + PPL/TOTALP * PCOEFS(2,2)
   6000 CONTINUE
        ROFF = 0.
        THRU = 0.
***************
*** 2412,2418 ****
  !------------------------------------------------------
  !=======================================================================
  !
!       SUBROUTINE INTERC(DTT,VCOVER,ZLT,TM,TC,TGS,CAPAC,WWW,PPC,PPL,ROFF,  &
            ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,EXTK,RNOFFS,FILTR,SMELT)
  !                                                         12 AUGUST 2000
  !=======================================================================
--- 2412,2418 ----
  !------------------------------------------------------
  !=======================================================================
  !
!       SUBROUTINE INTERC(DTT,VCOVER,ZLT,TM,TC,TGS,CAPAC,WWW,PPCMM,PPL,ROFF,  &
            ZDEPTH,POROS,CCX,CG,SATCO,SATCAP,SPWET,EXTK,RNOFFS,FILTR,SMELT)
  !                                                         12 AUGUST 2000
  !=======================================================================
***************
*** 2442,2451 ****
  !
        AP = PCOEFS(2,1)
        CP = PCOEFS(2,2)
!       TOTALP = PPC + PPL
        IF(TOTALP.LT.1.E-8)GO TO 6000
!       AP = PPC/TOTALP * PCOEFS(1,1) + PPL/TOTALP * PCOEFS(2,1)
!       CP = PPC/TOTALP * PCOEFS(1,2) + PPL/TOTALP * PCOEFS(2,2)
   6000 CONTINUE
  !
        ROFF = 0.
--- 2442,2451 ----
  !
        AP = PCOEFS(2,1)
        CP = PCOEFS(2,2)
!       TOTALP = PPCMM + PPL
        IF(TOTALP.LT.1.E-8)GO TO 6000
!       AP = PPCMM/TOTALP * PCOEFS(1,1) + PPL/TOTALP * PCOEFS(2,1)
!       CP = PPCMM/TOTALP * PCOEFS(1,2) + PPL/TOTALP * PCOEFS(2,2)
   6000 CONTINUE
  !
        ROFF = 0.

Fix a small syntax error (according to xlf) in phys/module_radiation_driver.F:

*** phys/module_radiation_driver.F.orig    2014-03-18 11:29:28.036740000 +1300
--- phys/module_radiation_driver.F    2014-03-18 11:37:28.952280000 +1300
***************
*** 1703,1709 ****
                kt=swdown(i,j)/max(ioh,1e-3) ! clearness index
                ! Optical air mass: Rigollier et al. (2000) doi: 10.1016/S0038-092X(99)00055-9
                airmass=exp(-ht(i,j)/8434.5)/(coszen(i,j)+ &
!                      0.50572*((1.5707963267948966-acos(coszen(i,j)))*57.295779513082323+6.07995)**-1.6364)
                ! kt correction for air-mass at large sza: Perez et al. (1990) doi: 10.1016/0038-092X(90)90036-C
                kt=kt/(0.1+1.031*exp(-1.4/(0.9+(9.4/max(airmass,1e-3)))))
                ! Diffuse fraction: Ruiz-Arias et al. (2010) (Eq 33) doi: 10.1016/j.enconman.2009.11.024
--- 1703,1709 ----
                kt=swdown(i,j)/max(ioh,1e-3) ! clearness index
                ! Optical air mass: Rigollier et al. (2000) doi: 10.1016/S0038-092X(99)00055-9
                airmass=exp(-ht(i,j)/8434.5)/(coszen(i,j)+ &
!                      0.50572*((1.5707963267948966-acos(coszen(i,j)))*57.295779513082323+6.07995)**(-1.6364))
                ! kt correction for air-mass at large sza: Perez et al. (1990) doi: 10.1016/0038-092X(90)90036-C
                kt=kt/(0.1+1.031*exp(-1.4/(0.9+(9.4/max(airmass,1e-3)))))
                ! Diffuse fraction: Ruiz-Arias et al. (2010) (Eq 33) doi: 10.1016/j.enconman.2009.11.024

ulimit -s unlimited # For compiler memory usage
./compile em_real 2>&1 | tee compile1.log
This will crash on  module_first_rk_step_part1 and you must compile it without optimization:

% cd dyn_em
% opt/ibmcmp/xlf/bg/11.1/bin/bgxlf90_r -o module_first_rk_step_part1.o -c -O0   -qnoipa -qarch=auto -qcache=auto -qtune=auto -w -qspill=1000000000 -qmaxmem=-1 -qfree=f90 -qnosmp -I../dyn_em -I../dyn_nmm -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/external/esmf_time_f90 -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/main -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/external/io_netcdf -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/external/io_int -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/frame -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/share -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/phys -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/chem -I/hpc/home/projects/packages/local.bgp/pkg/WRF/WRFV3.5/inc -I/bgp/local/pkg/netcdf/version/include -qrealsize=4 -qintsize=4 -qsuffix=f=f90 module_first_rk_step_part1.f90 -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/include -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/include

If you want WRF to output GRIB2 files, make this change to arch/Config_new.pl

% diff -c Config_new.pl*
*** Config_new.pl	2013-06-25 10:40:53.837437376 +1200
--- Config_new.pl-DIST	2013-06-25 10:40:50.867432249 +1200
***************
*** 185,191 ****
  # environment variables JASPERLIB and JASPERINC to paths to library and 
  # include files to enable this feature prior to running configure.  
  
!  $I_really_want_to_output_grib2_from_WRF = "TRUE" ;
  
   if ( $ENV{JASPERLIB} && $ENV{JASPERINC} && $I_really_want_to_output_grib2_from_WRF eq "TRUE" )
     {
--- 185,191 ----
  # environment variables JASPERLIB and JASPERINC to paths to library and 
  # include files to enable this feature prior to running configure.  
  
!  $I_really_want_to_output_grib2_from_WRF = "FALSE" ;
  
   if ( $ENV{JASPERLIB} && $ENV{JASPERINC} && $I_really_want_to_output_grib2_from_WRF eq "TRUE" )
     {

If you get many complaints from xlf90 saying "... A variable declared in a pure subprogram must not have the SAVE or STATIC attribute.  You may have specified the SAVE option explicitly or by default." then you must add "-qnosave" to FCOPTIM in configure.wrf.

On BGL

The build process for WRF v3 is more refined than WRF v2 and now has BG/L-specific options built in. These are the environment settings to use:

export PNETCDF=/bgl/local/pkg/parallelNetCDF/version
export NETCDF=/bgl/local/pkg/netCDF/version

export WRFIO_NCD_LARGE_FILE_SUPPORT=1

# To use GRIB2 I/O, set these paths to jasper:
export JASPERLIB=/bgl/local/pkg/jasper/version/lib
export JASPERINC=/bgl/local/pkg/jasper/version/include

We ran into a problem when linking wrf.exe, getting many reports of

relocation truncated to fit: R_PPC_REL24 against symbol ...

The cause of the problem was most likely that some address references
in the object modules were too far away for the address range of the
compiled instructions.

The fix for this was to link wrf.exe using the "-Wl,-relax" flag to
blrts_xlf90. This should be added to LDFLAGS_LOCAL in configure.wrf,
which for our installation now looks like:

LDFLAGS_LOCAL = -Wl,-relax -qstatic

Running the compile script on the FEN detects the correct options. However it was necessary to make the following minor change to the arch/configure_new.defaults to pick up the correct version of the xlf cpp:

# this might be different on different systems but we want the xlf version of cpp, not Linux's
#CPP             =       /opt/ibmcmp/xlf/9.1/exe/cpp -C -P
# CJM
CPP             =       /opt/ibmcmp/xlf/bg/11.1/exe/cpp -C -P

Also it was necessary to rebuild NetCDF for the BG/L because I don't think I got it right the first time around. I put the new version into into /hpc/bluefern/local.bgl/pkg/netCDF/bgl/3.6.2_mk2 and you must link against these libraries.

Also we discovered that when attempting to compile WRF with a new Registry file it is necessary to run ./clean -a first because this step deletes the Registry/Registry file and then the ./compile step makes a guess at which one to use. Hence the steps are:

  1. Do a ./clean -a
  2. Copy the new Registry file into place as Registry/Registry
  3. Run ./compile wrf or ./compile em_real

Bob Walkup of IBM made these very helpful suggestions for the WRFV3 configuration on Blue Gene:

You can get a modest improvement in WRF performance by choosing a 2D
decomposition (nproc_x, nproc_y in the wrf namelist.input) that can be
neatly folded onto the 3D torus network. Let's take an example where WRF
is started using virtual-node mode and 512 MPI processes, on 256 BGL nodes.
The shape of the 256-node partition is 8x4x8 in units of nodes on the torus
network. You could use the two processes on each node to form a logical
16x32 process mesh, where the "16" dimension = 8x2 is formed by going down
the X-axis in "TX" order ... ranks 0 and 1 go on the first node, ranks 2
and 3 go on the next node in the X direction etc. By repeating that
pattern, you can fill out sucessive Y locations, then fold back when you
get to the edge of the Y space. The torus wrap-around links are not
enabled until you get to blocks that are one or more midplanes in
extent ... so it is best to carefully fold the 2D sheet back and forth as
the mapping progresses. That way boundary exchange between
nearest-neighbors on the 2D grid will remain local on the 3D torus. It is
easy to generate a mapfile for this, but the performance gain is likely to
be modest, because for small partitions, the nodes are never very far apart
in terms of torus hops.

It is probably more important to compile all of the computationally
intensive routines with -O3. My experience is that one can use -qarch=440
for WRF, because the compiler will not be able to effectively use SIMD
instructions, but -O3 helps considerably. Also, you can link in the
libmass.a library for faster versions of exp, pow, log, etc.

There are options for I/O that can help. With WRF version 3, you can use
parallel netcdf for things like history-file writes (io_form=11). My
experience has been that parallel netcdf is helpful up to 2K MPI ranks or
so.

Those are the main BGL-related tips that I can think of. Jim Edwards
(IBMer at NCAR) has more experience with WRF on BGL, and can probably add
suggestions.

Regards,
Bob Walkup (walkup@us.ibm.com, 914-945-1512)

Compiling WRF-CHEM on AIX

# Define which model core to build (use ARW always):
export WRF_EM_CORE=1
export WRF_NMM_CORE=0
# Chemistry code is to be included in the WRF model build:
export WRF_CHEM=1
# Kinetic Pre-Processor (KPP) code:
export WRF_KPP=1
export FLEX_LIB_DIR=/usr/local/pkg/flex/version64/lib

# The manual warns not to use any shared-memory options for WRF-CHEM
# so be sure to use dmpar only for the supported platform:
./configure chem
./compile em_real

# Optionally:
./compile emi_conv

Compiling WRF-CHEM on BG/L

To compile WRF_CHEM, download and untar the WRF-CHEM distribution into a newly-untarred WRF directory (with matching version numbers) and then
Define which model core to build (use ARW for now).

export PNETCDF=/bgl/local/pkg/parallelNetCDF/version
export NETCDF=/bgl/local/pkg/netCDF/version

export WRFIO_NCD_LARGE_FILE_SUPPORT=1

# To use GRIB2 I/O, set these paths to jasper:
export JASPERLIB=/bgl/local/pkg/jasper/version/lib
export JASPERINC=/bgl/local/pkg/jasper/version/include

# Define which model core to build (use ARW always):
export WRF_EM_CORE=1
export WRF_NMM_CORE=0

# Chemistry code is to be included in the WRF model build:
export WRF_CHEM=1
# Kinetic Pre-Processor (KPP) code:
export WRF_KPP=1

export FLEX_LIB_DIR=/usr/lib
# export FLEX_LIB_DIR=/bgl/local/pkg/flex/version/lib
export YACC="yacc -d"

for PATCHFILE in module_add_emiss_burn.F module_aerosols_sorgam.F
do
        echo "Fixing $PATCHFILE"
        mv chem/$PATCHFILE chem/$PATCHFILE.DIST
        cp ../SITE/WRF-CHEM-KPP.3.2/$PATCHFILE.fix chem/$PATCHFILE
done

./configure
echo '
*** configure.wrf.orig  2010-11-11 14:41:02.670457780 +1300
--- configure.wrf       2010-11-11 14:43:27.176701627 +1300
***************
*** 112,118 ****
  ARCH_LOCAL      =       -DMOVE_NL_OUTSIDE_MODULE_CONFIGURE -DNONSTANDARD_SYSTEM_SUBR -DLANDREAD_STUB -DVERY_LARGE_MAXPROC=36768 -DBLUEGENE
  CFLAGS_LOCAL    =       -DNOUNDERSCORE -DNCARIBM_NOC99 $(MPI_INC)  -DLANDREAD_STUB
  LIB_LOCAL       =       $(MPI_LIB)
! LDFLAGS_LOCAL   =       -Wl,--allow-multiple-definition -qstatic
  CPLUSPLUSLIB    =
  ESMF_LDFLAG     =       $(CPLUSPLUSLIB)
  FCOPTIM               =       -O2 -qarch=440
--- 112,118 ----
  ARCH_LOCAL      =       -DMOVE_NL_OUTSIDE_MODULE_CONFIGURE -DNONSTANDARD_SYSTEM_SUBR -DLANDREAD_STUB -DVERY_LARGE_MAXPROC=36768 -DBLUEGENE
  CFLAGS_LOCAL    =       -DNOUNDERSCORE -DNCARIBM_NOC99 $(MPI_INC)  -DLANDREAD_STUB
  LIB_LOCAL       =       $(MPI_LIB)
! LDFLAGS_LOCAL   =       -Wl,-relax -qstatic
  CPLUSPLUSLIB    =
  ESMF_LDFLAG     =       $(CPLUSPLUSLIB)
  FCOPTIM               =       -O2 -qarch=440
***************
*** 130,136 ****
  # NYBlue
  CPP             =       /opt/ibmcmp/xlf/bg/10.1/exe/cpp -C -P
  # frost.ucar.edu
! CPP             =       /opt/ibmcmp/xlf/9.1/exe/cpp -C -P
  AR              =       ar
  ARFLAGS         =       ru
  M4            =       m4 -B 14000
--- 130,136 ----
  # NYBlue
  CPP             =       /opt/ibmcmp/xlf/bg/10.1/exe/cpp -C -P
  # frost.ucar.edu
! # CPP             =       /opt/ibmcmp/xlf/9.1/exe/cpp -C -P
  AR              =       ar
  ARFLAGS         =       ru
  M4            =       m4 -B 14000

*** chem/KPP/compile_wkc        2010-11-11 09:36:24.644901929 +1300
--- chem/KPP/compile_wkc.DIST   2010-11-09 14:56:29.933308186 +1300
***************
*** 43,51 ****
  echo "-----------------------------------------"


! ( cd $WKC_HOME/util/wkc; ./linker.csh unlink )

! ( cd $WKC_HOME/util/wkc; ./linker.csh link )

  #compile the coupler
  echo compile the coupler
--- 43,51 ----
  echo "-----------------------------------------"


! ( cd $WKC_HOME/util/wkc; linker.csh unlink )

! ( cd $WKC_HOME/util/wkc; linker.csh link )

  #compile the coupler
  echo compile the coupler
***************
*** 111,117 ****

  # direct addressing in the decomp subroutine (similar to a code by
  #  Edwin Spee, CWI, Amsterdam)
!  ( cd ${WKC_HOME}/util/write_decomp; ./write_decom.csh ${model} ../../.. )

  end

--- 111,117 ----

  # direct addressing in the decomp subroutine (similar to a code by
  #  Edwin Spee, CWI, Amsterdam)
!  ( cd ${WKC_HOME}/util/write_decomp; write_decom.csh ${model} ../../.. )

  end

*** ./chem/KPP/kpp/kpp-2.1/src/Makefile 2010-11-11 14:46:36.322334188 +1300
--- ./chem/KPP/kpp/kpp-2.1/src/Makefile.orig    2008-03-22 03:36:28.000000000 +1300
***************
*** 41,48 ****
  #CFLAGS= -Aa
  CFLAGS=
  CLFLAGS = -lm
- # Invoke local compiler for kpp:
- SCC = /usr/bin/xlc

  all: kpp

--- 41,46 ----
***************
*** 65,72 ****
        debug.o

  kpp:    $(OBJS)
!       @echo "  "$(SCC) $(CC_FLAGS) $(CLFLAGS) $(CFLAGS) -o kpp $(OBJS) -L$(FLEX_LIB_DIR) -lfl
!       @$(SCC) $(CC_FLAGS) $(CFLAGS) $(CLFLAGS) -o kpp $(OBJS) -L$(FLEX_LIB_DIR) -lfl
        @mv kpp ../bin

  clean:
--- 63,70 ----
        debug.o

  kpp:    $(OBJS)
!       @echo "  "$(SCC) $(CC_FLAGS) $(CLFLAGS) $(CFLAGS) $(OBJS) -L$(FLEX_LIB_DIR) -lfl -ll -o kpp
!       @$(SCC) $(CC_FLAGS) $(CFLAGS) $(CLFLAGS) $(OBJS) -L$(FLEX_LIB_DIR) -lfl -ll -o kpp
        @mv kpp ../bin

  clean:

*** chem/KPP/util/wkc/Makefile  2010-11-11 09:32:20.155796402 +1300
--- chem/KPP/util/wkc/Makefile.DIST     2010-11-11 09:24:54.192813551 +1300
***************
*** 17,26 ****
  
  
  registry : $(OBJ)
!       /usr/bin/xlc -o registry_kpp $(DEBUG) $(LDFLAGS) $(OBJ)
  
  .c.o :
!       /usr/bin/xlc -I../../../../inc $(CFLAGS) -c $(DEBUG) $<
  
  clean:
        /bin/rm -f $(OBJ) gen_comms.c
--- 17,26 ----
  
  
  registry : $(OBJ)
!       $(SCC) -o registry_kpp $(DEBUG) $(LDFLAGS) $(OBJ)
  
  .c.o :
!       $(SCC) -I../../../../inc $(CFLAGS) -c $(DEBUG) $<
  
  clean:
        /bin/rm -f $(OBJ) gen_comms.c
*** chem/KPP/util/write_decomp/Makefile 2010-12-08 17:34:28.435081695 +1300
--- chem/KPP/util/write_decomp/Makefile.DIST    2010-12-08 17:25:39.843434776 +1300
***************
*** 28,45 ****
        $(MAKE) comp
        ./write_decomp.exe
        $(MAKE) integr_edit
!       ./integr_edit.exe $(MECH)
        $(MAKE) clean 
  
  
  comp:  $(MODULES) $(OBJS)
!       xlf -o write_decomp.exe $(MODULES) $(OBJS)
        echo $(MECH) > mech.tmp
  
  
  
  integr_edit: integr_edit.o
!       xlc -o integr_edit.exe integr_edit.o
  
  
  include ${CPATH}/../configure.wrf
--- 28,45 ----
        $(MAKE) comp
        ./write_decomp.exe
        $(MAKE) integr_edit
!       integr_edit.exe $(MECH)
        $(MAKE) clean 
  
  
  comp:  $(MODULES) $(OBJS)
!       $(SFC) -o write_decomp.exe $(MODULES) $(OBJS)
        echo $(MECH) > mech.tmp
  
  
  
  integr_edit: integr_edit.o
!       $(SCC) -o integr_edit.exe integr_edit.o
  
  
  include ${CPATH}/../configure.wrf
***************
*** 51,62 ****
        $(SED_FTN2) $*.F > $*.b 
        $(CPP) -I${CPATH}/../inc $(CPPFLAGS) $*.b  > $*.f90
        $(RM) $*.b
!       xlf -c $(FCFLAGS) $(MODULE_DIRS) $(PROMOTION) $(FCSUFFIX) $*.f90
  
  
  .c.o:
        $(RM) $@
!       xlc -c $(CFLAGS) $*.c
  
  clean: 
        $(RM) *.o *.tmp *.mod module_kpp_* *~ *.inc *.exe *.f90
--- 51,62 ----
        $(SED_FTN2) $*.F > $*.b 
        $(CPP) -I${CPATH}/../inc $(CPPFLAGS) $*.b  > $*.f90
        $(RM) $*.b
!       $(SFC) -c $(FCFLAGS) $(MODULE_DIRS) $(PROMOTION) $(FCSUFFIX) $*.f90
  
  
  .c.o:
        $(RM) $@
!       $(SCC) -c $(CFLAGS) $*.c
  
  clean: 
        $(RM) *.o *.tmp *.mod module_kpp_* *~ *.inc *.exe *.f90

' | patch -p0 -N -R
cd chem/KPP/util/wkc
ln -s ../../../../tools/registry.h .
ln -s ../../../../tools/protos.h .
ln -s ../../../../tools/data.h .
ln -s ../../../../tools/sym.h .
cd ../../../..

./compile em_real 2>&1 | tee compile.log

# Optionally:
./compile emi_conv

Compiling emiss_v03

This is a little utility that comes with WRF-CHEM. The README says to use "pgf90 -Mfree -byteswapio -o emiss_v03.exe emiss_v03.F" but you can compile it with the XL compilers as follows:

  1. On the versions of emiss_v03.F that I could find, it was necessary to make a couple of changes to get the file to compile:
  • Change all occurrences of the LNBLNK function to LEN_TRIM
  • Add a space between STOP and '777' on line 697 (this line number varies between versions).
  1. Compile with xlf90 -o emiss_v03.exe -O2 emiss_v03.F

Compiling prep_chem_input

See AppsWRFchemEmissions

Compiling WRF v2

Due to the vast differences in their achitecture it is necessary to use different procedures in order to build WRF for use on the pSeries machine and the BG/L system. Instructions that are specific to these two platforms follow.

pSeries-specific Instructions

Preamble

There appears to be a problem with the build system that comes with WRF v2.x. Specifically one can follow the procedure laid out in the WRFV2/README file but the WRFV2/main directory does not get populated with the correct .exe files. If you looks closely at the compilation messages you will see messages of the form:

{{{"module_diffusion_em.f90", line 8.9: 1514-219 (S) Unable to access module symbol file for module module_big_step_utilities_em. Check path and file permissions of file. Use association not done for this module.1501-511  Compilation failed for file module_diffusion_em.f90.make: 1254-004 The error code from the last command is # make: 1254-005 Ignored error code 1 from last command.}}}

http://mailman.ucar.edu/pipermail/wrf-users/2007/000323.html mentions that optimization level O3 using blrts_xlf90 causes compliation failure for programs such as mediation_force_domain. Setting optimization to -O2 fixes this.

Build Procedure

In order to successfully build WRF on the pSeries system it is necessary to use the following procedure:

  1. Expressly set the compilation mode to 64-bit using the following command:
    export OBJECT_MODE=64
  2. Follow the procedure laid out in the WRFV2/README i.e.
    a. ./configure and choose Option 3. (AIX DM-Parallel (RSL_LITE, IBM_MPI, Allows Nesting) a. ./compile em_real
  3. Move into the WRFV2/dyn_em directory and run each of the following sets of commands (you can cut and paste each set of commands into your terminal window; beware of line wraps)
    a.

            rm -f module_big_step_utilities_em.o
            sed -f ../arch/standard.sed module_big_step_utilities_em.F > module_big_step_utilities_em.b
            /lib/cpp -I../inc -I. -C -P -DRSL_LITE -DDM_PARALLEL -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4  -DLWORDSIZE=4 -DNETCDF   -DTRIEDNTRUE -DGRIB1  -DINTIO  -DLIMIT_ARGS -DNO_NAMELIST_PRINT -DNATIVE_MASSV -DNMM_NEST=0 -I../external/RSL_LITE {{cat ../inc/dm_comm_cpp_flags}} -DEM_CORE=1  -DNMM_CORE=0 -DNMM_MAX_DIM=2600  -DCOAMPS_CORE=0  -DEXP_CORE=0  -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=16384 -DMAX_DOMAINS_F=21 module_big_step_utilities_em.b  > module_big_step_utilities_em.f90
            rm -f module_big_step_utilities_em.b
            mpxlf90_r -c -O2 -qarch=auto -w -qspill=20000  -qmaxmem=32767  -I../dyn_em          -I../dyn_nmm                        -I../external/io_netcdf -I../external/io_pnetcdf  -I../external/io_int  -I../external/esmf_time_f90  -I../frame -I../share -I../phys -I../chem -I../inc -qrealsize=4 -qintsize=4 -qsuffix=f=f90 module_big_step_utilities_em.f90
    

    a.

            rm -f module_diffusion_em.o
            sed -f ../arch/standard.sed module_diffusion_em.F > module_diffusion_em.b
            /lib/cpp -I../inc -I. -C -P -DRSL_LITE -DDM_PARALLEL -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4  -DLWORDSIZE=4 -DNETCDF   -DTRIEDNTRUE -DGRIB1  -DINTIO  -DLIMIT_ARGS -DNO_NAMELIST_PRINT -DNATIVE_MASSV -DNMM_NEST=0 -I../external/RSL_LITE {{cat ../inc/dm_comm_cpp_flags}} -DEM_CORE=1  -DNMM_CORE=0 -DNMM_MAX_DIM=2600  -DCOAMPS_CORE=0  -DEXP_CORE=0  -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=16384 -DMAX_DOMAINS_F=21 module_diffusion_em.b  > module_diffusion_em.f90
            rm -f module_diffusion_em.b
            mpxlf90_r -c -O2 -qarch=auto -w -qspill=20000  -qmaxmem=32767  -I../dyn_em          -I../dyn_nmm                        -I../external/io_netcdf -I../external/io_pnetcdf  -I../external/io_int  -I../external/esmf_time_f90  -I../frame -I../share -I../phys -I../chem -I../inc -qrealsize=4 -qintsize=4 -qsuffix=f=f90 module_diffusion_em.f90
    

    a.

            rm -f module_advect_em.o
            sed -f ../arch/standard.sed module_advect_em.F > module_advect_em.b
            /lib/cpp -I../inc -I. -C -P -DRSL_LITE -DDM_PARALLEL -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4  -DLWORDSIZE=4 -DNETCDF   -DTRIEDNTRUE -DGRIB1  -DINTIO  -DLIMIT_ARGS -DNO_NAMELIST_PRINT -DNATIVE_MASSV -DNMM_NEST=0 -I../external/RSL_LITE {{cat ../inc/dm_comm_cpp_flags}} -DEM_CORE=1  -DNMM_CORE=0 -DNMM_MAX_DIM=2600  -DCOAMPS_CORE=0  -DEXP_CORE=0  -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=16384 -DMAX_DOMAINS_F=21 module_advect_em.b  > module_advect_em.f90
            rm -f module_advect_em.b
            mpxlf90_r -c -O2 -qarch=auto -w -qspill=20000  -qmaxmem=32767  -I../dyn_em          -I../dyn_nmm                        -I../external/io_netcdf -I../external/io_pnetcdf  -I../external/io_int  -I../external/esmf_time_f90  -I../frame -I../share -I../phys -I../chem -I../inc -qrealsize=4 -qintsize=4 -qsuffix=f=f90 module_advect_em.f90
    

    a.

            rm -f module_em.o
            sed -f ../arch/standard.sed module_em.F > module_em.b
            /lib/cpp -I../inc -I. -C -P -DRSL_LITE -DDM_PARALLEL -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4  -DLWORDSIZE=4 -DNETCDF   -DTRIEDNTRUE -DGRIB1  -DINTIO  -DLIMIT_ARGS -DNO_NAMELIST_PRINT -DNATIVE_MASSV -DNMM_NEST=0 -I../external/RSL_LITE {{cat ../inc/dm_comm_cpp_flags}} -DEM_CORE=1  -DNMM_CORE=0 -DNMM_MAX_DIM=2600  -DCOAMPS_CORE=0  -DEXP_CORE=0  -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=16384 -DMAX_DOMAINS_F=21 module_em.b  > module_em.f90
            rm -f module_em.b
            mpxlf90_r -c -O2 -qarch=auto -w -qspill=20000  -qmaxmem=32767  -I../dyn_em          -I../dyn_nmm                        -I../external/io_netcdf -I../external/io_pnetcdf  -I../external/io_int  -I../external/esmf_time_f90  -I../frame -I../share -I../phys -I../chem -I../inc -qrealsize=4 -qintsize=4 -qsuffix=f=f90 module_em.f90
    

    a.

            rm -f module_small_step_em.o
            sed -f ../arch/standard.sed module_small_step_em.F > module_small_step_em.b
            /lib/cpp -I../inc -I. -C -P -DRSL_LITE -DDM_PARALLEL -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4  -DLWORDSIZE=4 -DNETCDF   -DTRIEDNTRUE -DGRIB1  -DINTIO  -DLIMIT_ARGS -DNO_NAMELIST_PRINT -DNATIVE_MASSV -DNMM_NEST=0 -I../external/RSL_LITE {{cat ../inc/dm_comm_cpp_flags}} -DEM_CORE=1  -DNMM_CORE=0 -DNMM_MAX_DIM=2600  -DCOAMPS_CORE=0  -DEXP_CORE=0  -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=16384 -DMAX_DOMAINS_F=21 module_small_step_em.b  > module_small_step_em.f90
            rm -f module_small_step_em.b
            mpxlf90_r -c -O2 -qarch=auto -w -qspill=20000  -qmaxmem=32767  -I../dyn_em          -I../dyn_nmm                        -I../external/io_netcdf -I../external/io_pnetcdf  -I../external/io_int  -I../external/esmf_time_f90  -I../frame -I../share -I../phys -I../chem -I../inc -qrealsize=4 -qintsize=4 -qsuffix=f=f90 module_small_step_em.f90
    

    a.

            rm -f module_bc_em.o
            sed -f ../arch/standard.sed module_bc_em.F > module_bc_em.b
            /lib/cpp -I../inc -I. -C -P -DRSL_LITE -DDM_PARALLEL -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4  -DLWORDSIZE=4 -DNETCDF   -DTRIEDNTRUE -DGRIB1  -DINTIO  -DLIMIT_ARGS -DNO_NAMELIST_PRINT -DNATIVE_MASSV -DNMM_NEST=0 -I../external/RSL_LITE {{cat ../inc/dm_comm_cpp_flags}} -DEM_CORE=1  -DNMM_CORE=0 -DNMM_MAX_DIM=2600  -DCOAMPS_CORE=0  -DEXP_CORE=0  -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=16384 -DMAX_DOMAINS_F=21 module_bc_em.b  > module_bc_em.f90
            rm -f module_bc_em.b
            mpxlf90_r -c -O2 -qarch=auto -w -qspill=20000  -qmaxmem=32767  -I../dyn_em          -I../dyn_nmm                        -I../external/io_netcdf -I../external/io_pnetcdf  -I../external/io_int  -I../external/esmf_time_f90  -I../frame -I../share -I../phys -I../chem -I../inc -qrealsize=4 -qintsize=4 -qsuffix=f=f90 module_bc_em.f90
    

    a.

            rm -f module_initialize_real.o
            sed -f ../arch/standard.sed module_initialize_real.F > module_initialize_real.b
            /lib/cpp -I../inc -I. -C -P -DRSL_LITE -DDM_PARALLEL -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4  -DLWORDSIZE=4 -DNETCDF   -DTRIEDNTRUE -DGRIB1  -DINTIO  -DLIMIT_ARGS -DNO_NAMELIST_PRINT -DNATIVE_MASSV -DNMM_NEST=0 -I../external/RSL_LITE {{cat ../inc/dm_comm_cpp_flags}} -DEM_CORE=1  -DNMM_CORE=0 -DNMM_MAX_DIM=2600  -DCOAMPS_CORE=0  -DEXP_CORE=0  -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=16384 -DMAX_DOMAINS_F=21 module_initialize_real.b  > module_initialize_real.f90
            rm -f module_initialize_real.b
            mpxlf90_r -c -qrealsize=4 -qintsize=4 -w -qspill=20000  -qmaxmem=32767  -I../dyn_em          -I../dyn_nmm                        -I../external/io_netcdf -I../external/io_pnetcdf  -I../external/io_int  -I../external/esmf_time_f90  -I../frame -I../share -I../phys -I../chem -I../inc -qsuffix=f=f90 -qnoopt module_initialize_real.f90
    
  4. Rerun the ./compile em_real command from the WRFV2 directory.
  5. Check the contents of the WRFV2/main directory. It should look something like:
    {{{-rwxrwxr-x 1 mti18 e001 4926 Mar 28 14:22 Makefile
    -rwxrwxr-x 1 mti18 e001 5376 Mar 28 14:22 convert_em.F
    -rwxrwxr-x 1 mti18 e001 7041 Mar 28 14:22 ideal.F
    rw-rr- 1 cjm133 z001 17858146 Apr 12 15:57 libwrflib.a
    -rwxr-xr-x 1 cjm133 z001 9883641 Apr 12 15:57 ndown.exe
    -rwxrwxr-x 1 mti18 e001 65521 Mar 28 14:22 ndown_em.F
    rw-rr- 1 cjm133 z001 59696 Apr 12 15:53 ndown_em.f90
    rw-rr- 1 cjm133 z001 128882 Apr 12 15:53 ndown_em.o
    -rwxr-xr-x 1 cjm133 z001 8987706 Apr 12 15:57 nup.exe
    -rwxrwxr-x 1 mti18 e001 36191 Mar 28 14:22 nup_em.F
    rw-rr- 1 cjm133 z001 29730 Apr 12 15:53 nup_em.f90
    rw-rr- 1 cjm133 z001 32774 Apr 12 15:53 nup_em.o
    -rwxr-xr-x 1 cjm133 z001 8683421 Apr 12 15:57 real.exe
    -rwxrwxr-x 1 mti18 e001 46545 Apr 11 13:46 real_em.F
    rw-rr- 1 cjm133 z001 44723 Apr 12 15:52 real_em.f90
    rw-rr- 1 cjm133 z001 88023 Apr 12 15:53 real_em.o
    -rwxrwxr-x 1 mti18 e001 45218 Mar 28 14:22 real_nmm.F
    -rwxrwxr-x 1 mti18 e001 803 Mar 28 14:22 wrf.F
    -rwxr-xr-x 1 cjm133 z001 17494228 Apr 12 15:57 wrf.exe
    rw-rr- 1 cjm133 z001 803 Apr 12 14:25 wrf.f90
    rw-rr- 1 cjm133 z001 1686 Apr 12 14:25 wrf.o
    -rwxrwxr-x 1 mti18 e001 8387 Mar 28 14:22 wrf_ESMFApp.F
    -rwxrwxr-x 1 mti18 e001 51234 Mar 28 14:22 wrf_ESMFMod.F
    -rwxrwxr-x 1 mti18 e001 88494 Mar 28 14:22 wrf_SST_ESMF.F

     
  6. If something goes wrong and you need to start again you can "clean" the directories of all object code etc by using the ./clean -a command in the WRFV2 directory

BG/L-specific Instructions

WRF v2

The BG/L-specific build of WRF is contained in the /hpc/bluefern/WRF/v2.2/WRFV2/ directory. Bob Walkup at IBM in the US (walkup@us.ibm.com) has very kindly created some BG/L-specific customizations to the following WRF build files:

  1. Top level Makefile
  2. arch/configure.defaults

According to Bob most of the changes pertain to the C-preprocessor with certain files requiring the -traditional cpp flag; these changes are reflected in the above custom files. Copies of these files are in the /hpc/bluefern/WRF/v2.2/ibmCustomisations directory.

With the above files in place all that is necessary is the normal WRF build sequence, namely:

  1. Set the NETCDFPATH=/bgl/local/pkg/netCDF/bgl/3.6.2 (as dicussed in AppsNetCDF);
  2. ./configure and choose Option 1
  3. ./compile em_real to build wrf.exe, real.exe etc

Note that the last step above may take several hours to complete; this is normal. If you only require wrf.exe then you could use the ./compile wrf option.


CategoryApps CategoryHpc

  • No labels