seeing as there are many issues with installation, here options that work with the latest version (12/2010 release) of exc!ting :
system - linux cluster
PBS queue manager
excitingser, excitingmpi and excitingsmp are all running fine.
Since I've seen a few "newbie" problems posted, here is some advice :
- http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ gives the link line for your particular system and mkl version. This is very useful because intel, in its infinite wisdom, changes the names of everything in any minor update, and you've got to get it just right.
- there are a few known unstable releases of the intel software. E.g. 11.0.64 was nasty; and ifort12 won't compile the WIEN2k code properly (haven't tested for exciting).
- compilers don't like to socialize. Make sure your ifort compiler and mkl libraries are the same release. Compile openmpi using the same compiler you're using to compile the exciting code, if you can. I don't know how fussy exciting is, but it matters for some other codes.
- some multi-user clusters acquire loads of different compilers and libraries. Try 'which ifort', 'which gcc', 'which mpif90', 'which mpirun' etc. to figure out what's really being used.
- I haven't checked optimization extensively ; I could imagine adding -xHost for extra speed gain, or -mp -prec-div -fpp-model xxx (I forget what the 'safe' option is called) for floating point accuracy.
- If you're going to copy the code below, make sure $MKLROOT is set to your equivalent of /home/jorissen/…/lib/em64t in your .cshrc or .bashrc file. It's probably not in your home directory, but rather in /opt/intel or so.
Here's the make.inc I'm using:
$ more ~/exciting/build/make.inc F90=ifort F90_OPTS = -O3 -ip -unroll -scalar_rep -w90 CPP_ON_OPT = -cpp -DXS -DISO -DTETRA -DLIBXC F77=$(F90) F77_OPTS = -O3 LIB_ARP =libarpack.a #export USE_SYS_LAPACK=true LIB_LPK = -L/home/jorissen/soft/intel/Compiler/11.1/075/mkl/lib/em64t/ $(MKLROOT)/libmkl_solver_lp6 4.a -Wl,--start-group $(MKLROOT)/libmkl_intel_lp64.a $(MKLROOT)/libmkl_intel_thread.a $(MKLROOT)/libmkl_core.a -Wl,--end-group -openmp -lpthread LIB_FFT = fftlib.a LIB_BZINT=libbzint.a LIBS= $(LIB_ARP) $(LIB_LPK) $(LIB_FFT) $(LIB_BZINT) F90_DEBUGOPTS=-g -O0 -warn all -check all -traceback -ftrapuv F77_DEBUGOPTS=-g -O0 -warn all -check all -traceback -ftrapuv #Ignore if you don't have MPI or smplibs MPIF90=mpif90 MPIF90_OPTS=$(F90_OPTS) $(CPP_ON_OPT) -DMPI -DMPIRHO -DMPISEC F77MT=$(F77) F90MT=$(F90) SMP_LIBS= $(LIB_ARP) $(LIB_LPK) $(LIB_FFT) $(LIB_BZINT) SMPF90_OPTS=$(F90_OPTS) SMPF77_OPTS=$(SMPF90_OPTS) MPISMPF90_OPTS=$(SMPF90_OPTS) -DMPI -DMPIRHO -DMPISECBUILDMPI=false MPIF90MT=$(MPIF90) BUILDMPI=true BUILDSMP=true
To submit a job to my PBS queue, I put the following script in my working directory:
$ more exciting.pbs.sh #!/bin/csh #PBS -l nodes=4:ppn=12 #PBS -N Test #PBS -o Test.sout #PBS -e Test.serr #PBS -q dode #PBS -V setenv NP 48 cd $PBS_O_WORKDIR mpirun -np $NP --hostfile $PBS_NODEFILE $EXCITINGROOT/bin/excitingmpi $
You'll want to edit the -l and -q options; NP; make sure $EXCITINGROOT is set correctly. If you want to use the serial version of exciting, you can use the line
mpirun -np 1 —hostfile $PBS_NODEFILE $EXCITINGROOT/bin/excitingsmp (or excitingser)
to submit the job.
If you don't use a queue, just write the NODEFILE yourself; e.g.
$ echo n01 slots=8 max_slots=8 > ./hosts $ mpirun -np 8 --hostfile ./hosts $EXCITINGROOT/bin/excitingmpi