Hp XC System 4.x Software Bedienungsanleitung

HP XC System SoftwareXC User GuideVersion 4.0HP Part Number: A-XCUSR-40aPublished: February 2009

List of Examples5-1 Submitting a Job from the Standard Input...505-2

Table 10-2 Output Provided by the bhist Command (continued)DescriptionFieldThe total unknown time of the job.UNKWNThe total time that the job has spen

$ sacct -j 123Jobstep Jobname Partition Ncpus Status Error---------- ------------------ ---------- ------- ---------- -----123

$ export SLURM_JOBID=150$ srun hostnamen1n2n3n4Note:Be sure to unset the SLURM_JOBID when you are finished with the allocation, to prevent aprevious S

Note:If you exported any variables, such as SLURM_JOBID and SLURM_NPROCS, be sure to unsetthem as follows before submitting any further jobs from the

Table 10-3 LSF Equivalents of SLURM srun Options (continued)LSF EquivalentDescriptionsrun Option-ext"SLURM[nodelist=node1,...nodeN]"Requests

Table 10-3 LSF Equivalents of SLURM srun Options (continued)LSF EquivalentDescriptionsrun OptionYou cannot use this option. LSF uses this optionto cre

Table 10-3 LSF Equivalents of SLURM srun Options (continued)LSF EquivalentDescriptionsrun OptionUse as an argument to srun when launchingparallel task

11 Advanced TopicsThis chapter covers topics intended for the advanced user. This chapter addresses the followingtopics:• “Enabling Remote Execution w

$ echo $DISPLAY:0Next, get the name of the local machine serving your display monitor:$ hostnamemymachineThen, use the host name of your local machine

$ sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTlsf up infinite 2 idle n[46,48]According to the information returned about this

About This DocumentThis document provides information about using the features and functions of the HP XC SystemSoftware. It describes how the HP XC u

proceed while another is waiting for I/O. On an HP XC system, there is the potential to usecompute nodes to do compilations, and there are a variety o

for i in ${HYPRE_DIRS}; \ do \ if [ -d $$i ]; \ then \ echo "Making $$i ..."; \ (cd $

Modified Makefile:all: $(MAKE) $(MAKE_J) struct_matrix_vector/libHYPRE_mv.astruct_linear_solvers/libHYPRE_ls.a utilities/libHYPRE_utilities.a

11.5 I/O Performance ConsiderationsBefore building and running your parallel application, I/O performance issues on the HP XCcluster must be considere

respectively. These subsections are not full solutions for integrating MPICH with the HP XCSystem Software.Figure 11-1 MPICH Wrapper Script#!/bin/cshs

A ExamplesThis appendix provides examples that illustrate how to build and run applications on the HPXC system. The examples in this section show you

Examine the partition information:$ sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTlsf up infinite 6 idle n[5-10]Examine the loca

date and time stamp: Submitted from host <lsfhost.localdomain>, CWD <$HOME>, 2 Processors Requested;date and time stam

steps through a series of commands that illustrate what occurs when you launch an interactiveshell.Examine the LSF execution host information:$ bhosts

Summary of time in seconds spent in various states by date and time PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL 11 0 124 0 0 0

Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that youmust hold down the key labeled Ctrl while you press anotherkey or mouse button.ENVI

srun hostnamesrun uname -aRun the job:$ bsub -I -n4 myjobscript.shJob <1006> is submitted to default queue <normal>.<<Waiting for di

Show the SLURM job ID:$ env | grep SLURMSLURM_JOBID=74SLURM_NPROCS=8Run some commands from the pseudo-terminal:$ srun hostnamen13n13n14n14n15n15n16n16

View the node state:$ sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTlsf up infinite 4 idle n[13-16]A.7 Submitting an HP-MPI Jo

EXTERNAL MESSAGES:MSG_ID FROM POST_TIME MESSAGE ATTACHMENT0 - - - - 1 lsfadmin

124

GlossaryAadministrationbranchThe half (branch) of the administration network that contains all of the general-purposeadministration ports to the nodes

operating system and its loader. Together, these provide a standard environment for bootingan operating system and running preboot applications.enclos

image server A node specifically designated to hold images that will be distributed to one or more clientsystems. In a standard HP XC installation, th

LVS Linux Virtual Server. Provides a centralized login capability for system users. LVS handlesincoming login requests and directs them to a node with

onboardadministratorSee OA.PparallelapplicationAn application that uses a distributed programming model and can run on multiple processors.An HP XC MP

Provides an overview of managing the HP XC user environmentwith modules, managing jobs with LSF, and describes how tobuild, run, debug, and troublesho

an HP XC system, the use of SMP technology increases the number of CPUs (amount ofcomputational power) available per unit of space.ssh Secure Shell. A

IndexAACML library, 45application development, 39building parallel applications, 45building serial applications, 42communication between nodes, 113com

CP3000, 20MKL library, 45system interconnect, 22CP3000BL, 20CP4000, 20ACML library, 45compilers, 40, 44designing libraries for, 46MKL library, 45softw

job accounting, 84job allocation informationobtaining, 97job manager, 86job scheduler, 86JOBID translation, 100Llaunching jobssrun, 81libraries, 27bui

developing, 39environment for developing, 24examples of, 115partitionreporting state of, 83PATH environment variablesetting with a module, 34Pathscale

UPC, 39user environment, 33VVampir, 77VampirTrace/Vampir, 75WWeb siteHP XC System Software documentation, 12Xxtermrunning from remote node, 107135

software components are generic, and the HP XC adjective is not added to any reference to athird-party or open source command or product name. For exa

• http://www.balabit.com/products/syslog_ng/Home page for syslog-ng, a logging tool that replaces the traditional syslog functionality.The syslog-ng t

MPI Web Sites• http://www.mpi-forum.orgContains the official MPI standards documents, errata, and archives of the MPI Forum. TheMPI Forum is an open g

Manpages for third-party software components might be provided as a part of the deliverablesfor that component.Using discover(8) as an example, you ca

18

1 Overview of the User EnvironmentThe HP XC system is a collection of computer nodes, networks, storage, and software, built intoa cluster, that work

© Copyright 2003, 2005, 2006, 2007, 2008, 2009 Hewlett-Packard Development Company, L.P.Confidential computer software. Valid license from HP required

$ head /proc/cpuinfoTable 1-1 presents the representative output for each of the platforms. This output may differaccording to changes in models and s

distributes login requests from users. A node with the login role is referredto as a login node in this manual.compute role The compute role is assign

the HP XC. So, for example, if the HP XC system interconnect is based on a Quadrics® QsNetII® switch, then the SFS will serve files over ports on that

Additional information on supported system interconnects is provided in the HP XC HardwarePreparation Guide.1.1.8 Network Address Translation (NAT)The

Modulefiles can be loaded into the your environment automatically when you log in to thesystem, or any time you need to alter the environment. The HP

1.4.2 Serial ApplicationsYou can build and run serial applications under the HP XC development environment. A serialapplication is a command or applic

1.5.4 How LSF and SLURM InteractIn the HP XC environment, LSF cooperates with SLURM to combine the powerful schedulingfunctionality of LSF with the sc

1.6 Components, Tools, Compilers, Libraries, and DebuggersThis section provides a brief overview of the some of the common tools, compilers, libraries

28

2 Using the SystemThis chapter describes the tasks and commands that the general user must know to use thesystem. It addresses the following topics:•

Table of ContentsAbout This Document...11Intended

overview about some basic ways of running and managing jobs. Full information and detailsabout the HP XC job launch environment are provided in “Using

For more information about using this command and a sample of its output, see “ExaminingSystem Core Status” (page 95)• The LSF lshosts command display

2.3.1 Determining the LSF Cluster Name and the LSF Execution HostThe lsid command returns the LSF cluster name, the LSF version, and the name of the L

3 Configuring Your Environment with ModulefilesThe HP XC system supports the use of Modules software to make it easier to configure andmodify the your

access the mpi** scripts and libraries. You can specify the compiler it uses through a variety ofmechanisms long after the modulefile is loaded.The pr

Table 3-1 Supplied Modulefiles (continued)Sets the HP XC User Environment to Use:ModulefileIntel C/C++ Version 8.1 compilers.icc/8.1/defaultIntel C/C+

Each module supplies its own online help. See “Viewing Modulefile-Specific Help” for informationon how to view it.3.3 Modulefiles Automatically Loaded

For example, if you wanted to automatically load the TotalView modulefile when you log in,edit your shell startup script to include the following inst

In this example, a user attempted to load the ifort/8.0 modulefile. After the user issued thecommand to load the modulefile, an error message occurred

4 Developing ApplicationsThis chapter discusses topics associated with developing applications in the HP XC environment.Before reading this chapter, y

2.3.1 Determining the LSF Cluster Name and the LSF Execution Host...322.4 Getting System Help and Information..

HP UPC is a parallel extension of the C programming language, which runs on both commontypes of multiprocessor systems: those with a common global add

4.3 Examining Nodes and Partitions Before Running JobsBefore launching an application, you can determine the availability and status of the system&apo

4.6.1 Serial Application Build EnvironmentYou can build and run serial applications in the HP XC programming environment. A serialapplication is a com

4.7.1.1 ModulefilesThe basics of your working environment are set up automatically by your system administratorduring the installation of HP XC. Howev

To compile programs that use SHMEM, it is necessary to include the shmem.h file and to usethe SHMEM and Elan libraries. For example:$ gcc -o shping sh

4.7.1.12 MKL LibraryMKL is a math library that references pthreads, and in enabled environments, can use multiplethreads. MKL can be linked in a singl

To compile and link a C application using the mpicc command:$ mpicc -o mycode hello.c To compile and link a Fortran application using the mpif90 comma

names. However, HP recommends an alternative method. The dynamic linker, during its attemptto load libraries, will suffix candidate directories with t

NOTE: There is no shortcut as there is for the dynamic loader.48 Developing Applications

5 Submitting JobsThis chapter describes how to submit jobs on the HP XC system; it addresses the following topics:• “Overview of Job Submission” (page

5.2 Submitting a Serial Job Using LSF...495.2.1 Submit

The srun command is only necessary to launch the job on the allocated node if the HP XC JOBSTARTER script is not configured to run a job on the comput

#include <unistd.h> #include <stdio.h> int main() { char name[100]; gethostname(name, sizeof(name)); printf("%s says Hello!\n&

The bsub command submits the job to LSF.The -n num-procs parameter, which is required for parallel jobs, specifies the number of coresrequested for th

variable that was set by LSF; this environment variable is equivalent to the number provided bythe -n option of the bsub command.Any additional SLURM

options that specify the minimum number of nodes required for the job, specific nodes for thejob, and so on.Note:The SLURM external scheduler is a plu

Example 5-9 Using the External Scheduler to Submit a Job to Run on Specific Nodes$ bsub -n4 -ext "SLURM[nodelist=n6,n8]" -I srun hostnameJob

Example 5-13 Using the External Scheduler to Constrain Launching to Nodes with a Given Feature$ bsub -n 10 -ext "SLURM[constraint=dualcore]"

Example 5-15 Submitting a Batch Script with the LSF-SLURM External Scheduler Option$ bsub -n4 -ext "SLURM[nodes=4]" -I ./myscript.shJob <

Example 5-18 Environment Variables Available in a Batch Job Script$ cat ./envscript.sh#!/bin/sh name=`hostname` echo "hostname = $name" echo

The ping_pong_ring application is submitted twice in a Makefile named mymake; the firsttime as run1 and the second as run2:$ cat mymake PPR_ARGS=1000

9.7 Job Accounting...849.

1This line attempts to submita program that does notexist.The following command line makes the program and executes it:$ bsub -o %J.out -n2 -ext "

5.6 Submitting a Job from a Host Other Than an HP XC HostTo submit a job from a host other than an HP XC host to the HP XC system, use the LSF -Roptio

62

6 Debugging ApplicationsThis chapter describes how to debug serial and parallel applications in the HP XC developmentenvironment. In general, effectiv

6.2.1 Debugging with TotalViewTotalView is a full-featured, debugger based on GUI and specifically designed to fill therequirements of parallel applic

6.2.1.3 Using TotalView with SLURMUse the following commands to allocate the nodes you need before you debug an applicationwith SLURM, as shown here:$

6.2.1.6 Debugging an ApplicationThis section describes how to use TotalView to debug an application.1. Compile the application to be debugged. For exa

6.2.1.7 Debugging Running ApplicationsAs an alternative to the method described in “Debugging an Application”, it is also possible to"attach"

68

7 Monitoring Node ActivityThis chapter describes the optional utilities that provide performance information about the setof nodes associated with you

A.4 Launching a Parallel Interactive Shell Through LSF...117A.5 Submitting a Simple J

7.2 Running Performance Health TestsYou can run the ovp command to generate reports on the performance health of the nodes. Usethe following format to

NOTE: Except for the network_stress and network_bidirectional tests,these tests only apply to systems that install LSF incorporated with SLURM. Thenet

Verify perf_health: Testing cpu_usage ... The headnode is excluded from the cpu usage test. Number of nodes allocated for this test i

The tests execution directory has been saved in:HOME_DIRECTORY/ovp_n16_mmddyy.testsDetails of this verification have been recorded in:HOME_DIRECTORY/o

74

8 Tuning ApplicationsThis chapter discusses how to tune applications in the HP XC environment.8.1 Using the Intel Trace Collector and Intel Trace Anal

Example 8-1 The vtjacobic Example ProgramFor the purposes of this example, the examples directory under /opt/IntelTrace/ITC iscopied to the user'

8.2 The Intel Trace Collector and Analyzer with HP-MPI on HP XCNOTE: The Intel Trace Collector (OTA) was formerly known as VampirTrace. The Intel Trac

Running a ProgramEnsure that the -static-libcxa flag is used when you use mpirun.mpich to launch a C orFortran program.The following is a C example ca

[0] Intel Trace Collector INFO: Writing tracefile vtjacobif.stf in /nis.home/user_name/xc_PDE_work/ITC_examples_xc6000mpirun exits with status: 0Runni

List of Figures4-1 Library Directory Structure...

80

9 Using SLURMHP XC uses the Simple Linux Utility for Resource Management (SLURM) for system resourcemanagement and job scheduling.This chapter address

The srun command handles both serial and parallel jobs.The srun command has a significant number of options to control the execution of yourapplicatio

Example 9-2 Displaying Queued Jobs by Their JobIDs$ squeue --jobs 12345,12346 JOBID PARTITION NAME USER ST TIME_USED NODES NODELIST(REASON)

Example 9-8 Reporting Reasons for Downed, Drained, and Draining Nodes$ sinfo -R REASON NODELIST Memory errors

10 Using LSFThe Load Sharing Facility (LSF) from Platform Computing is a batch system resource managerused on the HP XC system.On an HP XC system, a j

The LSF environment is set up automatically for the user on login; LSF commands and theirmanpages are readily accessible:• The bhosts command is usefu

“Translating SLURM and LSF JOBIDs” describes the relationship betweenthe SLURM_JOBID and the LSF JOBID.SLURM_NPROCSThis environment variable passes al

Example 10-2 Examples of Launching LSF Jobs Without the srun CommandThe following bsub command line invokes the bash shell to run the hostname command

• LSF integrated with SLURM only runs daemons on one node within the HP XC system. Thisnode hosts an HP XC LSF Alias, which is an IP address and corre

List of Tables1-1 Determining the Node Platform...201-

sometime in the future, depending on resource availability andbatch system scheduling policies.Batch job submissions typically provide instructions on

allocates the appropriate whole node for exclusive use by theserial job in the same manner as it does for parallel jobs, hencethe name “pseudo-paralle

request more than one core for a job. This option, coupled with the external SLURM scheduler,discussed in “LSF-SLURM External Scheduler”, gives you mu

Figure 10-1 How LSF and SLURM Launch and Manage a JobN 1 6N16User124666677775job_starter.sh$ srun -nl myscriptLogin node$ bsub-n4 -ext ”SLURM[nodes-4]

4. LSF prepares the user environment for the job on the LSF execution host node and dispatchesthe job with the job_starter.sh script. This user enviro

10.10.1 Examining System Core StatusThe bhosts command displays LSF resource usage information. This command is useful toexamine the status of the sys

10.10.3 Getting Host Load InformationThe LSF lsload command displays load information for LSF execution hosts.$ lsloadHOST_NAME status r15s r1m

on this topic. See the LSF manpages for full information about the commands described in thissection.The following LSF commands are described in this

date and time stamp: Started on 4 Hosts/Processors <4*lsfhost.localdomain>;date and time stamp: slurm_id=22;ncpus=8;slurm_alloc=n[5-8];Example 1

Example 10-6 Using the bjobs Command (Long Output)$ bjobs -l 24Job <24>, User <msmith>,Project <default>,Status <RUN>,