While every precaution has been taken in the preparation of this document, The Portland Group™ Compiler Technology, STMicroelectronics, Inc. (PGI®) makes no warranty for the use of its products and assumes no responsibility for any errors that may appear, or for damages resulting from the use of the information contained herein. PGI retains the right to make changes to this information at any time, without notice. The software described in this document is distributed under license from STMicroelectronics, Inc. and may be used or copied only in accordance with the terms of the license agreement. No part of this document may be reproduced or transmitted in any form or by any means, for any purpose other than the purchaser's personal use without the express written permission of PGI.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this manual, PGI was aware of a trademark claim. The designations have been printed in caps or initial caps.
PGF90 is a trademark and PGI, PGHPF, PGF77, PGCC, PGPROF, and PGDBG are registered trademarks of The Portland Group Compiler Technology, STMicroelectronics, Inc.  Other brands and names are the property of their respective owners.
PGI 4.1 Cluster Development Kit   (CDK™)          Installation  & Release Notes
Copyright © 2002
The Portland Group™ Compiler Technology
STMicroelectronics, Inc. - All rights reserved.
 Printed in the United States of America
First Printing: Release 4.1, April 2003
Technical support:	trs@pgroup.com
			http://www.pgroup.com

1	 PGI CDK 4.1 
Installation Notes

A Cluster is a collection of compatible computers connected by a network. The PGI CDK Cluster Development Kit supports parallel computation on clusters of Intel Pentium 2/3/4 or AMD Athlon/AthlonXP/AthlonMP compatible Linux workstations or servers interconnected by a TCP/IP-based network such as ethernet or fast ethernet.  Linux networking must be properly installed and configured on the cluster prior to installation of the PGI CDK.  Prior releases of the PGI CDK required that rsh work between all nodes in the cluster. With the 4.1 release, users can utilize ssh or rsh  for MPI-CH, PBS, and pgdbg implementation.
1.1	Introduction 
You can view the online HTML interface to the PGI CDK using any web browser.  Enter the following URL in a web browser:

       file:/mnt/cdrom/index.htm
running on a Linux system with the PGI CDK CD-ROM inserted and mounted on the CD-ROM drive.  If you do not know how to insert and mount a CD-ROM on your Linux system, see Step 3 below in section 1.2 or contact your system administrator.
Release 4.1 of the PGI CDK consists of the following PGI compilers and tools:
•	The PGHPF High Performance Fortran Compiler
•	The PGF90 OpenMP and autothreading Fortran 90 Compiler
•	The PGF77 OpenMP and autothreading FORTRAN 77 Compiler
•	The PGCC OpenMP and autothreading ANSI and K&R C compiler
•	The PGCC OpenMP and autothreading C++ Compiler
•	The Multi-process, Multi-thread supporting PGPROF graphical profiler 
•	The Multi-process, Multi-thread supporting PGDBG graphical debugger 
the following open source clustering utilities:
•	MPI-CH version 1.2.5, an implementation of the Message-Passing Interface (MPI) standard, compiled for use with the PGI compilers on Linux systems with a kernel revision of 2.2.10 or higher.
•	ScaLAPACK linear algebra math library for distributed-memory systems, including BLACS version 1.1 (the Basic Linear Algebra Communication Subroutines) and ScaLAPACK version 1.7 for use with MPI-CH and the PGI compilers on Linux systems with a kernel revision of 2.2.10 or higher.
•	PBS portable batch queuing system from Veridian Technologies, version 2.3.15, configured for Linux systems with a kernel revision of 2.2.10 or higher
and the following documentation and tutorial materials:
•	OSC Training Materials – an extensive set of HTML-based parallel and scientific programming training materials developed by the Ohio Supercomputer Center
•	Complete online HTML Documentation for the PGI compilers and tools
•	Online HPF tutorials that provide insight into cluster programming considerations.
•	Online UNIX man pages for all of the supplied software
•	A hard-copy CD-ROM media kit including the PGI User’s Guide, MPI The Complete Reference, Volume 1, the High Performance Fortran Handbook, How to Build a Beowulf, and a printed copy of these release notes.
Generally, clusters are configured with a “master” node from which jobs are launched and “slave” nodes that are used only for computation.  Typically, the master node is accessible from the general-purpose or “public” network and shares a file system with the other computers on your network using NFS.  The master node and all of the slave nodes are interconnected using a second “private” network that is only accessible from computers that are part of the cluster.  There are two common cluster configurations:
1)	The master node is used only for compilation and job submission, and only the slave nodes are used for computation.
2)	All nodes are used for computation, including the master node
To use MPI-CH in the first configuration, PBS should be installed.  Otherwise, the master node will be used as one of the computation nodes by the mpirun command by default; it is possible to exclude the master node in the second configuration if mpirun is invoked with the –nolocal option (see the man page for mpirun).  If you are using the first configuration, it is possible to install MPI-CH and run parallel MPI or HPF jobs without installing any of the other components.  However, if you will have multiple users running jobs on your cluster simultaneously, you will likely want to use PBS  to ensure your cluster nodes are allocated and used efficiently.
Typically, a master node has two network cards to allow communication to the outside network as well as to the cluster nodes themselves, which may be on their own subnet.  If this is the case on your cluster, then when the installation script prompts you for the name of the master node, you should use the name associated with the network card connected to the cluster nodes.
Also, it is important to note that in order for MPI-CH and PBS to run correctly, access from each node to every other node must be available via the ‘rsh’ or ‘ssh’ command.  For example, if a 3-node cluster consists of a master and 2 slaves named master, node1, node2, then as a user you should be able to issue the commands:
% rsh master date
% rsh node2 date
or
% ssh master date
% ssh node2 date
From node1, and similarly from node2 and master.
By default, all of the PGI compilers and tools will be installed on your system.  You will select which of the open source components to install.  At this point, you should have determined:
•	Which PGI CDK open source components you will install (see above for a list of these)
•	The hostnames of all the nodes that will be included in your cluster - you will need a list of these during the installation
•	Whether the master node will be strictly a front-end for compilation, job launching, etc or whether it will participate as a compute node
•	If you are installing PBS, which users at your site will have PBS queue manager permissions (you will need their usernames)
•	Whether the compute nodes can share files with the master node (this is strongly recommended)

Section 1.2 below describes how to install the PGI Fortran, C and C++ compilers and tools on Linux using the installcdk script from the PGI CDK CD-ROM.  NOTE: you must have root permissions to successfully execute the installcdk script.  You must install the software as specified in section 1.2 and then follow the instructions in section 1.3 for configuring and starting the FLEXlm license daemon.  
The FLEXlm license daemon enables use of the PGI compilers and tools by any user on any system networked to the system on which the PGI software is installed.  For example, users can compile, debug, and profile using the PGI CDK compilers and tools on any system on your general-purpose network, subject to the constraints on concurrent usage for the product you have purchased.  
Sections 2.1 - 2.5 describe basic usage of the Open Source components of the PGI CDK, including MPI-CH, the PBS batch queuing system, ScaLAPACK libraries, and the example benchmark programs and tutorials.
For the first 60 days after your purchase, you may send technical questions about the PGI CDK compilers and tools to the e-mail address trs@pgroup.com.  If you have purchased PGI’s Subscription Service, you will have access to e-mail service for an additional 12 months and will be notified by e-mail when maintenance releases occur and are available for electronic download and installation. Contact PGI at sales@pgroup.com for information about the PGI Subscription Service for the products you have purchased.
MPI-CH, PBS, and ScaLAPACK, are all open source software packages that are not formally supported by PGI.  All source code for these components is included on the PGI CDK CD-ROM in the cdk subdirectory.  Each of these components has end-user and implementer documentation, generally in the form of printable postscript, along with the source code.  Support for these products is generally provided by their respective user communities, which you can learn more about at the following URLs:
•	MPI-CH - http://www-unix.mcs.anl.gov/mpi/mpich contains a wealth of information, including online documentation, tutorials, FAQ files, patch distributions, and information on how to submit bug reports to the MPI-CH developers.
•	PBS - http://www.openpbs.org is the main site for the PBS product we integrate with the CDK.
•	ScaLAPACK- http://www.netlib.org/scalapack contains FAQ files and current distributions of ScaLAPACK.
Once you’ve completed installation of the PGI CDK, the MAILUSERS file on the PGI CDK CD-ROM includes a simple text template for an introductory e-mail to end-users of your cluster system.  It explains how to initialize one’s environment, access the man pages and documentation, and execute a simple program on the cluster.

1.2	Installing the PGI CDK Compilers 
Step 1  Create the directory in which you wish to install the PGI CDK Compilers and tools.  NOTE:  The installation directory you choose must be accessible from all nodes in the cluster using a uniform pathname.  In the example below, we assume /usr/pgi which is the default installation directory.  However, installation can occur in any directory where you have appropriate permissions.  Please make sure that the installation directory has the necessary ownership and permissions appropriate for your site by using the chown and chmod commands.
Set the environment variable PGI to the name of the installation directory.  Assuming csh:
       % setenv PGI /usr/pgi
Or, assuming sh, ksh, or bash:
       % export PGI=/usr/pgi
Step 2  All software should fit into less than 250 MB of disk space.  If you wish to install all of the source code for the open source components of the PGI CDK, about 350 MB of disk space is required.  Verify that you have sufficient space on the disk where your installation directory will be located.
Step 3  The installcdk script must be run to properly install the software, and you must have root permissions to execute this script. If you do not have root privileges, you will need to get help from your system administrator during the installation process.
If you are installing from a CD-ROM and you’re not sure how to access the CD-ROM drive on your system, check with your system administrator.  Typically, you must insert the CD-ROM into the CD-ROM drive on the master node and issue the following command:

       % mount /mnt/cdrom
while logged in as root to make the data on the CD-ROM accessible.  Next, issue the following command from your root window on the master node:

       % /mnt/cdrom/installcdk
NOTE:  If you have difficulty running this script, especially on a Slackware Linux system, check the permissions on /dev/null.  Permissions should be set to crw-rw-rw-.  If they are not, reset them using chmod and try running the install script again, or try  ‘mount –o exec /mnt/cdrom’. Otherwise, check and see whether the CDROM was mounted with ‘noexec’ set. If so, unmount it and then mount it with ‘mount –o exec /mnt/cdrom’
The install script will install all of the binaries for the PGI compilers and tools, MPI-CH, and ScaLAPACK in the $PGI directory tree in the appropriate bin, include, lib, and man subdirectories.  It will also install PBS in the /usr/local/pbs and /var/spool/pbs directories, and start the PBS daemons running on the master node and all of the compute nodes in your cluster.  You will be prompted for various information about how to configure your cluster as the script executes.  Once the installation script has completed, exit the root login.
Step 4  All of the PGI CDK compilers and development tools are license-managed.  The other components of the PGI CDK, including MPI-CH, ScaLAPACK, and the PBS batch scheduler, are open source products that are not license-managed. 
If you choose to create a temporary demo license for the PGI compilers and tools, the install script asks for your real name, your username, and your email address.  It then creates a fifteen-day license and prints a message like this:
    NOTE: your evaluation license will expire in 	
    14 days, 23.6 hours. For a permanent license, 
    please read the order acknowledgement that you 
    received.  Connect to https://www.pgroup.com/License   
    with the username and password in the order 
    acknowledgement.

    Name:   <your name>
    User:   <your username>
    Email:  <your e-mail address>
    Hostid: PGI=9BF378E0131FF0C3CD37F6
    FLEXlm hostid: 00a024a3dfe7
    Hostname: yourhost.yourdomain.com
    Installation: /usr/pgi
    PGI Release: 4.1-2
The message above is also saved to the file $PGI/license.info for retrieval at a later time.  
Once you have obtained your permanent license keys using your personalized account on the PGI web page, place them in the file $PGI/license.dat. Until the permanent license is obtained, the PGI CDK compilers will only be usable under the username specified above during generation of the temporary keys.
Step 5  You can view the online HTML interface to the PGI CDK documentation and tutorial materials using any web browser.  Assuming you use Netscape, issue the following command:

       % netscape $PGI/index.htm
You can view the online HTML manuals for the PGI compilers and tools directly by issuing the following command:

       % netscape $PGI/doc/index.htm
You may want to place a bookmark on these locations for easy future reference to the online manuals.
Step 6  Once the temporary or permanent license file is in place, execute the following commands to make the PGI CDK compilers accessible from a normal user shell window. 
Assuming csh:
       % setenv PGI /usr/pgi
       % set path = ( $PGI/linux86/bin /usr/local/pbs/bin \
         $path )
       % setenv MANPATH "$MANPATH":$PGI/cdk/man:\
         /usr/local/pbs/man
Or, assuming sh or ksh:
%export PGI=/usr/pgi
% export PATH=$PGI/linux86/bin:/usr/local/pbs/bin:$PATH
% export MANPATH=$MANPATH:$PGI/man:/usr/local/pbs/man
Note: you may see a warning message when you set the MANPATH environment variable if you do not already have MANPATH defined.  
Each person who will be using the PGI CDK Fortran, C, and C++ compilers and tools should add the above commands to his or her startup files (e.g. .cshrc) to enable access to the PGI compilers and tools by default upon future logins.
Step 7  You can verify the release number of the compilers you have installed using the dryrun V options on any of the compiler commands.  This will also show you the sequence of steps the compiler will use to compile and link programs for execution on your system.  
•	For Fortran 77, use "pgf77 -V x.f"
•	For Fortran 90, use "pgf90 -V x.f"
•	For HPF, use "pghpf -V x.f"
•	For C++, use "pgCC -V x.c"
•	For C, use "pgcc -V x.c"
Note that the files x.f or x.c need not exist in order for you to successfully execute these commands.
The base installation of the PGI CDK is now complete.  Follow the directions in section 1.3 to enable floating license capability, and the following sections for basic usage instructions for MPI-CH, PBS, and ScaLAPACK.

1.3	Using FLEXlm on Linux
The PGI CDK Fortran, C, and C++ compilers and tools are license-managed using the FLEXlm software license management system from Globetrotter Software.  The steps below describe how to set up and run license daemons on the master node of your cluster.  These daemons allow the PGI CDK compilers to be used on any machine on your public network, subject to the constraint on maximum number of concurrent users as specified in your license.  The master node will function as the license “server”, distributing seats as requested for use on other computers on your network. 
Alternatively, if you already use other software products managed using FLEXlm, you can incorporate the PGI products into the configuration you use for license serving those products. 
Step 1  Install the PGI CDK compilers as described in section 1.2 above.
Step 2  Once you have obtained permanent license keys (see section 1.2 above for how to obtain these), place them in a file named license.dat in the $PGI directory.  For example, the license.dat file should look similar to the following:
  SERVER <hostname> <hostid> 7496
  DAEMON pgroupd <install_dir>/linux86/bin/pgroupd

  FEATURE pghpf-linux86 pgroupd 4.100 31-dec-0 1 \  
  2B9CF0F163159E4ABE32 VENDOR_STRING=123456:16:cdk \
  HOSTID=<hostid> ck=49
  . . .
It will include features for each of the PGI CDK compilers and tools.  <hostname> and <hostid> should match those you submitted to PGI and <install_dir> must be changed to match the directory in which the compilers are installed.  In particular, <install_dir> should match the value of $PGI as defined above.
Step 4  When the license file is in place, execute the following commands to make the PGI products you have purchased accessible. If you are not using other products managed by FLEXlm, and have not previously set the environment variable LM_LICENSE_FILE, issue the following command to do so (assuming csh):

% setenv LM_LICENSE_FILE $PGI/license.dat
Or, assuming sh, ksh, or bash:

% export LM_LICENSE_FILE=$PGI/license.dat
If you are using other products managed by FLEXlm, and have previously set the environment variable LM_LICENSE_FILE, either incorporate the PGI license keys into your existing license file or issue the following command to append the PGI license file to the definition of LM_LICENSE_FILE (assuming csh):
% setenv LM_LICENSE_FILE \ "$LM_LICENSE_FILE":$PGI/license.dat
Or, assuming sh or ksh or bash:
% export LM_LICENSE_FILE= \ $LM_LICENSE_FILE:$PGI/license.dat
Each person who will be using the PGI CDK compilers should add the above commands to his or her startup files to enable access to the PGI compilers and tools by default upon future logins.
NOTE:  If LM_LICENSE_FILE is not set or exported, and the node-locked 15-day temporary license file $PGI/PGIinstall still exists, then $PGI/PGIinstall will be used for resolving compiler licenses.
Step 5  You must now start the license manager daemon.  Edit the shell script template $PGI/linux86/bin/lmgrd.rc.  If you have installed the PGI CDK compilers in a directory other than /usr/pgi, substitute the correct installation directory into the definition of the PGI environment variable on line 3 of the script. Now exit the editor and issue the following command to start the license server and PGI license daemon running on your system:

       % lmgrd.rc start
If you wish to stop the license server and license daemon at a later time, you can do so with the command:

       % lmgrd.rc stop
To make sure that the license server and PGI daemon are started each time your system is booted, log in as root, set the PGI environment variable as above, and then execute the following two commands:
	
% cp $PGI/linux86/bin/lmgrd.rc /etc/rc.d/init.d/lmgrd

% ln -s /etc/rc.d/init.d/lmgrd /etc/rc.d/rc3.d/S90lmgrd
Note that your system’s default runlevel may be something other than ‘3’, and if it is, that number should be used above in setting the correct subdirectory.  Run /sbin/runlevel to check the system’s runlevel.  Note also that if you're using a Linux distribution other than Red Hat, your rc files may be in a directory other than /etc/rc.d.  Some Linux distributions, such as Red Hat and Mandrake, include the chkconfig(8) utility that manages the runlevel scripts.  If your system has this tool and you wish to use it, then run the following commands:

   % cp $PGI/linux86/bin/lmgrd.rc /etc/rc.d/init.d
   % chkconfig -- add lmgrd.rc
The appropriate links will be created in the /etc/rc.d directory hierarchy.  For more information on chkconfig, please see the manual page.
In addition to the installcdk installation script, there are two additional shell scripts that may be of use to you:
•	uninstallcdk - uninstalls selected components of the PGI CDK.  This is useful, for example, if you wish to download an updated version of one of the open source components and rebuild and reinstall it on your cluster.
•	restartpbs - shuts down and restarts all of the PBS daemons on each node of your cluster.  If you find that your default queue is not operating properly for some reason, try restarting PBS using this script to see if it fixes the problem.
As with the installcdk script, you must have root privileges in order to execute these scripts.  
Installation of your PGI CDK Fortran, C, and C++ compilers and tools for Linux is now complete. If you have difficulties with the installation, send e-mail to trs@pgroup.com for assistance.   
The following sections describe basic usage of the open source PGI CDK clustering utilities.

2      Using the Open Source Cluster Utilities

Copy the directory $PGI/bench to a local working area so you can try an example program.  
2.1	Running an MPI-CH Program
NOTE: you must either work in a directory which is file-shared with all of the cluster nodes, or you must copy your MPI executables to a common directory on all compute nodes before invocation of mpirun.  In particular, this precludes you from working in /tmp unless you copy the executable to /tmp on each slave node prior to invocation of mpirun.  
First, try the MPI “hello world” program in the bench/mpihello subdirectory:

% cp -r $PGI/bench ./bench
% cd ./bench/mpihello
% pgf77 -o mpihello mpihello.f -lfmpich –lmpich
% mpirun mpihello
Hello world!  I'm node 0
% mpirun -np 4 mpihello
Hello world!  I'm node 0
Hello world!  I'm node 2
Hello world!  I'm node 1
Hello world!  I'm node 3
If you've installed PBS, you should also try submitting a batch job to make sure your PBS default queue is operational.  There is an example PBS batch script for submission of the above “hello world” program in the bench/mpihello subdirectory.  It assumes you have a cluster with 4 or more processors.  You'll need to modify the batch script, mpihello.pbs, to ensure the pathname information included is correct for your particular installation.
IMPORTANT PBS NOTE 1 
A batch job submitted using the PBS qsub command does not by default inherit the environment of the spawning shell.  Instead, PBS batch jobs execute in an environment that is initialized based on the submitting user's login/shell startup files.  It may be the case that a user's home directory and shell/startup files are not accessible from the slave nodes (which generally are on their own private network).  In this case, you must be sure that each end-user of PBS has a valid home directory in the /etc/passwd file on each cluster node (generally it is the sixth field of a login entry in /etc/passwd).  If the home directory entry for a given user on any node  is invalid, PBS jobs will quietly fail in ways that are difficult to diagnose.
IMPORTANT PBS NOTE 2 
In order for a PBS batch job to find the mpirun command, the necessary path initialization must be performed.  It is best to perform the initialization either in each user's login/shell startup files as noted above, or in /etc on each slave node if you want to initialize it globally.  If you aren't sure how to do this, contact your system administrator.  Alternatively, you can use the -v or -V options to qsub (see the qsub man page for more on these options) to pass environment variables to the submitted job's environment, or you can explicitly initialize the path environment variable within the PBS batch script.  The latter method is used in the example below.  If the environment of a given batch job is not properly initialized in one of these ways, PBS jobs can fail to execute in ways that are difficult to diagnose.

Now, try submitting a PBS batch job using the following command:

% qsub mpihello.pbs
% qstat
You'll need to type qstat quickly in order to see the "mpihello" job in the queue.  Be sure to look at the mpihello.log file when the job completes to see that the job has executed correctly.  You should see output something like the following:
% cat mpihello.log
Hello world!  I'm node 0
Hello world!  I'm node 2
Hello world!  I'm node 1
Hello world!  I'm node 3
%
If these simple tests don't work, refer to the IMPORTANT PBS NOTEs above.  Usually, it's either a problem with 
•	A user not having a valid home directory entry in  /etc/passwd on each cluster node
•	An incorrectly initialized PATH variable in the shells executing the PBS job
•	Inability of one or more of the slave nodes to find "mpirun" because the PGI software has been installed in a directory which is not visible to them
If these simple tests do work, you're ready to execute some of the more extensive tests listed below in section 2.5, Testing and Benchmarking.
The following sections include more detailed information on each of the open source components of the PGI CDK.
2.2	More About PBS
PBS (the Portable Batch-queuing System) is a very configurable batch scheduler developed by the NASA Ames Research Center and Veridian Technologies. 
The installcdk script installs PBS for a space-shared cluster; that is, multiple jobs can be run at the same time, but at most one job will use a given node at any given time. Optionally, one node can be designated as a front-end node (usually call the master node) that can be used to submit jobs.  Many more options are available; to learn more, read the PBS Administration Guide, which is found in the cdk/pbs subdirectory of the PGI CDK CD-ROM.
Check that the PBS man pages are properly installed by bringing up one of the man pages:

% export MANPATH=/usr/local/pbs/man:$MANPATH
% man qsub

NOTE:  you may have to stop and then re-start the default queue once after PBS installation is complete and prior to running your first job.  It's not clear why this is necessary, but if your test job seems to queue up and not run (you can check its status using the qstat command), try issuing the following commands:

% qstop default
% qstart default
and then re-submitting the job.  You must be registered as a PBS queue manager or be logged in as root to execute these commands.  If you need to add queue managers at a later time, you may do so using the "qmgr" command while logged in as root:

% qmgr
Qmgr: set server managers=<username>@<masternode>
Qmgr: quit
where <username> is replaced with the username of the person who will become a queue manager, and <masternode> is replaced with the simple hostname of the master node in your cluster.  NOTE: In this case, the hostname cannot be a full hostname.  That is, if the full hostname of your master node is *.pgroup.com, you would enter * in place of <masternode> in the set server command. 
As mentioned in the introduction, PBS is very configurable.  The above steps provide a simple means to install PBS and establish a single default space-shared queue.  We strongly encourage your cluster administrator to print out the file cdk/pbs/pbs_admin_guide.ps and read through it to learn more about PBS and how it can best be used on your cluster.  There is also a very active mail list for PBS, which you can learn more about by browsing the main PBS web page at http://www.openpbs.org. 
2.3	Linking with ScaLAPACK
The ScaLAPACK libraries are automatically installed as part of step 1 above. You can link with the ScaLAPACK libraries by specifying 
–Mscalapack on any of the PGI CDK compiler command lines.  For example:
    % pgf77 myprog.f -Mscalapack
The -Mscalapack option causes the following libraries, all of which are installed in $PGI/linux86/lib, to be linked in to your executable:
•	scalapack.a
•	blacsCinit_MPI-LINUX-0.a
•	blacs_MPI-LINUX-0.a
•	blacsF77init_MPI-LINUX-0.a
•	libblas.a
•	libmpich.a
You can run a program that uses ScaLAPACK routines just like any other MPI program.  The version of ScaLAPACK included in the PGI CDK is pre-configured for use with MPI-CH.  If you wish to use a different BLAS library, and still use the –Mscalapack switch, you will have to copy your BLAS library into $PGI/linux86/lib/libblas.a.
Alternatively, you can just list the above set of libraries explicitly on your link line.  You can test that ScaLAPACK is properly installed by running a test program as outlined below in section 2.4, Testing and Benchmarking.
2.4	Testing and Benchmarking
The directory bench on the PGI CDK CD-ROM contains various benchmarks and tests.  Copy this directory into a local working directory by issuing the following command:

% cp -r $PGI/bench .
NAS Parallel Benchmarks - The NPB2.3 subdirectory contains version 2.3 of the NAS Parallel Benchmarks in MPI.  Issue the following commands to run the BT benchmark on 4 nodes of your cluster:

% cd bench/NPB2.3
% make BT NPROCS=4 CLASS=W
% cd bin
% mpirun -np 4 bt.W.4
There are several other NAS parallel benchmarks available in this directory.  Similar commands are used to build and run each of them.  Try building the Class A version of BT if you'd like to run a larger problem (just substitute "A" for "W" in the commands above).
The example above runs the BT benchmark on 4 nodes, but does not use the PBS batch queuing system.  There is a pre-configured PBS batch file in the NPB2.3/bin sub-directory.  Edit the file, change the cd command in the second to last line of the script to point to your local working directory, and then try executing the following commands to run BT under control of PBS:

% cd bin
% qsub bt.pbs
You can check on the status of your job using the qstat command.
The hpfnpb subdirectory contains versions of 5 of the NAS Parallel Benchmarks coded in High Performance Fortran (HPF).  README files explain how to build and run each of these benchmarks on various platforms.  Use the instructions and makefiles in the linux86 subdirectories of each benchmark to test these programs on your cluster.
MPI-CH - These tests measure latency and bandwidth of your cluster interconnect for MPI-CH messaging.  To run these tests, execute the following commands:

% cd mpi
% make
% mpirun -np 2 mpptest
For more information, the runmpptest script can be executed.  PGI has noted significant latency increases on Linux when messages larger than about 7600 bytes are sent, so this script may take some time to run. See the mpich/examples/perftest directory for more information.
ScaLAPACK - This test will time execution of the 3D PBLAS (parallel BLAS) on your cluster:

% cd scalapack
% make
% mpirun -np 4 pdbla3tim
Matrix Multiplication - This test will time execution of a simple distributed matrix multiply on your cluster:

% cd matmul
% buildhpf
% mpirun -np 4 matmul_hpf

3          PGI CDK 4.1 Release Notes

This document describes changes between PGI CDK 4.1 and previous releases, as well as late-breaking information not included in the current printing of the PGI User’s Guide.
3.1	Supported Systems and Licensing
PGI CDK 4.1 is supported on systems using the Intel Pentium or Pentium Pro/II/III/4 or compatible processors, including AMD Athlon/AthlonXP, and running Linux with a kernel version of 2.2.10 or above. This includes versions of Linux that use glibc2.2.x, such as Redhat 6.0 to 9.0, and SuSE 6.1 to  8.2.
The PGI CDK Fortran, C, and C++ compilers are license managed.  The open source components of the PGI CDK, including MPI-CH, ScaLAPACK, and PBS, are open source software packages that are not license-managed.  For the PGI CDK compilers, the FLEXlm license manager controls the number of simultaneous users.  When the PGI CDK compilers are first installed, they are usable for 15 days without a license key.  Please contact PGI to obtain a permanent license key as soon as possible.
To make the PGI CDK compilers operational, you will need to follow the installation instructions in Section 1 above, including installation of the license daemon. 

3.2	PGI CDK 4.1 Contents
Release 4.1 of the PGI CDK consists of the following PGI compilers and tools:
•	PGHPF data parallel High Performance Fortran Compiler
•	PGF90 native OpenMP and auto-threading Fortran 90 Compiler
•	PGF77 native OpenMP and auto-threading F77 Compiler
•	PGCC native OpenMP and auto-threading ANSI and K&R C compiler
•	PGC++ native OpenMP and auto-threading ANSI C++ Compiler 
•	PGPROF graphical profiler
•	PGDBG graphical debugger 
the following open source clustering utilities:
•	MPI-CH version 1.2.5, an implementation of the Message-Passing Interface (MPI) standard, compiled for use with the PGI compilers under Linux systems with a kernel revision of 2.2.10 or higher
•	ScaLAPACK linear algebra math library for distributed-memory systems, including BLACS version 1.1 (the Basic Linear Algebra Communication Subroutines) and ScaLAPACK version 1.7 for use with MPI-CH and the PGI compilers under Linux systems with a kernel revision of 2.2.10 or higher.
•	PBS portable batch queuing system from Veridian, version 2.3.15.  PBS is automatically configured and built upon installation to ensure compatibility with the system on which it is installed
and the following documentation and tutorial materials:
•	OSC Training Materials – an extensive set of HTML-based parallel and scientific programming training materials developed by the Ohio Supercomputer Center
•	Complete online HTML Documentation for the PGI compilers and tools
•	Online UNIX man pages for all of the supplied software
•	A hard-copy CD-ROM media kit including the PGI User’s Guide, MPI The Complete Reference, Volume 1, the High Performance Fortran Handbook, How to Build a Beowulf, and a printed copy of these release notes.

3.3	New Features
Following are the new features included in The PGI CDK 4.1:
•	The MPI-CH libraries have been updated to version 1.2.5
•	Ssh and rsh are now supported for cluster process control. 
•	The Rogue Wave Standard Template Library has been replaced with  STLport, version 4.5. Rogue  Wave is NOT supported in release 4.1.

3.4	New Compiler Options
No new compiler oprtions are present in this release, though there have been some corrections for proper execution. Along with 4.0, the SSE/SSE2 type instructions are presented here. For general performance purposes, we recommend users try these three sets first; 
 -fast, -fastsse, and –fastsse –Mnontemporal. 
•	Mscalarsse  a newly introduced technology for CPUs that support SSE and/or SSE2 type instructions. Pentium III and AthlonXP CPUs support SSE instruction types, while the Pentium 4 CPU also supports the SSE2 instruction types. In prior releases only the vectorizing optimizations used these instructions, and now they are being utilized in all coding opportunities. Note: some versions of Linux have assemblers that do not support the newer SSE style instructions. This switch should only be used if your assembler ‘as’ accepts these instructions. Older versions of Linux (for example, Red Hat 6.2) do not accept the SSE2 type instructions.
•	fastsse – This switch combines the switches for                              ‘ –fast –Mscalarsse –Mvect=sse –Mcache_align –Mflushz’ that combines a set of optimizations that frequently work well together to improve performance.  This switch is meant only for machines with SSE instruction type support, as in the Pentium III and AthlonXP.  It will also work on Pentium 4s, and supports SSE and SSE2 instruction types in that case. See the warning above about your assembler version.
•	-Mnontemporal – This switch is important to note with     –fastsse  -Mvect=sse options. Some programs run slower with –fastsse, due to the prefetching used.                 –Montemporal offers a different data movement scheme. When tuning code, it is good practice to try ‘-fastsse    –Mnontemporal’ in addition to ‘-fastsse‘  alone, to see which will be faster.

3.5	Problems Corrected in Release 4.1
The following problems are corrected in the current release. A description of the problem is given, but some problems can only be described in general terms because of complexity or confidentiality.  

Technical Problem Reports (TPRs) Corrected in 4.1-1	
TPR	Lang	Description	Symptom	
2823	pgf90	Reading real ‘3.2e4’ with ‘I’ edit descriptor in list directed integer input did not cause error message.	No error message produced during execution.	
2826	pgf90 	Intrinsic call with a derived data type caused an internal compiler error.	pgf90  -c –Mfree xx.f
ICE: mkexpr1: bad id      14 (xx.f: 185)	
2835	pgf90	Fails to detect error with –Mstandard 
read(…,…,SIZE=n) …
which should read  
read(…,…,SIZE=n,ADVANCE='NO') …	No compiler error detected.	
2837	pgf90	A merge operation causes a false              –Mbounds error	0: Subscript out of range for … 	
2839	pgcc	pgcc generates assembly output with –Kpic that is invalid.
cvtsi2ss $88,%xmm0	pgcc xx.c -fastsse -Kpic 

Error: suffix or operands invalid for `cvtsi2ss'

2847	ALL	__STDC__,__PGI need to be undefined 
pgxx  –U__STDC__  fails	__STDC__, __PGI and other symbols remain defined.	
2849	pgf90 pgf77	programs compiled –I8 fail unless  IO specifiers like IOSTAT are declared integer*4 or logical*4	compiler error messages	
2851	pgf90	-Mstandard was not catching our f90 extension of allowing derived type members to be allocatable, as an error.	pgf90 –Mstandard x.f does not report errors.	
2855	pgf90pgCC	pgf90 –U  does not work. related to 2847	pgf90 –U xxx does not become –undef xxx	
2865	PgCC	s=NULL;cout<<s  fails	program fails	
2867	PgCC	pgCC
--one_instantiation_per_object p.cc	C++ prelinker: executing       —inf-loops	
2872	pgf90	array constructors with 2 or more implied DOs  using the same DO index fails.  	x = (/ (x(j), j=Nx/2+1, Nx), (x(j), j=1, Nx/2) /) fails	
2873	pgf90	function declared in the interface block of a module and later,in a program contained in that module that is passed as an ARGUMENT to another subprogram	Internal compiler error. Errors in Lowering       	
2874	pgf90	SCALE function in f90 gives wrong results for double precision.	wrong answers 	
2881	pgf90	DOT_PRODUCT(CONJG(a(k1:k2)),u(i(k1:k2))) returns wrong results	wrong results	
2882	pgCC	C++ ignores ‘pragma omp’ in templates	 1 thread in omp areas
	
2902	pgf90	-fastsse causes internal compiler error	ICE:replace_invar: nonzero subsc stride       	
2907	pgf90	matmul gives wrong complx answers 	wrong answers	
2912	pgf90	program causes Internal Compiler Error	ICE. exp_ref:IM_BASE op#2 not based sym     	
2916	pgf77 pgf90	OPEN(…,…,…,CONVERT='NATIVE') should ignore -byteswapio	results swapped when they shouldn’t be.	

3.6	PGCC C and C++ Compiler Notes
The Rogue Wave Standard Template Library has been replaced with STLport, version 4.5.  Rogue Wave is no longer supported. Users should look at the STLport license for any usage issues.
3.7	The PGI CDK 4.1 and libpthread
Previous releases of the PGI CDK Linux compiler products have included a customized version of libpthread.so called libpgthread.so. The purpose of this library is to give the user more thread stack space to run OpenMP and    –Mconcur compiled programs. With Release 8.0 Red Hat and equivalent releases, we are seeing libpthread.so and libpthread.a with ‘re-sizeable’ thread stack areas. In these cases 
1.	The filename $PGI/linux86/lib/libpghtread.so is a soft link to /usr/lib/libpthread.so.
2.	Instead of ‘setenv MPSTKZ 256M’, for example, to increase the libpgthread.so thread stack area, the Linux system call ‘limit stacksize 256M’ will now apply to thread stacks.

3.8	The PGI CDK 4.1 and glibc
Release 4.1 of the PGI CDK compilers and tools are built and validated under both the Linux 2.2.10 through 2.4.x kernels.  Distributions of Linux, from Red Hat 6.0 to 9.0 and SuSE 6.1 to 9.1, incorporate revision 2.2.10 or greater of the Linux kernel and glibc2.1.x or greater.  If you are using a version of Linux that is supported by the 4.1 CDK release, the PGI installation script will automatically detect it.  Your installation will be modified as appropriate for these systems..

3.9	The PGI BLAS LAPACK Libs
Precompiled versions of the BLAS and LAPACK math libraries are included for all supported Linux systems in the files $PGI/linux86/lib/libblas.a and $PGI/linux86/lib/liblapack.a.  These can be linked in to your applications by simply placing the –llapack –lblas options on the link line:
       % pgf77 myprog.F -lblas -llapack
Note that these libraries are compiled with switches that are relatively optimal but fully portable across the various IA32 architectures.  In particular, they do not take advantage of SSE/SSE2 instructions, or prefetch instructions.  If you would like to rebuild libblas.a and liblapack.a on a CPU that supports SSE/SSE2 type instructions, PGI recommends using the following options:
       -fast –pc 64 –Mvect=sse –Mcache_align -Kieee
NOTE: slmach.f and dlmach.f must be compiled –O0!
If you would like to rebuild libblas.a and liblapack.a on an AMD Athlon, PGI recommends using the following options:
       -fast –pc 64 –Mvect=prefetch –Kieee
As on the Pentium III, slmach.f and dlmach.f must be compiled –O0.

STLport license for any usage issues.
3.10	OpenMP Tutorial
A self-guided online tutorial is available to help you become familiar with how OpenMP parallelization directives.  In particular, the tutorial takes the user step by step through the process of parallelizing the NAS FT benchmark using OpenMP directives.  The tutorial can be found at:
    ftp://ftp.pgroup.com/pub/SMP
You can download this file using a web browser, and unpack the file using the following commands:
       % gunzip fftpde.tar.gz
       % tar xvf fftpde.tar
Change directories to the fftpde sub-directory, and follow the instructions in the README file.

3.11	Debugging with PGDBG 
3.11.1  PGDBG 4.1  Features
Note:  Most of this information was also in the PGI CDK 4.0 Release Notes. It is present here and will be part of the PGDBG User’s Guide in a future release, and will not be removed from the release notes until it has. PGDBG has had a number of corrections, and it now is supported under ssh, but beyond that the features have not changed from 4.0.
PGDBG 4.1 can debug SMP OpenMP (or Linux pthread) programs, as well as multiprocess cluster programs executed via mpirun.  The PGI license file restricts the total number of threads and processes that PGDBG will debug.
PGDBG 4.1 supports ssh as well as rsh . A new environment variable, PGRSH, should be set to ssh or rsh, to indicate the communication needed.
PGDBG's parallel debug capabilities are extensively documented at http://www.pgroup.com/docs.htm or at $PGI/doc/index.htm.  This documentation is intended to supplement Chapter 15 of the PGI User’s Guide.
The following enhancements are included in PGDBG 4.1:
•	Combined Multi-process and Multi-thread Support
•	SSH support 
-	be sure to ‘export PGRSH=ssh’
•	Multi-process Support
-	Process Ids obtained from mpirun
-	Same source and debug info used for all processes
-	full process control
-	process grouping
-	informative messages regarding  state and location

•	Process Control 
-	concise control of groups of processes
-	process synchronization 
-	configurable process stop and wait modes
-	serial, and process-only debug modes
•	OpenMP & Linuxthread Support
-	threads identified by OpenMP logical CPU ID
-	automatic thread detection and attach
-	full thread control in parallel regions
-	thread grouping
-	line level debugging preserved when a thread
-	enters a parallel region 
-	enters a serial region
-	hits an OpenMP barrier 
-	hits an OpenMP synchronize statement
-	enters an OpenMP sections program section
-	informative messages regarding thread state and location

•	Thread Control 
-	concise control of groups of threads
-	thread synchronization
-	configurable thread stop and wait modes
-	serial, and threads-only debug modes
•	GUI Enhancements 
-	Thread sub-window.  Lists each thread by its logical CPU ID.  Displays for each thread its state and stop location.  Threads are grouped by parent process.
-	Program I/O sub-window.  Pops up automatically when program prints to stdout.  The program I/O sub-window can also be raised from the Window menu.
-	Output written to stdout by the process being debugged is no longer block buffered.  
-	process grid.  Displays each process as a color coded button in a grid. Click on a grid element to refresh the GUI in the scope of that process. Each grid element is numbered with the process's logical  ID.
-	Process grouping.  Control processes in groups
-	Thread grid.  Displays each thread as a color coded button in a grid. Click on a grid element to refresh the GUI in the scope of that thread. Each grid element is numbered with the thread's logical CPU ID.
-	Thread grouping.  Control threads in groups.
•	Other Enhancements
-	Better support for Fortran arrays and pointers
3.11.2	PGDBG 4.1 Technical Information
Here are a number of details not documented in the PGDBG User’s Guide.
3.11.2.1	Threads and Signals
PGDBG intercepts all signals sent to any of the threads in a multi-threaded program, and passes them on according to that signal's disposition maintained by PGDBG (see the catch, ignore commands).
If a thread runs into a busy loop, or if the program runs into deadlock, control-C over the debugging command line to interrupt the threads. This causes SIGINT to be sent to all threads.  By default PGDBG does not relay SIGINT to any of the threads, so in most cases program behavior is not affected.
Sending a SIGINT (control-C) to a program while it is in the middle of initializing its threads (calling   omp_set_num_threads(), or entering a parallel region ) may kill some of the threads if the signal is sent before each  thread is fully initialized.  Avoid sending SIGINT in these situations.  When the number of threads employed by a program is large, thread initialization may take a while.
3.11.2.2	Signals Used by Internally by PGDBG
SIGTRAP indicates a breakpoint has been hit.  A message is displayed whenever a thread hits a breakpoint. SIGSTOP is used internally by PGDBG.  Its use is mostly invisible to the user. Changing the disposition of these signals in PGDBG will result in undefined behavior.
Reserved Signals: On linux86, the thread library uses SIGRT1, SIGRT3 to communicate among threads internally. In the absence of real-time signals in the kernel, SIGUSR1, SIGUSR2 are used. Changing the disposition of these signals in PGDBG will result in undefined behavior.
3.11.3	Scoping
Nested Subroutines 
To reference a nested subroutine you must qualify its name with the name of its enclosing function using the scoping operator @.
For example:
subroutine subtest (ndim)
integer(4), intent(in) :: ndim
integer, dimension(ndim) :: ijk
call subsubtest ()
contains
    subroutine  subsubtest ()
    integer :: I
    i=9
    ijk(1) = 1
    end subroutine subsubtest
    subroutine  subsubtest2 ()
    ijk(1) = 1
    end subroutine subsubtest2
end subroutine subtest           
program testscope
integer(4), parameter :: ndim = 4
call subtest (ndim)
end program testscope

pgdbg> break subtest@subsubtest
breakpoint set at: subsubtest line: 8 in "ex.f90" address: 0x80494091
pgdbg> names subtest@subsubtest 
i = 0
pgdbg> decls subtest@subsubtest 
arguments:
variables:
integer*4 i;
pgdbg> whereis subsubtest
function:       "ex.f90"@subtest@subsubtest    
Fortran 90 Modules 
To access a member mm of a Fortran 90 module M you must qualify mm
with M using the scoping operator @.  If the current scope is M the qualification can be omitted.

For example:

module M
implicit none
real mm
contains
subroutine stub
print *,mm
end subroutine stub
end module M
program test
use M
implicit none
call stub()
print *,mm
end program test              

pgdbg> Stopped at 0x80494e3, function MAIN, file M.f90, line 13
#13:       call stub()
pgdbg> which mm
"M.f90"@m@mm
pgdbg> print "M.f90"@m@mm
0
pgdbg> names m
mm = 0
stub = "M.f90"@m@stub
pgdbg> decls m
real*4 mm;
subroutine stub();
pgdbg> print m@mm
0
pgdbg> break stub
breakpoint set at: stub line:6 in "M.f90" address: 0x8049446      1
pgdbg> c
Stopped at 0x8049446, function stub, file M.f90, line 6
Warning: Source file M.f90 has been modified more recently than object file
#6:           print *,mm
pgdbg> print mm
0
pgdbg>    
3.11.4	Lexical Blocks 
Line numbers are used to name lexical blocks. The line number of the first instruction contained by a lexical block indicates the start scope of the lexical block.  
Below variable var is declared in the lexical block starting at line 5.  The lexical block has the unique name "lex.c"@main@5.  The variable var declared in "lex.c"@main@5 has the unique name "lex.c"@main@5@var.
For Example:
lex.c:
main()
{
    int var = 0;
    {
        int var = 1;
        printf("var %d\n",var);
    }
    printf("var %d\n",var)
}

pgdbg> n
Stopped at 0x8048b10, function main, file
/home/pete/pgdbg/bugs/workon3/ctest/lex.c, line 6
#6:         printf("var %d\n",var);
pgdbg> print var
1
pgdbg> which var
"lex.c"@main@5@var
pgdbg> whereis var
variable:       "lex.c"@main@var
variable:       "lex.c"@main@5@var
pgdbg> names "lex.c"@main@5
var = 1
3.11.5	Private Variables 
PGDBG understands private variables with some restrictions. In particular, inspecting private variables while debugging FORTRAN programs is not  supported. 
Private variables in C must be declared in the enclosing lexical block of the parallel region in order for them to be visible using PGDBG.

For example:
{
    #pragma omp parallel    
    {
        int i;
        ...
        /* i is private to 'this' thread */
        ...
    }
}  
In the above case, i would be visible inside PGDBG for each thread. However, in the following example, i is not visible inside PGDBG:
{
    int i;
    #pragma omp parallel private(i)  
    {
        ...
        /* i is private to 'this' thread 
           but not visible within PGDBG */
        ...
    }
}  

A private variable of a Thread A is accessed by switching the current thread to A, and by using the name (qualified if necessary) of the private variable.
3.11.6	Graphical User Interface (GUI) Notes
3.11.6.1	Setting  the Font
Use the xlsfonts command to list all fonts installed on your system, then choose one you like.  For this example, we choose a sony font that is completely specified by the following string:
-sony-fixed-medium-r-normal—24-230-75-75-c-120-iso8859-1
There are two ways to set the font that your PGDBG GUI uses.
1. Use your .Xresources file:
Xpgdbg*font : <chosen font>
pgdbg*font : <chosen font>
For example:
pgdbg*font : -sony-fixed-medium-r-normal--24-230-75-75-c-120-iso8859-1  
You will have to merge these changes into your X environment for them to take effect. You can use the following command:
       % xrdb -merge $HOME/.Xresources
2. Use the command line options : fn <font>.  For example:
% pgdbg -fn -sony-fixed-medium-r-normal--0-0-100-100-c-0-jisx0201.1976-0...
3.11.6.2	Control-C from GUI
The active window must be the command window (upper window) where the PGDBG prompt appears for control-C to interrupt the program being debugged. interrupt the program being debugged.
3.11.6.3	Shared Object Files
PGDBG supports debugging of dynamically linked executables that reference shared object files created using the compilers.  If the executable being debugged is dynamically linked, PGDBG will report when each shared object is loaded and/or unloaded. 
For example:
  pgdbg> ...
  pgdbg> n
  Stopped at 0x8048bee, function main, file   
  dynload.c, line 36
  #36: handle = dlopen("libpetesSO2.so",RTLD_NOW);
  pgdbg> n
  libpetesSO2.so loaded by ld-linux.so.2.
  Stopped at 0x8048c31, function main, file
  dynload.c, line 41
  #41:       if (handle){
  pgdbg> n
  Stopped at 0x8048c37, function main, file
  dynload.c, line 42
  #42:         dlclose(handle);
  pgdbg> n
  libpetesSO2.so unloaded by ld-linux.so.2.
  Stopped at 0x8048c42, function main, file
  dynload.c, line 45
  #45:     }
  pgdbg> ...
The global symbols defined by a dynamically linked shared object are visible during a PGDBG debug session.  These symbols are currently available only without type and line number information. The machine level PGDBG commands (breaki, dump, hwatch, disasm, etc) are useful for inspecting these symbols.  Each symbol is available with respect to the load status of its defining shared object.  
For example, dynamically-linkable Position Independent Code (PIC) is implemented using a Procedure Linkage Table (PLT) and Global Offset Table (GOT).  Each PIC function is bound lazily at run-time.  If a function has not been linked dynamically, PGDBG reports the address of its PLT entry as its address. If a function has been linked dynamically, PGDBG reports the virtual address of the function itself.  So, PGDBG reports the current or “effective” address of symbols with respect to dynamic linking and loading.  PGDBG treats global symbols defined in shared objects in a similar way.  The address of a global variable may be the address of its GOT entry or an absolute address, depending in part on its load status.

4	PGHPF 4.1

PGHPF 4.1 is PGI’s native (HPF-to-assembly code) High Performance Fortran compiler for IA-32 processor-based systems.  All features of Full HPF 1.1 and Fortran 90 are supported, with the few exceptions noted in this document.  In addition, several new HPF 2.0 and HPF/JA features have been added in this release.  Section 4.1 below for a list of features, which is a continuation of the 4.1 release information.  If you encounter any feature that is not supported, and not listed in section 4.3, Restrictions, please consider it a bug and report it to PGI at the e-mail address trs@pgroup.com.
4.1	Summary of Changes
The following features have been added to PGHPF 4.1 for IA-32 systems:
•	HPF/JA feature - support for REDUCTION type in INDEPENDENT clauses, including the new reduction operators FIRSTMAX, LASTMAX, FIRSTMIN, and LASTMIN and relaxation of the restrictions on allowable forms of references to reduction variables when the reduction type is specified. 
•	HPF 2.0 feature – support for the approved extension form of the ON directive, restricted to the body of an INDEPENDENT loop
•	HPF 2.0 feature – the HPF library procedures SORT_UP and SORT_DOWN are now supported
•	Reductions in nested INDEPENDENT loops are now supported
•	Constraints on the order of HPF mapping directives have been eliminated
•	Scaling analysis of HPF programs using PGPROF 4.1 – see section 3.12.1 for more information on this feature
4.2	Restrictions
This section lists Fortran 90 and HPF features that are not supported in PGHPF 4.1.
Type	Restriction	
Fortran 90 Pointers	•      Objects with the POINTER attribute cannot be DYNAMIC
•	Objects with the TARGET attribute cannot have CYCLIC or CYCLIC(N) distributions.  It may not be possible to detect this at compile-time in all cases, for example when a CYCLIC actual argument is passed to a dummy with the TARGET attribute
•	A scalar POINTER cannot be associated with a distributed array element.  For example

      integer, pointer :: p
      integer, target :: a(10),b(10)
!hpf$ distribute (block) :: a
      p=> a(1) ! unsupported
      p => b(1) ! supported
      end
•	A POINTER dummy variable cannot be used to declare other variables such as automatic arrays using the lbound(), ubound() and size() intrinsics.  For example:

 subroutine sub(p)
 integer, pointer, dimensions(:,:) :: p
 integer, dimension(lbound(p,1):  &
+ubound(p,1),size(p,2)) :: a ! does not work
	
Fortran 90 Derived Types	The DATA statement does not work with arrays of derived type.  As a work-around, use entity-style initialization.	
Named Constants	Named multi-dimensional array constants cannot be subscripted to yield a constant value.  For example, given the declaration:

      INTEGER, PARAMETER, DIMENSION(2,2) :: 
     &   X = RESHAPE((/1,2,3,4/),(/2,2/))
the following will not work:
   
      INTEGER, PARAMETER :: Y = X(1,2) ! Will not work
Because of this restriction, named multi-dimensional array constants cannot be used in:
•	Values in CASE statements
•	KIND parameters in declaration statements
•	KIND arguments to intrinsics
Initial values in parameter statements or type declaration statements	
HPF Library	The HPF_LIBRARY routines GRADE_UP, GRADE_DOWN, SORT_UP and SORT_DOWN require a DIM argument.  These routines do not support cyclic distributions of the selected dimension.	
PURE Procedures	The PGHPF 4.1 implementation of PURE conforms to the HPF 2.0 language specification, with the following exception:  in PURE subroutines PGHPF will not generate any communication for distributed COMMON variables or distributed MODULE variables.  The user is advised to pass distributed COMMON variables as arguments to a PURE subroutine, or use non-distributed COMMON variables.	
Optional Arguments	An F90 optional argument cannot be used as an align-target for any variable that is not also optional.  If an alignee is present, then the align-target must also be present.	
DISTRIBUTE and ALIGN	The following compile-time warning message:
PGHPF-W-301 – Non-replicated mapping for character/struct/union array, char_table, ignored (file.F: lineno)
indicates that PGHPF 4.1 ignores the distribution directives applied to character arrays, arrays subject to SEQUENCE directives, and NAMELIST arrays.	
INDEPENDENT loops	At present, only INDEPENDENT loops containing FORTRAN 77 constructs can be parallelized. In particular, the presence of array assignments, WHERE statements, FORALL statements, and ALLOCATE statements will eliminate INDEPENDENT loops from consideration for parallelization.	

5	 Contacting PGI & Online 
Documentation

You can contact us at the following address:
The Portland Group Compiler Technology  STMicroelectronics,Inc.
9150 SW Pioneer Court
Suite H
Wilsonville, OR  97070
Or contact us electronically using any of the following means:
Fax:	 +1-503-682-2637
Sales:	 sales@pgroup.com
Support:	 trs@pgroup.com
WWW:	 http://www.pgroup.com 	
Online documentation is available by pointing your browser at either your local copy of the documentation:
file:$PGI/doc/index.htm
or at our Web site:
http://www.pgroup.com