Running Batch Jobs On MPC:

When you login to mpc.uci.edu, you are connected to the first node which is also called the head node.   The remaining nodes are called the slave nodes, or, compute nodes.

The head node is meant for interactive login activity (like editing,  debugging, compiling programs, etc.) while the rest of the nodes, the slave nodes, are used for CPU intensive long running batch jobs.

The MPC System is made up of Full-time nodes, Part-time PC night Lab nodes, and Private nodes.

Full-time nodes are compute nodes dedicated 100% of the time to MPC, while Part-time nodes are available to MPC users during the night.    These Part-time  nodes are made up from PC labs around the UCI campus.

Private Nodes:  These nodes are private and are not available to the general MPC user community.    We run these nodes for various research teams on Campus in exchange for hardware contribution to MPC.

In 2001, we began a project at UCI/NACS to harvest idle computer resources on campus.    The idea was to see if we could get student PC labs which were not being used during the night to essentially become Beowulf compute nodes.

A lot of effort went into this and it worked.    Our PC student labs run Windows NT during the day for student use, and Linux during the night as an extension of our MPC Beowulf cluster.     The PC labs automatically reboot from Windows to Linux during the night without any human intervention.     Once the PC labs convert, jobs sitting on MPC queues are then released to run.

All this is transparent to the end-user:    Users simply submit jobs to the queues on MPC.    MPC then goes out and farms idle compute nodes and assigns the jobs to those nodes via PBS.
 


PBS:

PBS (Portable Batch System) qsub is used to submit jobs on MPC from the head node.    You should never directly log-in to any of the slave nodes.   Qsub balances the load on MPC by finding the least busy slave nodes and assigning your job to those nodes automatically.    If you directly connect to the slave nodes, the load will then become unbalanced which is not a good thing :-).

An example of how to construct and submit a PBS batch job is explained below.
 
 

Queues:

There are currently various queues defined on MPC.    Queues can be either shared or exclusive, full-time or part-time:
 

• Share queues means that your job runs in a shared environment, that is, your job runs with other jobs on the same nodes competing for resources like CPU time for example.

• Exclusive queues are queues in which your job does NOT share any resources(*) and therefore have exclusive use of the resources on the nodes.    (*) One resource is shared, which is the node-to-node communication bandwidth.
• Full-time queues point to MPC's full-time dedicated nodes, that is, nodes which are available to users 24/7 (24 hours a day, 7 days a week).
• Part-time queues point to MPC's night Lab nodes.     Our walk-in PC labs are used by students during the day as Window PC's and then become Linux Beowulf MPC nodes during the night when the labs are closed.      We are basically harvesting idle CPU's during the night and as such, these nodes are only available during specific times.     You  may  submit jobs to these queues during the day.  The queues will not run the jobs until night-time when the PC's become available.
• Private Nodes: These nodes are private and are not available to the general MPC user community.    We run these nodes for various research teams on Campus in exchange for hardware contribution to MPC.


Share Queues / Available Fulltime (24/7):

 

Queue Name	CPU Priority	Max Hours	Max  Nodes	CPU's per Node	Node Type
share	100%	24	4	1	Athlon 1.33GHz CPU, 512MB DDR Memory
share-medium	70%	72	4	1	Athlon 1.33GHz CPU, 512MB DDR Memory
share-low	40%	168	4	1	Athlon 1.33GHz CPU, 512MB DDR Memory

Exclusive Queues / Available Fulltime (24/7):

 

Queue Name	Max Hours	Max  Nodes	CPU's per Node	Node Type
athlon	24	10	1	Athlon 1.33GHz CPU, 512MB DDR Memory
opteron	24	15	2	AMD Opteron 240 1.4GHz CPU, 1GB DDR Memory. MSI K8D Motherboard

 

Exclusive PC-Lab Queues /  Available Part-time:

 

Queue Name	Hours of Operation	Max Nodes	CPU's per Node	Node Type
pc-mstb	10pm-7am Monday-Saturday) Sunday all day (24 hours)	50	1	Intel P4 2.5Ghz CPU, 512MB DDR Memory, 40GB HD
pc-msta	10pm-7am Monday-Saturday) Sunday all day (24 hours)	46	1	Intel P4 1.0Ghz CPU, 512MB Memory, 40GB HD
pc-itca	10pm-7am Monday-Saturday) Sunday all day (24 hours)	32	1	Intel P3 650Mhz 128MB Memory, 15GB HD
pc-itcb	10pm-7am Monday-Saturday) Sunday all day (24 hours)	20	1	Intel P4 1.7Ghz, 256MB Memory, 19GB HD

Communication Speed Note:  The PC MSB(a/b) LABs are currently running at 10Mbps.     We are planning on upgrading both labs to Gigabit & 100MB connection by December of 03'.
 
 
 

Private Nodes:

Queue Name	Owner of Nodes Contact Info	Max Nodes	CPU's per Node	Node Type
apep	Steven Lee Advanced Power & Energy Program	4	2	AMD Opteron 240 1.4GHz CPU, 1GB DDR Memory. MSI K8D Motherboard
cfd	Prof. Feng Liu Mechanical & Aerospace Engineering	10	2	AMD Opteron 240 1.4GHz CPU, 1GB DDR Memory. MSI K8D Motherboard
mathbio	Prof.  Qing Nie Mathematics	12	2	AMD Opteron 240 1.4GHz CPU, 1GB DDR Memory. MSI K8D Motherboard
chem	Prof. Athan J. Shaka Chemistry	7	2	AMD Opteron 240 1.4GHz CPU, 512MB DDR Memory. MSI K8D Motherboard
air	Prof. Donald Dabdub Mechanical & Aerospace Engineering	10	2	AMD Opteron 244 1.8GHz CPU, 1GB DDR Memory. MSI K8D Motherboard
tw1700	Prof.  Douglas J  Tobias   Chemistry Prof.  Stephen H. White Physiology & Biophysics	24	1	AMD Athlon 1700 512 MB DDR Memory All nodes Remotely Located

NACS has an excellent program in place in which we will run your compute nodes in exchange for 25% of your hardware or resources.

This is how it works, say you buy 4 compute nodes.    You get full ulimited (no time restrictions) access to your 3 compute nodes while the other node goes back into the MPC "common pool" of compute nodes.    All users on MPC including yourself, will have access to the pool of common nodes.    So, you basically have access to your nodes plus all of the nodes in the common pool of nodes on MPC.

In exchange for your hardware contribution, you get all of the following for free:

• System Administration
• Gigabit switch connectivity to your nodes
• Disk space ( we will have approximately 1.4TB soon )
• Backups ( nobody cares about backups until you lose data :-)
• Physical hosting of your nodes ( racks, colling, etc )
• UPS power protected electrical feed ( power spikes can shorten computers )
• Access to Software (like Portland Group compilers, etc)
• Immediate usage of your hardware ( all nodes are automatically setup ).


If you are intested in this, please contact Joseph A. Farran at jfarran@uci.edu.
 


Here is a quick explanation of what the queue headers mean:
 

• Queue Name is the name you specify in qsub when submitting your job.

• CPU Priority is the priority your job will have based on the share Queue Name you select.    This only applies to share queues.    Exclusive queues always have 100% user CPU priority.

• Max Hours is the maximum amount of hours your job may run on this queue.    If  your job exceeds this time, your job will automatically be killed and the next job waiting in the queue will run next.

• Max Nodes is the maximum number of nodes you may request for the given Queue Name.
• CPU's per Node correspond to the number of CPU's per node.    Some nodes have 1 CPU, others may have 2 or more.
• Node Type is the hardware configuration of that node.

 

Compiling, Running & Monitoring Parallel Jobs:

For Parallel  programs that use MPI,  you will want to compile with the appropriate MPI Script.    These scripts will compile your code using The Portland Parallel Compilers and link with the appropriate MPI libraries.    For example, if your parallel code is written in C, you will want to use mpicc.    The list is as follows:
 

• mpicc        For C programs.
• mpiCC      For C++ programs.
• mpif77       For FORTRAN 77 programs.
• mpif90       For Fortran 90 programs.

Hardware Flags:    Since MPC has different CPU hardware nodes available, you will want to use the "-tp" option when compiling to Optimize your code for the intended hardware.    The "-tp" options are as follows:
 

• -tp=k8-32|k8-64|k7|p5|p6|piii|piv|p7|px
•  Choose target processor type:

•     k8-32    AMD K8 Processor 32-bit mode
•     k8-64    AMD K8 Processor 64-bit mode
•     k7           AMD Athlon Processor
•     p5              Intel Pentium
•     p6              Intel P6 Pentium (Pentium Pro, II, III)
•     piii            Intel Pentium III
•     piv             Intel Pentium IV
•     p7              Intel P7 Pentium IV
•     px              Intel generic Pentium


The following will illustrate how to compile, submit, and monitor a parallel job on MPC.    The demo account has a simple parallel program which we can use (~demo/hello.c) for this illustration.

On MPC copy the following 2 files into your account as follows:

% cp  ~demo/share.pbs   .
% cp  ~demo/hello.c  .

Here is a quick explanation of the share.pbs script lines:

 

Entry	What is Does
#PBS  -N "Test"	The name of this job "Test"
#PBS  -l nodes=4:share	Requesting 4 shared nodes
#PBS  -q share	Run in the "share" Queue
mpirun -np 4 [...]	Number of processes the job will take (np).


The parallel "hello.c" program will report back on which node it is running along with "Hello World" message.    First, we need to compile the program:
 

% mpicc  hello.c  -o hello        ( or: pgcc hello.c -o hello -lmpich )

Now edit the "cd" line in "share.pbs" to reflect your working directory and then submit the job to the batch system:

% qsub  share.pbs

If everything went well, you will get back a job ID assigned to your job.    To check on the status of your job, enter:

% qstat -a
Once your jobs completes, you can view your output in log file.    Note, the job may finish before you get you see it in the qstat output.

 If you wish to see which nodes your job is running on,  enter:

 
% qstat -rn


The options "-rn" says to display running jobs "r" and the nodes "n" associated with the jobs.    Here is a sample output:

 
19952.mpc.uci.e  jfarran share  Test  8641  4  -- --  72:00 R 00:04
  node5/1+node6/1+node4/1+node11/0

Note the "4" marked in red above - that represents the number of nodes assigned to this job (job # 19952).    The assigned nodes that we were given is listed in purple.

Once you know the node(s) your job is running on, you can then monitor the job in greater detail by looking at MPC Cluster Load  and viewing the metrics of the nodes in question.    For example, you can see if your job is bound by disk / cpu / communication / memory / etc.

If something goes wrong and you wish to stop/delete your job from the queue, use:
 

% qdel   <job_id>


The <job_id> is the job Id you get from qstat -a.    In the example above, the job id would be 19952.
 
 
 
 

Serial Jobs:
 
For Serial Jobs, everything is basically the same as above, except that we do not use MPI  since this is not a parallel program and we only request 1 node at a time.

Here is an example of a Serial pbs job which can be found in ~demo/serial.pbs.
 

Entry	What is Does
#PBS -N "Test"	The name of this job  "Test".
#PBS -q  share	Run in the "share" queue.

For compiling your Serial program, use the appropriate compiler which can be found in the software section.    For this example, we do:
 

% pgcc  hello.c   -o hello
Now submit the job as before with qsub:
 
% qsub   serial.pbs
To monitor the job, you use the same commands described above.