Helix Systems

Environment Modules are a convenient way for users to dynamically set paths, library paths, and other required environment variables for an application. Instead of typing shell-dependent commands, for example, a user could now type:

module load ThisApp

to set up all environment variables for the program ThisApp, and:

module unload ThisApp

to unset all those environment variables.

Almost all the scientific applications, libraries, and interpreters such as Perl, Python and Java on Helix & Biowulf now use modules. Details are on the application webpages.

If you are not familiar with the use of modules, please see http://helix.nih.gov/Applications/modules.html for an introduction.

Please contact staff@helix.nih.gov with any questions.


12/5/2012

This email is for Biowulf users who have their /data directories on the /gs2 filesystem. If you use the "gpfs" property for your 'qsub' or 'swarm' jobs, this means you.

Biowulf staff will have to take the gs2 filesystem OFFLINE to repair filesystem damage incurred last week.

This means that when the filesystem is taken offline, any jobs currently accessing that filesystem will be terminated, and that jobs submitted requiring that filesystem will be rejected.

The software vendor informs us that it will take anywhere between 8 and 48 hours to repair the filesystem.

Jobs accessing or submitting to the /gs2 filesystem will be terminated starting TODAY at 10 am. Email will be sent to biowulf users when /gs2 is again available.

We apologize for this disruption to your work, but delaying this repair could have serious consequences for the integrity of data on /gs2.




11/28/2012

The Helix/Biowulf staff will present a series of classes in Jan 2013 Classes are free but registration is required. Helix/Biowulf users will have priority enrollment, and new users are highly encouraged to attend.

Introduction to Linux ---------------------- http://training.cit.nih.gov/?SS880 Date & Time: Wed, Jan 9, 9 am - 4 pm Location: Bldg 12A, Rm B51. This class is intended as a starting point for individuals new to Linux and UNIX. The class will center on basic UNIX/Linux concepts: logging in, navigating the file system, commands for interacting with files, running and viewing processes, checking disk space and other common tasks. The class will also cover the use of some services specific to Helix/Biowulf usage.

Bash Shell Scripting for Helix and Biowulf ------------------------------------------ http://training.cit.nih.gov/?SS890 Date & Time: Wed, Jan 16, 10 am - 4 pm Location: Bldg 12A, Rm B51. The default shell on many Linux systems is bash. Bash shell scripting provides a method for automating common tasks on Linux systems (such as Helix and Biowulf), including transferring and parsing files, creating qsub and swarm scripts, pipelining tasks, and monitoring jobs. This class will give a hands-on tutorial on how to create and use bash shell scripts in a Linux environment.

NIH Biowulf Cluster: Scientific Computing ----------------------------------------- http://training.cit.nih.gov/?SS610 Date & Time: Wed, Jan 23, 9 am - 4 pm Location: Bldg 12A, Rm B51 Morning: Introduction to the Biowulf Linux cluster, cluster concepts, accounts, logging in, storage options, interactive vs. batch jobs, how to set up and submit a simple batch job, batch queues, available software, and job monitoring. Afternoon: Hardware and network configuration, types of nodes, selection of nodes using properties, system software, parallel programs, programming tools.




12/5/2012

The /gs2 filesystem is again available and users who submit qsub or swarm jobs with the "gpfs" property may resume doing so.

File system repair took about 8 hours with damage believed to be limited to one user's directory entry for a single file. Biowulf staff will continue analysis of repair log files to confirm the scope of damage.




12/11/2012

The Biowulf staff is pleased to announce the addition of 24 large memory (256 GB) nodes to the cluster. These nodes will be for jobs that require more than 72 GB, and in contrast to most nodes in the cluster will be shareable by more than one job.

To allocate these nodes to your job, use the "mem=" option to the 'qsub' command, for example:

qsub -l nodes=1,mem=128gb myjob.bat

mem must be equal or greater than 72gb and less than or equal 250gb. Jobs requesting more than 250gb will continue to allocate the single 512GB node.

Multi-process or multi-threaded jobs should use the "ncpus=" option to 'qsub', for example:

qsub -l nodes=1,mem=192gb,ncpus=24 myjob.bat

ncpus must be 32 or less.

The 'freen' command now reports the largest block of available memory on the 256GB nodes:

-------------------- 256 GB nodes ---------- Available memory: Min - 72 GB, Max - 250 GB




3/27/2013

The Helix Systems staff maintains a small collection of 1000 genomes data (vcf files) in /fdb/1000genomes/. We are considering maintaining a copy of the BAM files (alignments) from 1000 Genomes.

These files are very large (currently 132 TB total, and growing), and since disk space is always in short supply, we need to seriously consider their value to the Helix/Biowulf user community before downloading them.

* Have you ever used the 1000 Genomes BAM files? * Do you currently have a copy of any of the 1000 Genomes data on your Helix/Biowulf account? * Do you plan to use the 1000 Genomes BAM files? If so, when?

Thanks for your input! Helix Systems Staff.




4/17/2013

For NIH Helix & Biowulf users who have Linux or Mac systems in their labs, there is a new file transfer program available for the Helix Systems.

'bbcp' is a high-performance version of scp which can provide significantly increased file transfer performance over scp. You can use 'bbcp' on Biowulf or Helix to transfer files to your local Linux or Mac system; you can also download a version of 'bbcp' for your local system and use it there.

For details on using 'bbcp' on Biowulf see the section entitled "Commandline File Transfer: Macs & Unix/Linux: bbcp" at http://helix.nih.gov/Documentation/transfer.html




3/12/2013

The Helix/Biowulf staff will present a series of classes in April 2013. Classes are free but registration is required. Helix/Biowulf users will have priority enrollment, and new users are highly encouraged to attend.

Introduction to Linux ---------------------- http://training.cit.nih.gov/class_details.aspx?cId=NIHCIT-SS880 Date & Time: Wed, Apr 10, 9 am - 4 pm Location: Bldg 12A, Rm B51. This class is intended as a starting point for individuals new to Linux and UNIX. The class will center on basic UNIX/Linux concepts: logging in, navigating the file system, commands for interacting with files, running and viewing processes, checking disk space and other common tasks. The class will also cover the use of some services specific to Helix/Biowulf usage.

Bash Shell Scripting for Helix and Biowulf ------------------------------------------ http://training.cit.nih.gov/class_details.aspx?cId=NIHCIT-SS890 Date & Time: Wed, Apr 17, 10 am - 4 pm Location: Bldg 12A, Rm B51. The default shell on many Linux systems is bash. Bash shell scripting provides a method for automating common tasks on Linux systems (such as Helix and Biowulf), including transferring and parsing files, creating qsub and swarm scripts, pipelining tasks, and monitoring jobs. This class will give a hands-on tutorial on how to create and use bash shell scripts in a Linux environment.

NIH Biowulf Cluster: Scientific Computing ----------------------------------------- http://training.cit.nih.gov/class_details.aspx?cId=NIHCIT-SS610 Date & Time: Wed, Apr 24, 9 am - 4 pm Location: Bldg 12A, Rm B51 Morning: Introduction to the Biowulf Linux cluster, cluster concepts, accounts, logging in, storage options, interactive vs. batch jobs, how to set up and submit a simple batch job, batch queues, available software, and job monitoring. Afternoon: Hardware and network configuration, types of nodes, selection of nodes using properties, system software, parallel programs, programming tools.




3/18/2013

The Helix Systems staff is pleased to announce a utility tool that lets users monitor their usage of CPU and storage on Biowulf and Helix.

To view your usage, go to https://helixweb.nih.gov/usage/

You will need to authenticate with your NIH login username and password. This utility is only available within the NIH network.

The website also displays papers that cited Helix and Biowulf during this period. If you have additional papers that are not listed, please let us know by sending email to staff@helix.nih.gov

Helix Systems Staff.


5/21/2013

Recent publication citing Helix/Biowulf resources:

Improving Peptide Identification Sensitivity in Shotgun Proteomics by Stratification of Search Space

Gelio Alves and Yi-Kuo Yu DOI: 10.1021/pr301139y (2013) J. Proteome Research :