Skip to content

Using GHI at NERSC

Introduction

GPFS/HPSS Interface (GHI) is a way of interacting with the HPSS archive that offers the benefits of a more familiar file system interface. Often users want to store complex directory structures or large bundles in HPSS which can be difficult to do with the traditional HPSS access tool. GHI can be used to easily move data between HPSS and the GPFS file system with a few simple commands.

Files archived/migrated into HPSS with GHI can’t be easily accessed with other HPSS tools like ftp, hsi, or htar. The assumption is that you will continue to use GHI to manage these files.

GHI is best thought of as a merging of two file systems (a single namespace), i.e. if you delete a GHI controlled file on the GPFS file system, it will also be deleted in HPSS.

Warning

This installation of GHI at NERSC is experimental. Hardware backing GHI is not designed for production use, and thus does not have the same safeguards that are present in the production systems. Please don’t store any unique data in GHI and please let us know first if you’re going to try to access it at large scales.

Example Use Cases

Large Volumes of Infrequently Accessed Data

A group has 100s of TBs of data that is only accessed twice a year for reanalysis. The group doesn’t want to keep this data on GPFS in between analyses because it’s very large, so they use ghi put to initially put the files into HPSS. Then they use ghi punch to remove all but the first few bytes from GPFS and free up space in their directories. Then, when they’re ready to analyse the data, then bring it back to GPFS using ghi stage. They can check which file system the data is on using ghi ls.

Archiving Complex Directory Structures

A project has many thousands of files of varying size ranging from a few bytes to hundreds of GBs (some files are too large to be bundled by htar) they’d like to archive. These files are organized in a complex directory structure that’s necessary to maintain in order to track the material in the files. The aggregate size of this data is around 100TB. Normally, they’d have to use tar to bundle each directory together (or parts of directories to keep the bundles around 500GB) and then put them into HPSS. Instead, they use the ghi put option to put the entire directory into HPSS.

Using GHI

Access

Currently you can only access GHI on NERSC's data transfer nodes using the /global/projectm file system. If you don’t have a usable directory there, let us know and we’ll make one. You can reach the data transfer nodes via ssh to dtn0[1-4].nersc.gov.

Commands

GHI can be accessed via the ghi command. This command should already be in your path on the NERSC data transfer nodes.

Usage

The ghi command requires an “action” argument. The possible choices are:

  • ls: show what file system the files are currently on
  • put: copy the files to HPSS
  • stage: move the files back from HPSS to spinning disk
  • punch: move the files to only HPSS
  • pin: keep the files from being removed from spinning disk

GHI ls

This is used to tell you which file system the files are on:

  • “G” means only on GPFS
  • “H” means only on HPSS (with only a small stub remaining on GPFS)
  • “B” mean on both GPFS and HPSS

Also, if a file is “pinned” there will be a “P” shown when listing.

You can also use extra flags to see more information:

  • -a: Show hidden files
  • -l:Long listing (same as “-l” for ls)
  • -e: Show HPSS tape position
  • -w: Show name of file in HPSS
  • -n: Same as “-l” but use numeric user and group ID
  • -?: Print List Help

Ghi ls can also be used to display the name of the file in HPSS (note it is a UUID that’s used internally for system tracking).

List one file
dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
B  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt

The “B” means the file is on both GPFS and HPSS (“H” is HPSS only, and “G” is GPFS only)

List many files

To list many files, you can either list them on the command line or put them in a text file. You can also list every file in a directory with

dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis
And recursive list all directories by adding a wildcard (*) at the end
dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/*

List via text file

To list files in a input text file, use the “-f” option:

dtn> cat /global/projectm/projectdirs/nstaff/elvis/my_list.txt
/global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
dtn> ghi ls -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt
B  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt

GHI put

This is used to put a file or set of files into HPSS.

dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/test.txt
G  /global/projectm/projectdirs/nstaff/elvis/test.txt
dtn> ghi put /global/projectm/projectdirs/nstaff/elvis/test.txt
dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/test.txt
B  /global/projectm/projectdirs/nstaff/elvis/test.txt

You can also put many files using wildcards (*) or by putting them into a separate text file and using the -f flag following the syntax described in the ghi ls section.

GHI stage

This is used to bring a file or set of files back from HPSS.

Note

Ghi stage is asynchronous. This means that the prompt could return before all files are finished being retrieved from HPSS. You can check the progress of a stage using the ghi ls command.

dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/test.txt
H  /global/projectm/projectdirs/nstaff/elvis/test.txt

dtn> ghi stage /global/projectm/projectdirs/nstaff/elvis/test.txt
option -t t 0
Will not wait for results after posting stage request(s) to SD...
Staging 1 file...
Splitting the scan results for processing.
Grouping the scan results by media.
Sorting the results by media order.
Setting up socket to SD.
Sending stage request(s) to the SD.
All stage requests acknowledged by SD.

Temporary files/directories left behind are /global/projectm/scratch/.ghi/ghi_stage.ngfsv491.nersc.gov.16869
They may be removed with the command: rm -r /global/projectm/scratch/.ghi/ghi_stage.ngfsv491.nersc.gov.16869

dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/test.txt
B  /global/projectm/projectdirs/nstaff/elvis/test.txt

You can also stage many files using wildcards (*) or by putting them into a separate text file and using the -f flag following the syntax described in the ghi ls section.

GHI punch

This is used to remove a file or set of files from GPFS. The file must be put into HPSS before it can be punched. When a file is punched, only a small stub is left behind on the GPFS file system. This is usually done to free up space for other more frequently used data. You must have write access to a file in order to punch it.

Note

Files must be put into HPSS using ghi put before they can be punched.

dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
B  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt

dtn> ghi punch /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt

dtn> ghi ls /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
H  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt

You can also stage many files using wildcards (*) or by putting them into a separate text file and using the -f flag following the syntax described in the ghi ls section.

GHI pin

You can use ghi pin to keep files on the GPFS file system. A pinned file will remain on the GPFS file system even if you punch it later. Pinned files have an extra "P" in the output when you list them with ghi ls. You can unpin files by invoking ghi pin with the -u flag.

In the example below, the pinned file remains on both HPSS and GPFS even after a punch command is issued. You can use ghi pin with either a list of files on the command line or -f for a text file list following the syntax described in the ghi ls section.

dtn> ghi ls -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt
B  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
B  /global/projectm/projectdirs/nstaff/elvis/blah.txt

dtn> ghi pin /global/projectm/projectdirs/nstaff/elvis/blah.txt

dtn> ghi ls -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt
B  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
BP /global/projectm/projectdirs/nstaff/elvis/blah.txt

dtn> ghi punch -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt

dtn> ghi ls -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt
H  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
BP /global/projectm/projectdirs/nstaff/elvis/blah.txt
Unpinning a file

You unpin a file by adding the -u flag to the pin command.

dtn> ghi ls -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt
H  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
BP /global/projectm/projectdirs/nstaff/elvis/blah.txt

dtn> ghi pin /global/projectm/projectdirs/nstaff/elvis/blah.txt -u

dtn> ghi ls -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt
H  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
B  /global/projectm/projectdirs/nstaff/elvis/blah.txt

dtn> ghi punch -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt

dtn> ghi ls -f /global/projectm/projectdirs/nstaff/elvis/my_list.txt
H  /global/projectm/projectdirs/nstaff/elvis/strace_ghi.txt
H  /global/projectm/projectdirs/nstaff/elvis/blah.txt