Otago HCS¶

Otago High Capacity Storage (HCS)¶

On this Page

To see where HCS fits within Storage check out Storage Overview
What is HCS (High Capacity Storage)
How to mount HCS on the login node
How to mount HCS on the cluster nodes
How to generate auto-renewal Kerberos tickets

HCS is the main data storage pool on the Otago campus. HCS can be mounted on desktop and lab computers. It is also possible to mount HCS shares on the cluster login node to transfer data, but the connection is sub-optimal for high-speed computing and cannot easily be mounted across compute nodes. We recommend using HCS for your primary storage needs and transferring your data to your working area in /projects/ when you want to work on it, then transferring results back to HCS for long-term storage.

HCS is also the primary backup location for research data. Cluster storage /projects and /weka are not backed up. You will need to become familiar with transferring data to and from HCS in order to back up your data.

Note

If you do not have an HCS share available for your department/group, please fill out the HCS Access Form .

Methods for Accessing HCS¶

How you access your data on HCS will depend on your use case. The three main use cases are detailed below:

Accessing HCS share from the login node - Good for moving/copying small amounts of data, or browsing on the command line
Accessing HCS shares from a compute node - Good for when you need to access data on HCS as part of a job
Using rclone to copy data or mount HCS, also available on an OnDemand HPC Desktop in the file browser - Good for transferring small to medium-sized data sets and for browsing HCS data on the HPC desktop

Info

For best performance it is best to have a copy of the data you want to use on the research storage. For large transfers this is best done with Globus or rclone.

For small one-off data transfers, you can mount your HCS share on the login node. This is not recommended for large data transfers or for data that will be used in a job, as the connection is not optimal for high-speed computing. It can also be mounted on an HPC desktop for smaller data sets and transfers.

To mount your HCS share on the login node, you will need a valid Kerberos ticket. If you do not have one, you can generate it by running the kinit command and entering your University password.

Otago HCS (High Capacity Storage) shares can be accessed on the cluster provided you have security access and obtain a Kerberos ticket (password authentication). To access your HCS share: 1. Check if you have a Kerberos ticket klist. If not, obtain one with kinit and enter your University password. 3. Navigate to your HCS share /mnt/auto-hcs/<your_share_name> (the last portion of the HCS address) eg. if my share is \\storage.hcs-p01.otago.ac.nz\its-rtis then navigate to /mnt/auto-hcs/its-rtis

Occasionally, it may take up to a minute for this access to become available after running kinit.

Otago MDS storage (Windows Managed Desktop Share) can be accessed on the cluster if you have a Kerberos ticket. To access your MDS share: 1. Check if you have a Kerberos ticket klist. If not, obtain one with kinit and enter your University password. 2. Navigate to your MDS share /mnt/auto-mds/<first_initial_of_your_username>/<your_username> eg. if my MDS share is \\registry.otago.ac.nz\mdr\Profiles-V2\h\higje06p then navigate to /mnt/auto-mds/h/higje06p

Note

It can sometimes take up to 1 minute for the authentication process to complete after logging in and your HCS share to be available through this method

Accessing HCS on the cluster nodes - Auks¶

The Auks Slurm plugin enables users to save their Kerberos ticket from the login node onto the Auks server. This saved ticket can then be used on any Slurm compute node to access HCS shares. The Kerberos ticket is automatically renewed by Auks and remains valid for up to 7 days. After this period, the ticket must be manually renewed. Once a user saves their Kerberos ticket, the renewal process happens automatically.

Whenever a user logs in to the Aoraki login node using their password, a new krbtgt (Kerberos Ticket-Granting Ticket) is issued. However, if the user logs in via SSH keys or accesses nodes through the OnDemand shell, they must manually obtain a new valid ticket by running the kinit command. When you type your password after running kinit, the terminal intentionally suppresses any visible feedback. Confirm your password by pressing Enter.

STAFFSTUDENTS

Terminal

kinit userx01p@REGISTRY.OTAGO.AC.NZ
Password for userx01p@REGISTRY.OTAGO.AC.NZ:

Terminal

kinit studx012@STUDENT.OTAGO.AC.NZ
Password for studx012@STUDENT.OTAGO.AC.NZ:

A krbtgt (Kerberos Ticket-Granting Ticket) is valid for 10 hours, as indicated by the "Expires" date and time. During this period, the ticket can be renewed using the kinit -R command. This renewal extends the "Expires" time but does not change the "Renew Until" date and time. The Auks system handles this renewal process automatically for the user.

As long as a user has a valid krbtgt ticket, they can communicate with the Auks server seamlessly.

Check if you have a valid krbtgt ticket:

Terminal

[studx012@aoraki-login ~]$ klist

Ticket cache: KCM:40005987:63840
Default principal: studx012@STUDENT.OTAGO.AC.NZ

Valid starting Expires Service principal
11/26/2024 08:56:14 11/26/2024 18:56:14 krbtgt/STUDENT.OTAGO.AC.NZ@STUDENT.OTAGO.AC.NZ
    renew until 12/03/2024 08:56:14

[studx012@aoraki-login ~]$ kinit -R
[studx012@aoraki-login ~]$ klist

Ticket cache: KCM:40005987:63840
Default principal: studx012@STUDENT.OTAGO.AC.NZ

Valid starting Expires Service principal
11/26/2024 11:22:40 11/26/2024 21:22:40 krbtgt/STUDENT.OTAGO.AC.NZ@STUDENT.OTAGO.AC.NZ
    renew until 12/03/2024 08:56:14

Ping the server to verify the connection:

Terminal

[studx012@aoraki-login ~]$ auks -p

Auks API request succeed

Save the current krbtgt on the Auks server:

Terminal

[studx012@aoraki-login ~]$ auks -a

Auks API request succeed

Now you can use the --auks=yes option with srun or #SBATCH --auks=yes in your Slurm batch script:

Terminal

srun --auks=yes --partition=aoraki ls -lah /mnt/auto-hcs/hcs_share_name

HCS shares are automatically mounted on each node when requested. The user's krbtgt, retrieved from the Auks server, is used to generate Kerberos tickets for communication with the HCS storage servers. However, obtaining these tickets can take some time. If the HCS share is not yet mounted on the node, the job or script may fail.

To mitigate this, it is recommended to include a command such as sleep 20 in batch scripts before accessing the auto-mounted share.

Generating Auto-Renewal Kerberos Tickets¶

Info

If you want to keep using your data over hours or days you will need to renew your Kerberos ticket. Kerberos tickets are only valid for 1 hour and then need to be renewed, either by generating one via the command line or automatically by generating a keytab file and setting a cron job to renew the ticket at a specified interval.

To see if you have a valid Kerberos ticket you can type klist

You do not have a valid ticket if you get a message like: klist: Credentials cache 'KCM:51776880' not found
You do have a ticket if you get a message similar to:

STAFFSTUDENTS

Terminal

klist

Ticket cache: KCM:51776880 Default principal: higje06p@REGISTRY.OTAGO.AC.NZ

Valid starting     Expires            Service principal
27/09/23 15:01:57  28/09/23 01:01:57  krbtgt/REGISTRY.OTAGO.AC.NZ@REGISTRY.OTAGO.AC.NZ
    renew until 04/10/23 15:01:57

To generate an auto-renewing ticket (so it does not expire):

Create a keytab file — replace username with your Otago username:

Terminal

ktutil

ktutil: addent -password -p <username>@REGISTRY.OTAGO.AC.NZ -k 1 -e aes256-cts (enter your password)
Ktutil: wkt /home/<username>/<username>.keytab
Ktutil: quit

To test: kinit <username>@REGISTRY.OTAGO.AC.NZ -k -t <username>.keytab
klist (to check if ticket is valid)
Then create a cron job to renew your ticket: crontab -e and enter the following line to renew your ticket every hour: 0 * * * * kinit <username>@REGISTRY.OTAGO.AC.NZ -k -t /home/<username>/<username>.keytab
Ensure the permissions on your .keytab file are 0600 to keep it secure!

Terminal

klist

Ticket cache: KCM:51776880 Default principal: higje06p@STUDENT.OTAGO.AC.NZ

Valid starting     Expires            Service principal
27/09/23 15:01:57  28/09/23 01:01:57  krbtgt/STUDENT.OTAGO.AC.NZ@STUDENT.OTAGO.AC.NZ
    renew until 04/10/23 15:01:57

To generate an auto-renewing ticket (so it does not expire):

Create a keytab file — replace username with your Otago username:

Terminal

ktutil

ktutil: addent -password -p <username>@STUDENT.OTAGO.AC.NZ -k 1 -e aes256-cts (enter your password)
Ktutil: wkt /home/<username>/<username>.keytab
Ktutil: quit

To test: kinit <username>@STUDENT.OTAGO.AC.NZ -k -t <username>.keytab
klist (to check if ticket is valid)
Then create a cron job to renew your ticket: crontab -e and enter the following line to renew your ticket every hour: 0 * * * * kinit <username>@STUDENT.OTAGO.AC.NZ -k -t /home/<username>/<username>.keytab
Ensure the permissions on your .keytab file are 0600 to keep it secure!

What's next?

Find out how to move data on and off the Research Cluster on Data Transfer

Otago HCS¶

Otago High Capacity Storage (HCS)¶

Methods for Accessing HCS¶

Accessing HCS on the Login node¶

Accessing HCS on the cluster nodes - Auks¶

Generating Auto-Renewal Kerberos Tickets¶