Skip to content

Storage Guidelines

Storage on the Research Cluster is a shared resource, and each area is provided for a particular purpose. These guidelines cover what each area is for, how to get one, how long data should stay there, and who is responsible for keeping it safe.

For the full detail on each area, see the Storage Overview.

At a Glance

Home Projects Weka Otago HCS
Provided With every account On request On request Via AskOtago
Intended for Scripts and configuration Data you are actively working on High-throughput scratch Long-term storage and backup
Backed up
Default quota 40 GB Set on request 0 GB Managed by Core Digital

Home Directories

Every user of the Research Cluster gets a home directory at /home/<username>. It is provided for managing your scripts, configuration files, and other smaller files used in your computations.

  • The hard quota is 40 GB. Once you hit it you will not be able to write any more data.
  • You will be emailed at your Otago address once you reach 30 GB.
  • Home directories are backed up: one snapshot per day for the last 7 days, one per week for the last 4 weeks, and one per month for the last 6 months.

Home directories are not the place for research datasets. If you are working with data at any scale, request a projects allocation.

Projects and Weka

/projects and /weka are available on request through the Storage Request page.

Both are allocated to a group rather than an individual, so a request must name:

  • The department the allocation belongs to, and
  • The principal investigator, who is the data owner and is responsible for the data held in the allocation.

Quotas are set when the allocation is created and can be revised on request.

These areas are for data that is in use. The expected pattern is to copy your source data in from HCS, work on it, transfer the results back, and remove the working copy once you are finished. Weka in particular is scratch space — data written to /tmp/ during a job is deleted when the job ends.

Backups Are Your Responsibility

Only home directories are backed up

Nothing in /projects or /weka is backed up. If you delete something there, or a disk fails, it is gone. It is your responsibility to make sure your important data is safe.

Your research data should have a copy on Otago HCS, which is backed up and is the recommended long-term home for it. Treat the cluster copy as a working copy that you can afford to lose.

Recommended practice:

  1. Keep the authoritative copy of your data on HCS.
  2. Copy what you need into /projects to work on.
  3. Transfer results back to HCS when you are done.
  4. Remove the working copy from the cluster.

Setting up Globus to transfer between the two on a schedule is a good way to keep this up to date without having to remember. See Data Transfer for the options.

Note

If you do not have an HCS share for your department or group, fill out the HCS Access Form.

Questions

If you are unsure which storage area suits your work, how much to request, or how to go about backing up your data, email the eResearch Support team at rtis.support@otago.ac.nz — we are happy to help you work it out.

What's next?