Storage
Looking for a short summary?
A concise description of the storage solution specifications and performance is available at Overview → Storage.
Continue reading below for per-user and per-project data storage organization.
The storage is organized into four distinct spaces called filesets, each designed for specific use cases. The following table outlines the key attributes of each fileset:
| Filesets | Burst Buffer | Snapshots & Backup | Space Administrator | Usage | Capacity | Quotas |
|---|---|---|---|---|---|---|
| /gpfs/home | No | Yes | Individual user | User home directories | 200 TB | per user |
| /gpfs/projects | No | Yes | Project members | Shared project workspaces | 1.5 PB | per project |
| /gpfs/scratch | Yes | No | Project members | Temporary high-speed storage | 1 PB | per project |
| /gpfs/softs | No | Yes | IT staff | Pre-installed software stack | 50 TB |
The /gpfs/scratch fileset is not backed up
As it is designed for temporary storage, users must transfer any critical data elsewhere before job completion.
The /gpfs/scratch directory is cleaned up during maintenance windows
In late May and November. A reminder will be sent 15 days beforehand.
The /gpfs/home directory contains each user's personal files, configurations, and scripts for managing jobs. User home directories are organized by organization and follow this structure: /gpfs/home/<organization>/<user>. This directory serves as the default starting point when logging into Lucia and is often referred to as $HOME or ~.
Access to your home directory, /gpfs/home/<organization>/<user>/, is restricted to you by default. Others will not be able to access it unless specifically shared.
Each project has two main directories where shared data is stored:
- /gpfs/projects: For long-term collaborative data storage (e.g., software, input files, job results).
- /gpfs/scratch: A high-speed, temporary workspace for job execution that requires fast I/O.
UNIX groups
The directories under /gpfs/projects and /gpfs/scratch are group-owned by the corresponding project's UNIX group.
Example
UNIX groups project01 and project02 respectively own their project directories /gpfs/projects/company/project01/ and /gpfs/projects/company/project02/
Project membership
A user can access all projects associated with the UNIX groups they belong to. Each user can check their group memberships using the id command.
Quotas set limits on both storage capacity and file count. The file limit depends on the block limit. By default,
- Projects with up to 500GB of block limit have a minimum of 500k files.
- For each additional GB of block space, the file limit increases by 1k, capped at 10,000k files for projects with block limits greater than 10,000GB (this can be increased upon request).
Quotas apply as follows:
/gpfs/home: quota based on the user’s UNIX username,/gpfs/projectsand/gpfs/scratch: quotas based on the project’s UNIX group, with limits varying per project.
Disk quota exceeded
For non-project UNIX groups, /gpfs/projects and /gpfs/scratch quotas are minimal (16KB and 1 file). For this reason, misconfigured permissions or ownership on project directories may cause a Disk quota exceeded error. If this occurs, refer to FAQ > Disk quota exceeded for guidance on how to fix permissions.
Default quota policies
By default, quotas for users' home directory and industrial projects are set as follows :
| Fileset | Block Soft Limit | Block Hard Limit | File Soft Limit | File Hard Limit | Grace Period |
|---|---|---|---|---|---|
/gpfs/home |
200GB | 230GB | 1000k files | 1300k files | 7 days |
/gpfs/projects |
1000GB | 1300GB | 2000k files | 1300k files | 7 days |
/gpfs/scratch |
1000GB | 1300GB | 1000k files | 1300k files | 7 days |
Grace period
Exceeding the soft limit triggers a grace period. After 7 days or upon reaching the hard limit, further writes are blocked until usage is reduced below the soft limit.
How new files are assigned to a project's quota
Project directories use the setgid bit to ensure that all newly created files and subdirectories automatically inherit the project’s UNIX group. This guarantees that their storage and file usage is charged to the corresponding project quota.
The presence of s in the group-execute position (drwxrws---) indicates that the setgid bit is active. As a result, anything created inside the directory will keep the project’s group ownership, regardless of the user’s default group.
Example
project01's directories has the setgid bit (s) set on its UNIX group permissions :
Running an application using the project's quota
Applications sometimes ignore the directory’s setgid bit and create files under your default UNIX group. To make sure new files are attributed to the project, run newgrp <project>. This temporarily changes your active group so that all newly created files and directories inherit the project’s group ownership and therefore count toward the project’s quota.
Viewing quota usage and limits
To check a user's quota usage and limits:
- For
/gpfs/home:
To check a project's quota usage and limits:
-
For
/gpfs/projects: -
For
/gpfs/scratch: -
For all filesets:
Snapshots
A snapshot is a read-only capture of a storage unit's state at a specific point in time. On Lucia, snapshots of the home (/gpfs/home), projects (/gpfs/projects), and softs (/gpfs/softs) filesets are taken nightly around midnight and are retained for 7 days.
If you inadvertently delete a file or directory, you can restore it yourself using a snapshot.
Snapshots can be accessed from any directory within a fileset via a hidden .snapshots subdirectory. Note that this directory is virtual, it will not appear in standard directory listings (e.g., ls -a) and shell autocompletion will not work. However, it can be browsed like any normal directory by manualy typing the full path.
For instance, listing available snapshots in your home directory:
ls -l ~/.snapshots
total 97
drwxr-xr-x 22 username username 16384 Jun 10 22:58 snap_ess-home_20260611-0000_daily
drwxr-xr-x 22 username username 16384 Jun 11 03:37 snap_ess-home_20260612-0000_daily
drwxr-xr-x 22 username username 16384 Jun 12 08:46 snap_ess-home_20260613-0000_daily
drwxr-xr-x 22 username username 16384 Jun 13 06:21 snap_ess-home_20260614-0000_daily
drwxr-xr-x 22 username username 16384 Jun 14 08:42 snap_ess-home_20260615-0000_daily
drwxr-xr-x 22 username username 16384 Jun 15 20:23 snap_ess-home_20260616-0000_daily
drwxr-xr-x 22 username username 4096 Jun 16 22:12 snap_ess-home_20260617-0000_daily
Also, since the .snapshots directory is available at all levels of the directory tree, the following paths are equivalent:
/gpfs/projects/.snapshots/snap_ess-projects_20260611-0000_daily/company/project_name/some_directory/
/gpfs/projects/company/project_name/.snapshots/snap_ess-projects_20260611-0000_daily/some_directory/
/gpfs/projects/company/project_name/some_directory/.snapshots/snap_ess-projects_20260611-0000_daily/
To recover data, simply copy the files or directories from the snapshot back to your "live" directory, e.g.:
cp -p ~/.snapshots/snap_ess-projects_20260611-0000_daily/some_directory/some_script.slurm ~/some_directory/
Backups
In addition to snapshots, we perform daily backups of the home, projects, and softs filesets. These backups are stored on tape at a remote site and are kept for 30 days.
If you need to recover data that is no longer available via snapshots, please submit a request by creating a ticket on our issue tracking system.