Skip to content

Storage

The storage is organized into four distinct spaces called filesets, each designed for specific use cases. The following table outlines the key attributes of each fileset:

Filesets Capacity Burst Buffer Usage Backup Notes
/gpfs/home 200 TB No User home directories Yes Quotas applied per user.
/gpfs/projects 1.5 PB No Shared project spaces Yes Quotas applied per group.
/gpfs/scratch 1 PB Yes Temporary high-speed storage No Not backed up.
/gpfs/softs 50 TB No Pre-installed software libraries Yes Centrally managed.

Data in home, projects, and softs is backed up using IBM Spectrum Protect, while scratch is excluded.

The /gpfs/scratch fileset is not backed up

As it is designed for temporary storage, users must transfer any critical data elsewhere before job completion.

The /gpfs/scratch directory is cleaned up during maintenance windows

In late May and November. A reminder will be sent 15 days beforehand.

Users' storage

The /gpfs/home directory contains each user's personal files, configurations, and scripts for managing jobs. User home directories are organized by organization and follow this structure: /gpfs/home/<organization>/<user>. This directory serves as the default starting point when logging into Lucia and is often referred to as $HOME or ~.

Access to your home directory, /gpfs/home/<organization>/<user>/, is restricted to you by default. Others will not be able to access it unless specifically shared.

Example

By default, your HOME directory is restricted to you. 

> ls -ld $HOME 
drwx------ 7 <user> <user> 4096 Nov 19 18:57 /gpfs/home/<organization>/<user>

Projects' storage

Each project has two main directories where shared data is stored:

  • /gpfs/projects: For long-term collaborative data storage (e.g., software, input files, job results).
  • /gpfs/scratch: A high-speed, temporary workspace for job execution that requires fast I/O.

UNIX groups

The directories under /gpfs/projects and /gpfs/scratch are group-owned by the corresponding project's UNIX group.

Example

UNIX groups project01 and project02 respectively own their project directories /gpfs/projects/company/project01/ and /gpfs/projects/company/project02/

> ls -l /gpfs/projects/company/
drwxrws--- 3 root project01 4096 Mar 14 13:37 /gpfs/projects/company/project01/
drwxrws--- 4 root project02 4096 Mar 14 13:37 /gpfs/projects/company/project02/

Project membership

A user can access all projects associated with the UNIX groups they belong to. Each user can check their group memberships using the id command.

Example

User toto belongs to UNIX groups project01, project02, and project03, granting access to the directories and files associated with each of these projects.

> id
uid=xxxx(toto) gid=xxxx(toto) groups=xxxx(toto),xxxx(project01),xxxx(project02),xxxx(project03)

Quota

Quotas set limits on both storage capacity and file count. The file limit depends on the block limit. By default,

  • Projects with up to 500GB of block limit have a minimum of 500k files.
  • For each additional GB of block space, the file limit increases by 1k, capped at 10,000k files for projects with block limits greater than 10,000GB (this can be increased upon request).

Quotas apply as follows:

  • /gpfs/home : quota based on the user’s UNIX username,
  • /gpfs/projects and /gpfs/scratch : quotas based on the project’s UNIX group, with limits varying per project.
Disk quota exceeded

For non-project UNIX groups, /gpfs/projects and /gpfs/scratch quotas are minimal (16KB and 1 file). For this reason, misconfigured permissions or ownership on project directories may cause a Disk quota exceeded error. If this occurs, refer to FAQ > Disk quota exceeded for guidance on how to fix permissions.

Default quota policies

By default, quotas for users' home directory and industrial projects are set as follows :

Fileset Block Soft Limit Block Hard Limit File Soft Limit File Hard Limit Grace Period
/gpfs/home 200GB 230GB 1000k files 1300k files 7 days
/gpfs/projects 1000GB 1300GB 2000k files 1300k files 7 days
/gpfs/scratch 1000GB 1300GB 1000k files 1300k files 7 days
Grace period

Exceeding the soft limit triggers a grace period. After 7 days or upon reaching the hard limit, further writes are blocked until usage is reduced below the soft limit.

How new files are assigned to a project's quota

Project directories use the setgid bit to ensure that all newly created files and subdirectories automatically inherit the project’s UNIX group. This guarantees that their storage and file usage is charged to the corresponding project quota.

The presence of s in the group-execute position (drwxrws---) indicates that the setgid bit is active. As a result, anything created inside the directory will keep the project’s group ownership, regardless of the user’s default group.

Example

project01's directories has the setgid bit (s) set on its UNIX group permissions :

> ls -ld /gpfs/{projects,scratch}/company/project01/
drwxrws--- 3 root project01 4096 Mar 14 13:37 /gpfs/projects/company/project01/
drwxrws--- 4 root project01 4096 Mar 14 13:37 /gpfs/scratch/company/project01/
Running an application using the project's quota

Applications sometimes ignore the directory’s setgid bit and create files under your default UNIX group. To make sure new files are attributed to the project, run newgrp <project>. This temporarily changes your active group so that all newly created files and directories inherit the project’s group ownership and therefore count toward the project’s quota.

Viewing quota usage and limits

To check a user's quota usage and limits:

  • For /gpfs/home: 
    mmlsquota -u <username> --block-size g ess:home

To check a project's quota usage and limits:

  • For /gpfs/projects: 

    mmlsquota -g <project_name> --block-size g ess:projects

  • For /gpfs/scratch: 

    mmlsquota -g <project_name> --block-size g ess:scratch

  • For all filesets: 

    mmlsquota -g <project_name> --block-size g ess