File System Administration#

Instructions for creating and configuring a file system for cloud storage on CONVERGE Horizon are included below.

Create a File System#

Follow these steps to create a new file system on CONVERGE Horizon:

  1. In the CONVERGE Horizon web interface, go to the File Systems page.

  2. Click Create File System in the upper right.

  3. Fill out the file system options.

  4. Review the configuration and pricing estimate on the right. When you are satisfied with the settings, click Create File System.

Your new file system will be added to the list on the File Systems page. When the status changes to Online, the file system is ready for use.

Your team can access the file system using the File Manager or using standard command-line utilities. The file system is automatically mounted to job clusters when you run jobs.

View the File System Log#

Each file system has a system log showing detailed information from the CONVERGE Horizon backend.

On the File Systems page, click on the name of the file system to open the File System Details page. Expand the System Log section to view the system log. Only the most recent information from the system log is displayed.

../_images/system_log.png

An example system log for a file system.#

View File System Metrics#

You can view real-time metrics to keep tabs on file system performance. On the File Systems page, click on the name of a file system to open the File System Details page. Expand the Node Metrics section to see recent data for CPU utilization, memory utilization, and disk IOPS.

Edit a File System#

Edit an existing file system if you need to increase the size, change the auto-expand settings, configure lifecycle policies, or configure allowed IPs. There are two ways to open the Edit File System page:

  • On the File Systems page, click the Edit (edit-button) button in the row for the file system you want to modify.

  • On the File System Details page, click Edit in the lower right.

When you specify the new file system settings, the pricing estimate will be updated to show the new monthly cost. If you are satisfied with these details, click Update File System.

Migrate a File System#

Migrate an existing file system to a new file system if you need to change the data center location, change the node type, or reduce the size. When you start the migration, a new file system will be created and your data will be copied from the old file system to the new file system (this will incur an additional data transfer cost if you are moving data to a different data center). The old file system will not be deleted automatically; you can delete it manually after confirming that your data was migrated successfully.

On the File System Details page, click Migrate in the lower right to open the Migrate File System modal. Specify a data center, node type, and name for the new file system. CONVERGE Horizon will create the new file system with the smallest possible size, subject to these constraints:

  • The new size must equal or exceed the minimum size for the selected node type.

  • The amount of data copied from the old file system cannot be more than 70% of the new size.

The entire contents of the File Manager home directory (/mnt/fs/files) will be copied to the new file system. The applications cache (/mnt/fs/.cache/apps), which contains versions of CONVERGE that you ran on the old file system, will NOT be copied to the new file system. Once you start running jobs on the new file system, CONVERGE and other applications will be installed as needed.

The data transfer cost will be calculated and displayed in the Migrate File System modal. When you are ready, check the box to accept the transfer charge (if applicable) and then click Start Migration. The old file system will be read-only in the web interface while the migration is underway, although it will remain accessible via SSH. Do not write to the old file system during the migration; any changes you make might not be copied to the new file system.

When the migration is done, the status of both file systems will change to Online. After confirming that the migration was successful, you can delete the old file system.

Hibernate a File System#

You can hibernate a file system if your team will not be using it for an extended period of time. During hibernation, the file system’s storage device is detached from the VM node and the node is deleted, while your data remains intact on the storage device. This allows you to continue storing data securely in the cloud without paying the hardware cost of the node (storage costs still apply).

Follow these steps to hibernate a file system:

  1. In the CONVERGE Horizon web interface, go to the File Systems page.

  2. Click on the file system name to open the File System Details page.

  3. In the bottom right, click Hibernate.

  4. At the prompt, enter the name of the file system to confirm that you want to continue.

  5. Click Hibernate.

You can wake the file system when you are ready to start using it again:

  1. In the CONVERGE Horizon web interface, go to the File Systems page.

  2. Click on the name of the hibernating file system to open the File System Details page.

  3. In the bottom right, click Wake.

CONVERGE Horizon will provision a new VM node and attach the storage device to the new node. You can access the File Manager when the file system status changes back to Online.

Mount a File System to a Local Device#

If you have added an SSH key to your account, you can use an SSH or SFTP client to mount a CONVERGE Horizon file system to a local device. This allows you to interact with directories and files stored in the cloud as if they were on your own computer. For example, you can load the output files from a job that is running in the cloud directly into post-processing tools that are running on your local system.

Various SSH/SFTP clients are widely available, and we do not officially support specific clients. If you are interested in this functionality, we recommend working with your IT team to identify and install a suitable client for your system.

The process for mounting a file system will vary depending on which client you are using. Below, we’ve provided example commands for mounting a file system with rclone.

Example: Mounting a File System With Rclone

In a Linux/macOS terminal or Windows Command Prompt (cmd), run rclone config to add the file system’s virtual machine to your rclone configuration file. Select the option to add a new remote and then answer the prompts as shown below. Any prompt not listed below can be left blank to accept the default value.

Recommended Rclone Configuration#

Prompt

Response

name

Enter any name for the remote (in the examples below, we use remote_name as a placeholder for the remote name)

Storage

sftp (option 51 on Linux and option 55 on Windows)

host

Enter the IP address or DNS name shown in File System Details > Connection Information (in the examples below, we use x.x.x.x as a placeholder for the host)

user

opc

key_file

Enter the full path to your private SSH key (e.g., /home/user/.ssh/id_ed25519)

After the configuration is completed, answer Yes (y) when prompted to keep this remote, and then quit the configuration (q). The resulting rclone configuration file (rclone.conf) looks something like this:

[remote_name]
type = sftp
host = x.x.x.x
user = opc
key_file = /home/user/.ssh/id_ed25519

If you want to confirm that your rclone.conf file has the correct format, you can find the file path with rclone config file.

Example commands for mounting the file system to your local disk are shown below. To mount only a specific directory within the file system, change /mnt/fs/files to /mnt/fs/files/dir.

On Linux systems, the mount point (tmp in these examples) must exist before you run rclone mount. We recommend creating an empty directory within your home directory for the mount point. On Windows systems, the mount point will be created by rclone and cannot already exist.

For Windows, we recommend using --sftp-skip-links to skip any soft links that exist in the file system.

# Linux
rclone mount remote_name:/mnt/fs/files /home/user/tmp

# Windows
rclone mount --sftp-skip-links remote_name:/mnt/fs/files C:\Users\user\tmp

Refer to rclone documentation for more options and details.

Delete a File System#

When your team is done using a file system and you have downloaded all of your data from the cloud, you can delete the file system.

Caution

All files on the file system will be permanently deleted when the file system is deleted. Before proceeding, confirm that your team has downloaded any files they want to keep.

  1. In the CONVERGE Horizon web interface, go to the File Systems page.

  2. Click on the file system name to open the File System Details page.

  3. In the bottom right, click Delete.

  4. At the prompt, enter the name of the file system to confirm that you want to continue.

  5. Click Delete.