Bulk Uploading Assets to Specify Cloud via SSH/SFTP

[Grant]

This guide is intended for IT Administrators of institutions migrating their Specify instance to Specify Cloud.

Bulk Uploading Assets to Specify Cloud via SSH/SFTP

For institutions migrating to Specify Cloud hosting that already have assets associated with records in the database, we offer a few methods for uploading large batches of attachment files directly to the asset server. This is particularly useful for initial data migrations or when moving terabytes of legacy data.

[!help] Size Note
If you have fewer than 25GB of assets, we recommend sharing the assets with us via Google Drive, OneDrive, or similar service as we can perform the migration steps for you behind-the-scenes.

Prerequisites

Before you can begin, you must coordinate with the Specify support team to enable this access.

1. Request Access: Email support@specifysoftware.org and state that you need direct asset server access for bulk uploads along with your public SSH key (instructions for generating an SSH key pair).

2. User Setup: The SCC team will provision a dedicated Linux user for your institution on the asset server. You will be provided with a username which you can use to connect to the asset server for file transfer purposes.

Connecting to the Server

Once your user is configured, you can connect using standard file transfer protocols.

• For Graphical Interface: Use clients like FileZilla, Cyberduck, or WinSCP. Set the protocol to SFTP (SSH File Transfer Protocol).

• For Command Line: You can use standard scp or rsync commands.

Directory Structure

When you log in to your user’s home directory, you will see a subdirectory named attachments.

[!note]
This directory is a direct link (mount) to the private S3 storage bucket for your institution. Changes made here are reflected immediately in your Specify Cloud storage usage.

Inside attachments, you will find the following structure:

• /attachments/originals/ <— Upload your files here
• /attachments/thumbnails/

Upload Process

1. Transfer Files: Upload your high-resolution images or documents into the originals folder.
2. Automatic Processing: Once the files land in originals, the asset server detects the new files.
3. Thumbnail Generation: The system will automatically generate thumbnails and place them in the thumbnails directory when the attachments are viewed in Specify.

For more information on how files are linked to attachments in Specify, see:

How does Specify 7 work with the Web Asset Server?

This article describes how attachments in Specify are linked to attachments on the Web Asset Server. Linking an Attachment to the Source When an attachment is uploaded to Specify, it is added to the database as a set of attachment records (in the attachment table and the joining attachment table, in this case collectionobjectattachment). In this example, I uploaded an attachment for a Collection Object with the catalog number 29499: [Puntius sophore (Hamilton, 1822) specimen] Cit…

Example: Using Rsync

[!tip]
If you are uploading from a Linux or macOS terminal, rsync is the most robust method as it supports resuming interrupted transfers.

# Syntax: rsync [options] /local/path/to/images/ user@asset-server.address:~/attachments/originals/

rsync -avzP /mnt/data/herbarium_images/ institute_user@assets.specifycloud.org:~/attachments/originals/