Hi @phb,
Specify is designed to be a “single source of truth,” where each database record is intrinsically linked to its corresponding digital asset. The database record contains metadata and a pointer to the asset file’s location. As I mentioned, the attachments were always uploaded successfully, and the pointer record was created in the database. The failure did not occur during data capture; rather, it was due to a server configuration error that directed the transfer to the wrong destination. Specify does not regularly check if all assets are accessible, as this verification happens in real-time when the asset is previewed or viewed in the app. If an asset is unavailable, the preview will not display, and the user will see a “404 Not Found” message along with a description of the issue (e.g., Missing original) when attempting to view the full resolution asset.
To reiterate, your core database records were never at risk. For all our cloud-hosted institutions, we perform nightly backups that include a multi-step verification process. Before the backup runs, our system automatically confirms that all database tables are present. After it completes, the file is checked again for integrity and completeness. Any failure in this process triggers an alert to our technical team. We ensure we always have a sound, complete, and reliable backup of your core records. By default, daily backups are stored for one week, monthly backups for one year, and yearly backups are kept indefinitely.
We ensure data integrity through a multi-layered approach that begins at the database level. The database schema enforces strict relational integrity, which makes it impossible to create orphaned records or link to non-existent data. Layered on top of that, there is business logic in the code to apply more complex validation and institution-specific rules during data entry. This includes uniqueness rules on critical fields, such as catalog numbers, which are enforced to prevent the creation of duplicate records within a collection. The audit log captures changes to records made in the software directly.
The issue we resolved was confined to the storage location of the asset files. Now everything should be consolidated on your server, restoring it as the single, authoritative location. To verify the data is present, you just need to verify that all pointers (attachmentLocation in attachment) correspond to an asset file on your server.
Given this, your plan to run data integrity checks is an implementation of best practices that we fully encourage and endorse. You can use the Specify APIs (or use your direct SQL access) to automate these checks and provide precisely the kind of verification and peace of mind that your curators need. If you develop a workflow that may be useful for other users or if you have any questions, please let us know!
Best,
Grant