Install Specify 7 with Docker

[mcruz]

Forward from @Specify

Thank you to Matthew Cruz (@mcruz) at The University of Michigan for writing this tutorial! We are grateful for your efforts in providing quality education and resources to Specify users.

This guide outlines the process of moving a Specify 6.8.00 database into Specify 7. If you already have updated your database to Specify 6.8.01 or 6.8.02 you can safely ignore the steps involved in upgrading the database from 6.8.00 to 6.8.01.

If you have any questions about this process you can create a post in the Get Help category or email support@specifysoftware.org.

Important: You must have access to the docker-compositions repository on GitHub to follow this tutorial.

Click here to request access or email support@specifysoftware.org with your GitHub username, member institution or collection, and any additional questions or notes for us!

If you are interested in alternative deployment options you can see all of the supported methods of installing Specify 7 here.

---

Cloning a database for local use

When you first install MySQL it creates a local MySQL server and puts it on localhost:3306.

Then you can connect to that server and create databases.

Make a SQL Dump of an existing Specify database

Use “SpBackupRestore 6.8.00” to make a sql dump of a 6.8.00 collection database.

This application can be found in the “Specify” application folder in which Specify was installed.

On a Mac it can be found as shown in this image:

1. Launch the application

2. Log in with the “IT Username and Password”. You may need to obtain this from an administrator.

3. Verify paths then click “Backup”

   You may want to customize the “Backup Location” path since this is where the SQL dump file will be saved.

4. Monitor progress of the backup.

5. When the backup is complete, the window will return to this view:

6. Quit Specify Backup & Restore

Obtain the Specify 7 Docker GIT repo

1. Clone the git repo at: https://github.com/specify/docker-compositions

2. Open the local copy of the git repo in SublimeText.

   Shown below is the “README.md” file in the “seed-database” folder.

   

Configure the DB seed

1. Move the SQL dump file into the /all-in-one/seed-database folder and rename it to something appropriate.

   Backups Location

   Note that the *.stats file can be deleted.

   

   seed-database location

   Notice that the backup was renamed from {timestamp}.sql to {databaseName}{specifyVersion}.sql

   So in this case ummp6800.sql refers to a Specify 6.8.0.0 dump of the ummp database.

   This naming convention is arbitrary however be sure not to not include symbols like . or - in the filename as that seems to cause problems.

   

   docker-compose.yml

   The relevant section of the docker-compose file that loads the MariaDB with the seed database is shown here:

   

Customize the “docker-compose.yml” file

mariadb Container

This image shows the lines that must be added or modified to configure the MariaDB SQL database.

1. Add the “ports” block

   “13306:3306” is interpreted as follows:

   13306 is the port number you will use on the computer you are working on. This equates to “localhost:13306”.

   “3306” is the port number that the MariaDB SQL server is going to be running on within the Docker container.

   This line says to route requests made to “localhost:13306” on the computer you are working on to “localhost:3306” on the Docker container.

   This is how you are able to use tools like MySQL or Specify on your computer to connect to the database within the Docker container.

2. Configure the database account-information block

   Set the value of “MYSQL_ROOT_PASSWORD” to a secure new password.

Set the value of “MYSQL_DATABASE” to the name of the Specify database.

Set the value of “MYSQL_USER” and “MYSQL_PASSWORD” to the “IT Username and Password” that was originally configured on the seed database.

specify7 Container

This image shows the lines that must be modified to configure the Specify 7 application.

1. Configure the database account-information block

   Set the value of “DATABASE_NAME” to the name of the Specify database you indicated when configuring the “mariadb” section. They must match.

   Set the values of “MASTER_NAME” and “MASTER_PASSWORD” to match what you indicated when configuring the “mariadb” section. They must match “MYSQL_USER” and “MYSQL_PASSWORD”.

2. Comment out the “ASSET_SERVER_URL” and “ASSET_SERVER_KEY” properties if you are not planning to configure the asset server at this time.

   If you choose not to configure the asset server but forget to comment out these lines, then the setup will fail and you will experience errors when attempting to run Specify7.

3. For debugging purposes, it is recommended to set “LOG_LEVEL” to “INFO” and “SP7_DEBUG” to true.

   For production releases, “LOG_LEVEL” should be set to “WARN” and “SP7_DEBUG” to false otherwise too much processing will be spent logging detailed messages and debug operations, which can affect the performance of the application.

specify7-worker Container

This image shows the lines that must be modified to configure the Specify 7 worker. WorkBench upload and validate operations are carried out by this separate worker process.
For information on what the worker does, see this page: GitHub - specify/specify7: Specify 7.

This section has some special properties that are different from the “specify7” container, however the marked lines in the image above are customized the same way as the “specify7” section.

asset-server Container

Comment out the “asset-server” if you are not planning to configure the asset server at this time.
If you choose not to configure the asset server but forget to comment out these lines, then the setup may fail and you may experience errors when attempting to run Specify7.

Launch Docker

Docker must be installed and running in order for the following steps to work.

Shown below is Docker running. It does not have any containers running. Your instance of Docker may have containers unrelated to this Specify setup running if you manage other applications.

Run docker-compose

The following steps will use Docker to process the docker-compose file and create all the containers necessary to run Specify 7.

1. Using terminal, cd to the directory of the git repository and the “all-in-one” folder.

2. Run docker-compose up -d to execute the build process.

Verify that the build finished successfully

The image below shows no errors and “done” listed next to each container.

View the created containers in Docker

Shown below is the container footprint for Specify 7. Notice that the “all-in-one_specify6_1” container has a status of “EXITED (0)”. This is correct because that container starts up, runs and configures some files for Specify 7 to use, and then shuts down and is not needed any further.

Monitor the mariadb container logs to verify when the database has been seeded.

You can monitor in real-time any messages printed to the console as the container generates them.

1. Run docker ps to view all the running Docker containers in order to obtain the “CONTAINER ID” of the container that you want to monitor the logs of.

   

2. Run docker logs -f 081981dc2da3 in order to continuously monitor the logs for the “mariadb” container.

   The -f flag means “follow” and it is used to see in real-time any messages printed to the console as the container generates them.

3. Check the status of the seeding.

Shown below: The seeding is in-progress.

The highlighted line in the image below shows the last message printed where the container will pause for seeding the SQL database with the SQL dump file that was provided. Depending on the size of the database seed, this could take several minutes. In this example, it takes just one minute.

4. Seeding complete!

The lines below ending with the highlighted line show that the seeding has successfully finished.

9. Use MySQL Workbench to view the DB

Use MySQL Workbench to connect to the MariaDB SQL database in the container and verify that the database has been seeded by the SQL dump you specified.

1. Go to “Database > Manage Connections”

2. Click the “New” button and fill in the information as shown.

   Connection Name: S7-All-in-One

   Hostname: localhost

   Port: 13306

   Username: root

   

3. Click “Test Connection”

   You may get the following “Connection Warning”.

   It is OK.

   Click “Continue Anyway”.

   

   You should get the following success message. If you do not, then you must debug your configuration.

   

4. Click “Close”

5. Click on the connection you just created.

   Your “MySQL Connections” home screen should now list your newly created connection as shown below. Click on it to open a connection to the database.

   

   Not-Ready State: You connected too soon: The database has not been seeded yet

   The seeding can take a few minutes so if you attempt to connect before the DB is seeded and ready you will likely see the following.

   The image below shows that there are no databases listed on the left under “SCHEMAS”.

   

   Ready-State

   The database is listed and has been populated with data.

   Shown here is the “ummp” database with its “Tables” list expanded.

   A query was run on the “accession” table to verify that data exists in the database resulting from the seeding.

   

6. Install Specify 6.8.01

7. Download Specify 6.8.01

8. Install Specify 6.8.01 but be careful NOT to install it as a replacement on top of the 6.8.00 version.

   

9. Name the installation directory to be obviously related to 6.8.01 such as: “6.8.01-Specify-6.8.01”.

   

10. Go into the /bin folder and rename every app that was installed by prefixing it with “6.8.01-” as shown in the image below. This will help you to avoid accidentally running the wrong version of Specify. Shown below I renamed the Sp

    

Upgrade the 6.8.0.0 database to 6.8.0.1

At this point in time you will need to upgrade the database.

1. Launch the “6.8.01-Specify4GB” app

2. Enter connection information

   Be sure to indicate the server as “localhost” and the port number as “13306” so that Specify can connect to the database in the mariadb Docker container.

   WARNING: If you make a mistake and do not change the server and port number correctly, then you might accidentally upgrade the wrong database if you have a MySQL server running on your actual machine [outside of a container].

   

3. Specify Schema Update

   Click “Continue”

   

4. Backup

   Click “Continue”

   

5. Enter the “IT Login” credentials

   These will match what you put in the “docker-compose.yml” file.

   

6. Monitor the Schema Update progress

   In-Progress

   

   Finished

   

7. Finish logging into 6.8.01

Note that you will get some warnings about the attachment server not being present if you chose not to configure the attachment server in the docker-compose.yml file.

Attempting to log into S7 at this point will yield an error

This is likely because the S7.7 container probably expected to work on the 6.8.01 database but was seeded with 6.8.00 and has been running alongside it despite we just completed updating to 6.8.01.

In order to log in, you must launch the app running at localhost. You can do this by going to a web browser and typing in localhost in the address bar or you can launch it via a button from Docker next to the nginx container. NGINX is the web server container that is used to capture and route requests to the Specify 7 app server container.

The list of tables in MySQL Workbench shows it is missing:

Restart S7 in order to pick up the missing tables

Click the “RESTART” button next to the following S7 containers:

1. Restart “all-in-one_specify7_1”

2. Restart “all-in-one_specify7-worker_1”

   

3. Confirm spuserpolicy is now present

15. Verify installed Specify version

16. Go to localhost in the web browser.

17. Log in

    

18. Go to User Tools by clicking on your username in the top right of the navigation menu and click on About Specify.

19. Confirm that your expected Specify versions are installed.

Create a 6.8.01 SQL dump to use as a database seed

This will allow us to go back and configure the mariadb container to seed with the 6.8.01 updated version of the database so that we do not have to keep repeating the update process because it was starting with a 6.8.00 database at the beginning of this document.

1. Launch the “6.8.01-SpBackupRestore” app and log in with the “IT Username and Password” that was configured in the docker-compose.yml file.

2. Click “Backup”

3. Monitor progress

4. Finished…

Sample API Queries

1. http://localhost/documentation/api/tables/

2. http://localhost/api/specify/collectionobject/?catalognumber__gte=50516&limit=2

Query for all collection objects with a catalognumber greater than or equal to 50516 and return a limit of 2 items.

[mcruz]

@mcruz

Restart S7 in order to pick up the missing tables

This forum post explains how to address update issues if Specify 7 is complaining about missing tables:

Upgrading to Specify 7.7+: HTTP 500 error

If you are updating to Specify 7.7 or newer, Specify is supposed to create new tables related to the security and accounts features. If you encounter the following error upon updating to 7.9.3, you should follow the same instructions: "exception": "ProgrammingError", "message": "1146", "data": "Table 'db_name.uniquenessrule' doesn't exist" In your docker-compose.yml file, you should use your root user credentials instead of the master user credentials for the first run in Specify 7. You just…