Specify-7 containers on RHEL with Podman not working correctly

Hi,
I’m trying to run the Dockerized Specify all-in-one installation on our office Red Hat Enterprise Linux dev/internal server.

I have replaced host.docker.internal by 172.17.0.1 in my custom docker-compose file as explained here on the Specify post: :video_: Install Specify 7 with Docker (macOS & Ubuntu) - Documentation / Configuration & Installation - Specify Community Forum (specifysoftware.org)

I was successful in connecting to the DB with DBeaver (port 3306) however the web portal gives me a NGINX error message (502 Bad Gateway).

Here are the Specify7 Docker logs:

File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/__init__.py", line 85, in Connect

    return Connection(*args, **kwargs)

  File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/connections.py", line 204, in __init__

    super(Connection, self).__init__(*args, **kwargs2)

django.db.utils.OperationalError: (2005, "Unknown MySQL server host 'mariadb' (-2)")

There seems to be an issue with Specify-7 trying to fin the DB.

My docker-compose.yml file:

version: '3.7'
services:

  mariadb:
    restart: unless-stopped
    image: mariadb
    command: --max_allowed_packet=1073741824
    ports:
      - "${MYSQL_EXTERNAL_PORT}:3306"
    volumes:
      - "database:/var/lib/mysql"
      - "./seed-database:/docker-entrypoint-initdb.d"
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}

  specify7:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7.7
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://172.17.0.1/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  specify7-worker:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7.7
    command: ve/bin/celery -A specifyweb worker -l INFO --concurrency=1
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://172.17.0.1/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service
    init: true
    volumes:
      - "attachments:/home/specify/attachments"
    environment:
      - SERVER_NAME=172.17.0.1
      - SERVER_PORT=80
      - ATTACHMENT_KEY=${ASSET_SERVER_KEY}
      - DEBUG_MODE=false

  specify6:
    image: specifyconsortium/specify6-service:6.8.02
    volumes:
      - "specify6:/volumes/Specify"

  nginx:
    restart: unless-stopped
    image: nginx
    ports:
      - "${SPECIFY_EXTERNAL_PORTAL_PORT}:80"
    volumes:
      - "static-files:/volumes/static-files:ro"
      - "specify6:/volumes/specify6:ro"

      - "./nginx/specify.conf:/etc/nginx/conf.d/default.conf:ro"

  report-runner:
    restart: unless-stopped
    image: specifyconsortium/report-runner

  redis:
    restart: unless-stopped
    image: redis:6.0

volumes:
  attachments: # the asset-servers attachment files
  database: # the data directory for mariadb
  specify6: # provides Specify 6 files to Specify 7 and the web server
  static-files: # provides Specify 7 static files to the web server

My .env file with the variables for the compose file:

YOUR_IP_HERE=s0-bsc-alfcwdev
SPECIFY_EXTERNAL_PORTAL_PORT=8090
MYSQL_EXTERNAL_PORT=3307
MYSQL_ROOT_PASSWORD=root
MYSQL_DATABASE=specify
MYSQL_USER=master
MYSQL_PASSWORD=master
SPECIFY_SECRET_KEY=ifindyourlackoffaithdisturbing
ASSET_SERVER_KEY=thiswillbeadaylongremembered

It appears that the issue lies with the Specify-7 service being unable to find the database. The error message "Unknown MySQL server host 'mariadb' (-2)" suggests that the hostname ‘mariadb’ specified in the environment variable DATABASE_HOST cannot be resolved.

You should start by considering if you need this section:

  mariadb:
    restart: unless-stopped
    image: mariadb
    command: --max_allowed_packet=1073741824
    ports:
      - "${MYSQL_EXTERNAL_PORT}:3306"
    volumes:
      - "database:/var/lib/mysql"
      - "./seed-database:/docker-entrypoint-initdb.d"
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}

The image: mariadb: specifies the Docker image to be used for the MariaDB container. If you are using a MariaDB installation outside of Docker, you can remove this section and the accompanying volume named database.

Once this is removed, you can set it up a new variable named something like {MYSQL_EXTERNAL_IP} which would need to be set to the IP of the MariaDB installation.

In the environment section for both the specify7 and specify7-worker, you can just set the database host and port to the IP and port of the MariaDB installation.

    environment:
      - DATABASE_HOST={MYSQL_EXTERNAL_IP}
      - DATABASE_PORT={MYSQL_EXTERNAL_PORT}

I recommend referencing the just-specify7 configuration which expects a local installation of MariaDB instead of a Docker one: https://github.com/specify/docker-compositions/blob/production/just-specify7/docker-compose.yml


It also looks like you are using the v7.7 image for both the specify7 and specify7-worker section. I recommend changing this to v7 so that you can receive the latest releases. Specify 7.7 over one year old and we just launched Specify 7.9.


If you are still encountering issues with the asset server, I recommend commenting out the ASSET_SERVER_URL and ASSET_SERVER_KEY just to confirm that Specify is up and running properly without. This will rule out other issues that may be causing a problem.

    environment:
      - DATABASE_HOST={MYSQL_EXTERNAL_IP}
      - DATABASE_PORT={MYSQL_EXTERNAL_PORT}
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      # - ASSET_SERVER_URL=http://172.17.0.1/web_asset_store.xml
      # - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}

Here’s a modified version of your docker-compose.yml file that may help you resolve the issues you are encountering!

version: '3.7'
services:

  specify7:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST={MYSQL_EXTERNAL_IP}
      - DATABASE_PORT={MYSQL_EXTERNAL_PORT}
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://172.17.0.1/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  specify7-worker:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7
    command: ve/bin/celery -A specifyweb worker -l INFO --concurrency=1
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST={MYSQL_EXTERNAL_IP}
      - DATABASE_PORT={MYSQL_EXTERNAL_PORT}
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://172.17.0.1/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service
    init: true
    volumes:
      - "attachments:/home/specify/attachments"
    environment:
      - SERVER_NAME=172.17.0.1
      - SERVER_PORT=80
      - ATTACHMENT_KEY=${ASSET_SERVER_KEY}
      - DEBUG_MODE=false

  specify6:
    image: specifyconsortium/specify6-service:6.8.02
    volumes:
      - "specify6:/volumes/Specify"

  nginx:
    restart: unless-stopped
    image: nginx
    ports:
      - "${SPECIFY_EXTERNAL_PORTAL_PORT}:80"
    volumes:
      - "static-files:/volumes/static-files:ro"
      - "specify6:/volumes/specify6:ro"

      - "./nginx/specify.conf:/etc/nginx/conf.d/default.conf:ro"

  report-runner:
    restart: unless-stopped
    image: specifyconsortium/report-runner

  redis:
    restart: unless-stopped
    image: redis:6.0

volumes:
  attachments: # the asset-servers attachment files
  specify6: # provides Specify 6 files to Specify 7 and the web server
  static-files: # provides Specify 7 static files to the web server

Hello,
I switched from v7.7 to v7. Thanks

I was trying to use the containerized version on MariaDB included in the all-in-one installation. Do you recommend using an external MariaDB for production deployment instead? If so, I will install MariaDB independently on my development server.

I also tried commenting out the ASSET_SERVER_URL and ASSET_SERVER_KEY in the compose file but getting the same log error messages as before. Not sure how commenting out the asset server config will help in finding the DB connection error reported in the logs or the NGINX 502 Bad Gateway error message?

$ docker logs specify7-basic-prototype-installation_specify7_1

Updating static files in /volumes/static-files/.
Applying Django migrations.
Traceback (most recent call last):
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2005, "Unknown MySQL server host 'mariadb' (-2)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 25, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/base.py", line 361, in execute
    self.check()
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/base.py", line 387, in check
    all_issues = self._run_checks(
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 64, in _run_checks
    issues = run_checks(tags=[Tags.database])
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/checks/registry.py", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/checks/database.py", line 10, in check_database_backends
    issues.extend(conn.validation.check(**kwargs))
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/mysql/validation.py", line 9, in check
    issues.extend(self._check_sql_mode(**kwargs))
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/mysql/validation.py", line 13, in _check_sql_mode
    with self.connection.cursor() as cursor:
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2005, "Unknown MySQL server host 'mariadb' (-2)")
[2023-10-03 14:59:07 +0000] [7] [INFO] Starting gunicorn 20.1.0
[2023-10-03 14:59:07 +0000] [7] [INFO] Listening at: http://0.0.0.0:8000 (7)
[2023-10-03 14:59:07 +0000] [7] [INFO] Using worker: sync
[2023-10-03 14:59:07 +0000] [14] [INFO] Booting worker with pid: 14
[2023-10-03 14:59:07 +0000] [15] [INFO] Booting worker with pid: 15
[2023-10-03 14:59:07 +0000] [16] [INFO] Booting worker with pid: 16

Maybe this can help… Why is specify7 listening to http://0.0.0.0:8000?
[2023-10-03 14:59:07 +0000] [7] [INFO] Listening at: http://0.0.0.0:8000 (7)

Thanks

specify7 is listening at http://0.0.0.0:8000 inside of the container because the Django server is configured to do that. So, if you were to make a request to specify7 container to port 8000 from outside (hence to 0.0.0.0 relative to specify7 container, but potentially completely different IP address for outside), Django will handle the requests (which is what we are actually doing, just that nginx server acts as a proxy in between).

I also tried replacing “172.17.0.1” with “host.docker.internal” in my compose file and adding an extra host “host.docker.internal:host-gateway” to specify7 container but it did not work. The Specify7 just shuts down.

My RHEL server being in production mode, this is a possible reason this approach did not work:

“The host has a changing IP address (or none if you have no network access). We recommend that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host. This is for development purpose and will not work in a production environment outside of Docker Desktop for Windows / Mac.”

Source: How to connect to the Docker host from inside a Docker container? | by Tim van Baarsen | Medium

version: '3.7'
services:

  mariadb:
    restart: unless-stopped
    image: mariadb:10.11
    command: --max_allowed_packet=1073741824
    ports:
      - "${MYSQL_EXTERNAL_PORT}:3306"
    volumes:
      - "database:/var/lib/mysql"
      - "./seed-database:/docker-entrypoint-initdb.d"
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}

  specify7:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7
    init: true
    depends_on:
      - mariadb
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://host.docker.internal:${SPECIFY_EXTERNAL_PORTAL_PORT}/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false
    extra_hosts:
      - "host.docker.internal:host-gateway"

  specify7-worker:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7
    command: ve/bin/celery -A specifyweb worker -l INFO --concurrency=1
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://host.docker.internal:${SPECIFY_EXTERNAL_PORTAL_PORT}/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service
    init: true
    volumes:
      - "attachments:/home/specify/attachments"
    environment:
      - SERVER_NAME=host.docker.internal
      - SERVER_PORT=${SPECIFY_EXTERNAL_PORTAL_PORT}
      - ATTACHMENT_KEY=${ASSET_SERVER_KEY}
      - DEBUG_MODE=false

  specify6:
    image: specifyconsortium/specify6-service:6.8.02
    volumes:
      - "specify6:/volumes/Specify"

  nginx:
    restart: unless-stopped
    image: nginx:alpine
    ports:
      - "${SPECIFY_EXTERNAL_PORTAL_PORT}:80"
    volumes:
      - "static-files:/volumes/static-files:ro"
      - "specify6:/volumes/specify6:ro"
      - "./nginx/specify.conf:/etc/nginx/conf.d/default.conf:ro"

  report-runner:
    restart: unless-stopped
    image: specifyconsortium/report-runner

  redis:
    restart: unless-stopped
    image: redis:6.0

volumes:
  attachments: # the asset-servers attachment files
  database: # the data directory for mariadb
  specify6: # provides Specify 6 files to Specify 7 and the web server
  static-files: # provides Specify 7 static files to the web server

The problem seems to be with rootless Podman on our RHEL. For security reasons the federal government is not allowed to use rootful Docker on their servers.

We have access to rootless Podman and podman-compose.
However, in rootless mode we can not get the podman-compose to work correctly. There seems to be container connectivity and pod issues in rootless mode.

My team has used rootless Podman before for individual (isolated) containers without issues… however for more complex cases needing pods, networks and secure intercontainer communication I think we will need deeper understanding of Podman before moving forward. There are significant differences with rootful Docker.

I will probably create a temporary dev VM on my side with rootless Podman to test things out. This will probably take some time to figure out. I’m new to Podman and have never work with pods and Kubernetes configurations files. If you have tips please let me know. I will also explore the Kompose tool that converts docker-compose files to Kubernetes configuration files.

I will certainly keep you informed once I find a solution for this.
Thanks again for availability and help.

Cheers
Héryk

PS. Why is running containers in rootful mode (ex: With Docker) is a security issue?: Rootless Containers from Scratch - Liz Rice, Aqua Security - YouTube