Specify7 in Docker keeps on failing on port 8080

Get Help
1

For the past months we’ve been struggling to get specify7 up and running in a new server environment using docker. The servers run RHEL8. We’re using the just-specify setup. The database and asset server run on separate servers. All necessary ports have been opened for and between the servers.

We’ve gotten the webserver up and running several times, but somehow, and this is a recurrent issue, whenever all docker containers are restarted, it gives a 502 Bad Gateway error.

Looking at the containers shows the following:

[root@specifyweb01fl specify]# docker compose ps
NAME                        IMAGE                                             COMMAND                  SERVICE             CREATED             STATUS                          PORTS
specify-nginx-1             docker.io/nginx                                   "/docker-entrypoint.…"   nginx               3 hours ago         Up 3 hours                      0.0.0.0:80->80/tcp, :::80->80/tcp
specify-redis-1             docker.io/redis:6.0                               "docker-entrypoint.s…"   redis               3 hours ago         Up 3 hours                      6379/tcp
specify-report-runner-1     docker.io/specifyconsortium/report-runner         "/docker-entrypoint.…"   report-runner       3 hours ago         Up 3 hours                      8080/tcp
specify-specify7-1          docker.io/specifyconsortium/specify7-service:v7   "/opt/specify7/docke…"   specify7            3 hours ago         Up 3 hours                      8000/tcp
specify-specify7-worker-1   docker.io/specifyconsortium/specify7-service:v7   "/opt/specify7/docke…"   specify7-worker     3 hours ago         Restarting (1) 42 seconds ago

For some reason, the specify7-worker keeps on wanting to restart. I am not sure why and if that is a problem…

Here follow the logs I picked out from the containers:
specify-specify7-1.log.2.txt (62.8 KB)
specify-redis-1.log.2.txt (1.4 KB)
specify-specify7-worker-1.log.2.txt (69.1 KB)
specify-report-runner-1.log.2.txt (15.4 KB)
specify-nginx-1.log.2.txt (2.4 KB)

There are errors reported in the logs for specify7-worker-1 and specify-specify7-1 that basically come down to the same thing:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='specifyapp01fl.unicph.domain', port=8080)

So there is no connection to the web asset server on port 8080, which is strange, because the firewall should not be blocking this port and the site has worked before without us changing this condition. Also, checking on the web asset server itself we can the service is listening to port 8080 is up and running:

[root@specifyapp01fl specify]# docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED       STATUS       PORTS                               NAMES
ca2758643207   nginx:alpine                              "/docker-entrypoint.…"   13 days ago   Up 13 days   0.0.0.0:80->80/tcp, :::80->80/tcp   specify-nginx-1
929897907b78   specifyconsortium/specify-asset-service   "/bin/sh -c 've/bin/…"   13 days ago   Up 13 days   8080/tcp                            specify-asset-server-1

This is the web asset server docker.compose.yml:

specifyapp01fl.docker-compose.yml (1.3 KB)

From the command line, I can also connect to the web asset server with no problems:

[root@specifyweb01fl specify]# curl http://specifyapp01fl.unicph.domain/web_asset_store.xml
<?xml version="1.0" encoding="UTF-8"?>
<urls>
    <url type="read"><![CDATA[http://specifyapp01fl.unicph.domain:8080/fileget]]></url>
    <url type="write"><![CDATA[http://specifyapp01fl.unicph.domain:8080/fileupload]]></url>
    <url type="delete"><![CDATA[http://specifyapp01fl.unicph.domain:8080/filedelete]]></url>
    <url type="getmetadata"><![CDATA[http://specifyapp01fl.unicph.domain:8080/getmetadata]]></url>
    <url type="testkey">http://specifyapp01fl.unicph.domain:8080/testkey</url>
</urls>

But we cannot actually connect to the web asset server via port 8080:

[root@specifyweb01fl specify]# curl http://specifyapp01fl.unicph.domain:8080/testkey
curl: (7) Failed to connect to specifyapp01fl.unicph.domain port 8080: Connection refused

But why is the specify7 server trying to do that anyway? Isn’t the web asset server itself only supposed to access port 8080 or also specify7?

Anyways, I’ve asked our IT department to make sure that this port is open between these servers (from web01fl to app01fl)

2

Hi @fedoras,

Thank you so much for providing in-depth information about the issue you are having! The docker-compose.yml and logs are especially useful.

Can you send us your modified docker-compose.yml file for the just-specify7 instance as well? This would be useful for reference as we investigate this further.

Would you be available for a Zoom session with one of our programmers and myself tomorrow, Friday, or sometime next week?

It looks like it would be most convenient to meet at 8:00 AM CDT / 3:00 PM CEST, so please let us know what works for you!

  • 2023-10-12T13:00:00Z
  • 2023-10-13T13:00:00Z
  • 2023-10-16T13:00:00Z or 2023-10-16T14:00:00Z
  • 2023-10-17T13:00:00Z or 2023-10-17T14:00:00Z

I have asked a couple of our developers to look into this before the meeting. If we have any comments before we meet, we will send them here.

Thank you!

1 Like
3

Hey, I may have a fix for your problem. I created a new asset server docker image for you to try out specifyconsortium/specify-asset-service:connection_fix

Restart the asset and nginx server, and try again to connect to http://specifyapp01fl.unicph.domain/web_asset_store.xml. Then try restarting your specify7 and specify7 worker containers to see if their connections work.

Let me know if that works for you and I’ll create a new branch in the web-asset-server repo

2 Likes
4

Hi there and thanks for the quick reponses…

The specify7 docker-compose is the following:

specifyweb01fl.docker-compose.yml (2.2 KB)

Please note that I have outcommented the URL & key to the web asset server again and whenever I do that, the site starts working again. So clearly, it stumbles on not being able to access the web asset server.

I would gladly be available for a meeting at the first time you mentioned (today) meaning at 2023-10-12T13:00:00Z

In the meantime, I will try the fix proposed by @alec.white

Thanks!

5

Hello again

Good news! Applying the fix by @alec.white has cured the problem so now I can access Specify7 while it no longer says “Attachment Server Unavailable” and I can get the attachment metadata displayed.

image

The only issue left now is the mapping of collections and media folders…

The original web asset server was set up with the use of the collection_dirs setting:

# Map collection names to directories.  Set to None to store
# everything in the same originals and thumbnail directories.  This is
# recommended unless some provision is made to allow attachments for
# items scoped above collections to be found.

COLLECTION_DIRS = {
  'NHMD Archaeozoology Comparative Birds': 'NHMD_Ornithology',
  'NHMD Archaeozoology Comparative Mammals': 'NHMD_Mammalogy',
  'NHMD Biocultural Collections': 'NHMD_Biocultural_Botany',
  'NHMD Vascular Plants':'NHMD_Botany',
  'NHMD Entomology': 'NHMD_Entomology',
  'NHMD Exhibitions': 'NHMD_Exhibitions',
  'NHMD Herpetology': 'NHMD_Herpetology',
  'NHMD Invertebrate Zoology': 'NHMD_Invertebrate_Zoology',
  'NHMD Mammalogy': 'NHMD_Mammalogy',
  'NHMD Ornithology': 'NHMD_Ornithology',
  'NHMD Quaternary Zoology': 'NHMD_Quaternary_Zoology',
  'NHMD Vertebrate Paleontology': 'NHMD_Paleontology',
  'NHMD Invertebrate Paleontology': 'NHMD_Paleontology',
  'NHMD Danekrae': 'NHMD_Paleontology',
  'NHMD Amber': 'NHMD_Amber',
  'NHMD Ichthyology': 'NHMD_Ichthyology',
  'FIMUS Havpattedyr': 'FIMUS_Havpattedyr',
  'MSJN Paleontology': 'MSJN_Paleontology',
  'Museum Salling - Naturhistorie': 'MUSERUM_Naturhistorie',
  'Naturama Samlinger': 'Naturama_Samlinger',
  'Museum Mors Geologisk Samling': 'MUSERUM_Naturhistorie',
  'OESM Collection': 'OESM_Collection',
  'NHMA Fresh water': 'NHMA_Fresh_water',
  'NHMA Entomology': 'NHMA_Fresh_water',
  'NHMA Malacology': 'NHMA_Fresh_water',
  'NHMA Invertebrates wet': 'NHMA_Fresh_water',
  'NHMA Monitoring': 'NHMA_Fresh_water',
  'NHMA Vertebrate': 'NHMA_Fresh_water'
}

As you can see in the docker.compose.yml for the asset server I gave earlier, I tried to add this there too, but then had to outcomment that line, because it was not accepted and gives this error message:

parsing /root/specify/docker-compose.yml: yaml: line 29: did not find expected key

For the life of me, I cannot tell how to make the dockerized web asset server work with this particular setting.

Do we have to give up having our images grouped into neat little folders?

6

One more thing: I tried adding a new image, but saving was blocked with the following error message:

image
image1253×562 61.6 KB

7

Feel free to send a zoom link to: fedor.steeman@snm.ku.dk

8

Hi @fedoras,

The developer is not in the office yet, but we can set up a call as soon as he arrives! Apologies for the delay.

9

I am not a developer, but from what I gather in our Dockerfile on GitHub, it does not appear that the docker-compose.yml file supports setting up COLLECTION_DIRS at this time. This may be possible with some modification on our end.

See these lines in the Dockerfile:

While the SERVER_NAME, SERVER_PORT, ATTACHMENT_KEY, and DEBUG_MODE environmental variables are entered, there is no support in our current docker deployment method for the user to submit the collection directories through the compose file.

The RUN command could likely be modified to passthrough the you are setting in the compose file, but it is currently structured as it needs to be stored in the settings.py file rather than the environmental variable format the docker-compose.yml and Dockerfile are set up to expect.

1 Like
10

No problems! We can try another time.

11

Interesting… :open_mouth:

12

Hi @fedoras,

I’ve gone ahead and created a feature request for this on the web-asset-server repository and I have discussed it with one of our developers.

Thank you for your report!

It seems that running it outside of Docker is the only official way to specify the COLLECTION_DIRS and set it permanently at this time. Modifying the settings.py file inside the container and restarting would work temporarily, but upon restart the same steps would need to be taken again.

13

Our developer pushed a new docker image with updates in the settings.py file that may solve this issue. He let me know the following:

The image on Docker Hub is specifyconsortium/specify-asset-service:collection_dirs

services:
  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service:collection_dirs
    init: true

You would only need to add double quotes around the COLLECTION_DIRS dictionary value in your docker-compose.yml file.

For example:

services:

  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service
    init: true
    volumes:
      # Store all attachments outside the container, in a separate
      # volume
      - "attachments:/mediedata/"
    environment:
      # Replace this with the URL at which asset server would be
      # publicly available
      - SERVER_NAME=specifyapp01fl.unicph.domain
      - SERVER_PORT=80
      - ATTACHMENT_KEY=Of2et3dOp5Bon8he
      - DEBUG_MODE=false
      - COLLECTION_DIRS = "{
          'NHMD Archaeozoology Comparative Birds': 'NHMD_Ornithology',
          'NHMD Archaeozoology Comparative Mammals': 'NHMD_Mammalogy',
          'NHMD Biocultural Collection': 'NHMD_Biocultural_Botany',
          'NHMD Vascular Plants':'NHMD_Botany',
          'NHMD Entomology': 'NHMD_Entomology',
          'NHMD Exhibitions': 'NHMD_Exhibitions',
          'NHMD Herpetology': 'NHMD_Herpetology',
          'NHMD Invertebrate Zoology': 'NHMD_Invertebrate_Zoology',
          'NHMD Mammalogy': 'NHMD_Mammalogy',
          'NHMD Ornithology': 'NHMD_Ornithology',
          'NHMD Quaternary Zoology': 'NHMD_Quaternary_Zoology',
          'NHMD Vertebrate Paleontology': 'NHMD_Paleontology',
          'NHMD Invertebrate Paleontology': 'NHMD_Paleontology',
          'NHMD Danekrae': 'NHMD_Paleontology',
          'NHMD Amber': 'NHMD_Amber',
          'FIMUS Havpattedyr': 'FIMUS_Havpattedyr',
          'MSJN Paleontology': 'MSJN_Paleontology',
          'Museum Salling - Naturhistorie': 'MUSERUM_Naturhistorie',
          'Naturama Samlinger': 'Naturama_Samlinger',
          'Museum Mors Geologisk Samling': 'MUSERUM_Naturhistorie',
          'OESM Collection': 'OESM_Collection',
          'NHMA Fresh water': 'NHMA_Fresh_water',
          'NHMA Entomology': 'NHMA_Fresh_water',
          'NHMA Malacology': 'NHMA_Fresh_water',
          'NHMA Invertebrates wet': 'NHMA_Fresh_water',
          'NHMA Monitoring': 'NHMA_Fresh_water',
          'NHMA Vertebrate': 'NHMA_Fresh_water'
        }"

volumes:
  attachments: # the asset-servers attachment files
14

Thanks! I’ve tried it but get the following error message:

[root@specifyapp01fl specify]# docker compose up -d
parsing /root/specify/docker-compose.yml: yaml: line 19: mapping values are not allowed in this context

This is my docker compose file now:

asset-server.collectiondirs.docker-compose.yml (2.3 KB)

BTW I noticed a shift of the server port from 8080 to 80. Is this supposed to be?

15

Hi fedoras, I think the issue is that the .yml syntax as presented is not valid (at least according to my IDE). Perhaps something like this would work instead (at least it passed through my linter without complaining). Note I have shortened COLLECTION_DIRS for simplicity.

Based on syntax - Environment variable assignment in Docker Compose - colon way - Stack Overflow there doesn’t seem to be a difference between using = or : under environment, but perhaps it may not work with the way Specify has the container setup. Just thought I would throw this out there in case it does end up being a quick fix. The quotes around the dictionary don’t seem to matter for the yml, but may try both with and without quotes to see how it is interpreted by python.

    services:

      asset-server:
        restart: unless-stopped
        image: specifyconsortium/specify-asset-service:collection_dirs
        init: true
        volumes:
          # Store all attachments outside the container, in a separate
          # volume
          - "attachments:/mediedata/"
        environment:
          SERVER_NAME: specifyapp01fl.unicph.domain
          SERVER_PORT: 80
          ATTACHMENT_KEY: r6cNZJ3KaS6AxbJm
          DEBUG_MODE: false
          COLLECTION_DIRS: "{ 'NHMD Archaeozoology Comparative Birds': 'NHMD_Ornithology', 'NHMD Archaeozoology Comparative Mammals': 'NHMD_Mammalogy'}"

    volumes:
      attachments: # the asset-servers attachment files
17

Thanks, Mark!

So I now made sure that the yml of the docker-compose is valid. I tried again and it downloaded the hotfix image and started running.

However, now I’m getting “502 Bad Gateway” whenever I try to download an image!

The docker.compose.yml:

version: '3.7'
services:

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"

    volumes:
      - "static-files:/volumes/static-files:ro"
      - "specify6:/volumes/specify6:ro"
      - "webpack-output:/volumes/webpack-output:ro"

      - "./nginx.conf:/etc/nginx/conf.d/default.conf:ro"

  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service:collection_dirs
    init: true
    volumes:
      # Store all attachments outside the container, in a separate volume
      - "attachments:/media/"
    environment:
      # Replace this with the URL at which asset server would be publicly available
      - SERVER_NAME=specifyapp01fl.unicph.domain
      - SERVER_PORT=8080
      - ATTACHMENT_KEY=Of2et3dOp5Bon8he
      - DEBUG_MODE=false
      - COLLECTION_DIRS = "{'NHMD Biocultural Collection':'NHMD_Biocultural_Botany','NHMD Vascular Plants':'NHMD_Botany'}"

volumes:
  attachments: # the asset-servers attachment files
  database: # the data directory for mariadb
  specify6: # provides Specify 6 files to Specify 7 and the web server
  static-files: # provides Specify 7 static files to the web server
  webpack-output:

The logs:

[root@specifyapp01fl specify]# docker logs --tail 30 specify-nginx-1
2023/10/17 03:35:16 [notice] 1#1: using the "epoll" event method
2023/10/17 03:35:16 [notice] 1#1: nginx/1.25.2
2023/10/17 03:35:16 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)
2023/10/17 03:35:16 [notice] 1#1: OS: Linux 4.18.0-348.20.1.el8_5.x86_64
2023/10/17 03:35:16 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/10/17 03:35:16 [notice] 1#1: start worker processes
2023/10/17 03:35:16 [notice] 1#1: start worker process 21
2023/10/17 03:35:48 [error] 21#21: *1 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.175.116.37, server: localhost, request: "GET /fileget?coll=NHMD+Biocultural+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg HTTP/1.1", upstream: "http://192.168.96.3:8080/fileget?coll=NHMD+Biocultural+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg", host: "specifyapp01fl.unicph.domain"
10.175.116.37 - - [17/Oct/2023:03:35:48 +0000] "GET /fileget?coll=NHMD+Biocultural+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0" "-"
2023/10/17 03:35:48 [error] 21#21: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 10.175.116.37, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "specifyapp01fl.unicph.domain", referrer: "http://specifyapp01fl.unicph.domain/fileget?coll=NHMD+Biocultural+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg"
10.175.116.37 - - [17/Oct/2023:03:35:48 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://specifyapp01fl.unicph.domain/fileget?coll=NHMD+Biocultural+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0" "-"
2023/10/17 03:35:56 [error] 21#21: *4 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.175.116.37, server: localhost, request: "GET /fileget?coll=NHMD+Collection+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg HTTP/1.1", upstream: "http://192.168.96.3:8080/fileget?coll=NHMD+Collection+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg", host: "specifyapp01fl.unicph.domain"
10.175.116.37 - - [17/Oct/2023:03:35:56 +0000] "GET /fileget?coll=NHMD+Collection+Botany&type=O&filename=611265.jpg&downloadname=611265.jpg HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0" "-"

And:

[root@specifyapp01fl specify]# docker logs --tail 1000 specify-asset-server-1
Traceback (most recent call last):
  File "server.py", line 15, in <module>
    import settings
  File "/home/specify/settings.py", line 75
    COLLECTION_DIRS = eval(COLLECTION_DIRS)

It looks like something goes wrong with passing the collection_dirs parameter to the settings file inside docker…

I tried to enter the container with bash to see if things could be manually adjusted, but got the following message:

[root@specifyapp01fl specify]# docker exec -it specify-asset-server-1 bash
Error response from daemon: Container f6da6225e983a9d0e71bd0f28b9fbf61a47a2ee6a61fd108ad191bba595a835d is restarting, wait until the container is running

It looks like the asset server container has issues and keeps on restarting:

[root@specifyapp01fl specify]# docker ps
CONTAINER ID   IMAGE                                                     COMMAND                  CREATED          STATUS                          PORTS                               NAMES
2568851f7219   nginx:alpine                                              "/docker-entrypoint.…"   52 seconds ago   Up 51 seconds                   0.0.0.0:80->80/tcp, :::80->80/tcp   specify-nginx-1
f6da6225e983   specifyconsortium/specify-asset-service:collection_dirs   "/bin/sh -c 've/bin/…"   52 seconds ago   Restarting (1) 22 seconds ago                                       specify-asset-server-1
18

Another things is that this image:

specifyconsortium/specify-asset-service:collection_dirs

… doesn’t appear to have the connection fix that this image does:

specifyconsortium/specify-asset-service:connection_fix

It would be nice if both fixes could be combined into a single image.

19

There appears there is some work re-configuring the docker-compose.yml and how environmental variables are handled in this pull request. Perhaps the COLLECTION_DIRS could be added as an optional parameter in the example.env.

Also, as this transition to an .env file is taking place, documentation on which parameters are required vs optional would be nice. For example, the nginx components inside docker someone doesn’t necessarily have to run, but are nice to have in the example in case someone just wants a grab and go version.

I could see the COLLECTION_DIRS concept being useful for others, and it would also probably be better for you, and the specify team, if it was implemented at this stage while the example env is being setup anyways, so as not to have to maintain separate docker containers (as you mention in your next post)

20

Thanks Fedor for pointing out the difference in the branches. I pushed a new docker image with both changes to connection_fix. Try that out for now. I’m looking into the problem in more detail today and hope to come to a solution.

1 Like
21

Thank you and looking forward!

Another issue that I have to point out is the fact that on the side of Specify7, the collection name passed to the web asset server is currently incorrect. Although I am logged in to the Biocultural Botany collection, the collection name passed to the web asset server is the first one in the list of collections, which happens to be Vertebrate Paleontology.

http://specifyapp01fl.unicph.domain/fileget?coll=NHMD+Vertebrate+Paleontology&type=O&filename=610981.jpg&downloadname=610981.jpg

In actuality, it ought to be rendered as such to get it to work:

http://specifyapp01fl.unicph.domain/fileget?coll=NHMD+Biocultural+Collection&type=O&filename=610981.jpg&downloadname=610981.jpg

As long as the currently selected collection is not passed via the query string parameters to the web asset server, the COLLECTION_DIRS setting won’t work.

I believe there is a special setting in Specify7 (“WEB_ATTACHMENT_COLLECTION”) to be set to TRUE in order to enable this. This would also mean we should be able to add this setting to the docker.compose.yml of Specify7 and have this passed through to inside the container.

Looking at the settings from inside the container, I can see that this value is derived from environmental variable “ASSET_SERVER_COLLECTION”:

root@b5f167ef552a:/opt/specify7/settings# cat local_specify_settings.py
import os
DATABASE_NAME = os.environ['DATABASE_NAME']
DATABASE_HOST = os.environ['DATABASE_HOST']
DATABASE_PORT = os.environ.get('DATABASE_PORT', '')
MASTER_NAME = os.environ['MASTER_NAME']
MASTER_PASSWORD = os.environ['MASTER_PASSWORD']
DEPOSITORY_DIR = '/volumes/static-files/depository'
REPORT_RUNNER_HOST = os.getenv('REPORT_RUNNER_HOST', '')
REPORT_RUNNER_PORT = os.getenv('REPORT_RUNNER_PORT', '')
WEB_ATTACHMENT_URL = os.getenv('ASSET_SERVER_URL', None)
WEB_ATTACHMENT_KEY = os.getenv('ASSET_SERVER_KEY', None)
WEB_ATTACHMENT_COLLECTION = os.getenv('ASSET_SERVER_COLLECTION', None)
SEPARATE_WEB_ATTACHMENT_FOLDERS = os.getenv('SEPARATE_WEB_ATTACHMENT_FOLDERS', None)
CELERY_BROKER_URL = os.getenv('CELERY_BROKER_URL', None)
CELERY_RESULT_BACKEND = os.getenv('CELERY_RESULT_BACKEND', None)
CELERY_TASK_DEFAULT_QUEUE = os.getenv('CELERY_TASK_QUEUE', DATABASE_NAME)
ANONYMOUS_USER = os.getenv('ANONYMOUS_USER', None)

Would it help simply adding this setting to the docker.compose.yml?