Attachment server unavailable in all-in-one Specify-7 docker-compose setup on Windows

Get Help
1

Hello, I’m working with the Specify-7 all-in-one docker-compose setup. I was successful in getting everything to work correctly except for the asset-server. When I try to add an image attachement to a collection object in the Specify-7 portal I get a “Attachment server unavailable” error message. Anyone else has this bug or a solution? Thanks

PS. I’m using Windows 10 Entreprise with Docker Desktop

image
image920×262 6.48 KB

2

Hi @Heryk,

Can you add the following line in this image to your /etc/hosts (instructions) file on the system hosting Specify:

127.0.0.1       host.docker.internal

image
image1554×1158 155 KB

After this change is made, you should be able to upload attachments properly.

If you are still having trouble, please send us a copy of your docker-compose.yml file so that I can troubleshoot further!

Thank you!

3

Hi Grant,

I commented out this line in my hosts file:

#127.0.0.1 kubernetes.docker.internal

And added your line before saving the file and rebooting my machine.

127.0.0.1 host.docker.internal

Unfortunately it did not fix my issue. I have the same error message from the attachment server. Everything else seems to work just fine.

I pasted below my docker-compose.yml file with the associated .env file. To facilitate security and deployments on different machines, I put all the docker-compose variables in the .env file. I thus have different .env files on my development, pre-prod and production machines but I’m always using the same docker-compose.yml file.

.env file:

YOUR_IP_HERE=192.168.112.1
SPECIFY_EXTERNAL_PORTAL_PORT=8090
MYSQL_EXTERNAL_PORT=3307
MYSQL_ROOT_PASSWORD=root
MYSQL_DATABASE=specify
MYSQL_USER=master
MYSQL_PASSWORD=master
SPECIFY_SECRET_KEY=ifindyourlackoffaithdisturbing
ASSET_SERVER_KEY=thiswillbeadaylongremembered

docker-compose.yml file:

version: '3.7'
services:

  mariadb:
    restart: unless-stopped
    image: mariadb
    command: --max_allowed_packet=1073741824
    ports:
      - "${MYSQL_EXTERNAL_PORT}:3306"
    volumes:
      - "database:/var/lib/mysql"
      - "./seed-database:/docker-entrypoint-initdb.d"
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}

  specify7:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7.7
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://host.docker.internal/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  specify7-worker:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7.7
    command: ve/bin/celery -A specifyweb worker -l INFO --concurrency=1
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://host.docker.internal/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service
    init: true
    volumes:
      - "attachments:/home/specify/attachments"
    environment:
      - SERVER_NAME=host.docker.internal
      - SERVER_PORT=80
      - ATTACHMENT_KEY=${ASSET_SERVER_KEY}
      - DEBUG_MODE=false

  specify6:
    image: specifyconsortium/specify6-service:6.8.02
    volumes:
      - "specify6:/volumes/Specify"

  nginx:
    restart: unless-stopped
    image: nginx
    ports:
      - "${SPECIFY_EXTERNAL_PORTAL_PORT}:80"
    volumes:
      - "static-files:/volumes/static-files:ro"
      - "specify6:/volumes/specify6:ro"

      - "./nginx/specify.conf:/etc/nginx/conf.d/default.conf:ro"

  report-runner:
    restart: unless-stopped
    image: specifyconsortium/report-runner

  redis:
    restart: unless-stopped
    image: redis:6.0

volumes:
  attachments: # the asset-servers attachment files
  database: # the data directory for mariadb
  specify6: # provides Specify 6 files to Specify 7 and the web server
  static-files: # provides Specify 7 static files to the web server
5

Can you add

    extra_hosts:
      - "host.docker.internal:host-gateway"

to specify7 service and retry?

6

Hi!
I added your 2 lines into my compose file et restarted my containers… However I still get the same error message.

Just to make sur I modified the right thing…
Here is a copy/paste of the “Specify7” section of my docker-compose.yml file that I updated…

  specify7:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7.7
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://host.docker.internal/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false
    extra_hosts:
      - "host.docker.internal:host-gateway"
7

that is correct!

  1. Can you check if going to localhost/web_asset_store.xml and host.docker.internal/web_asset_store.xml returns a result? If it doesn’t, can you make sure that the /etc/hosts file contains 127.0.0.1 host.docker.internal? Additionally, can you share the result you get by going tohost.docker.internal/web_asset_store.xml?
  2. Can you send the Nginx configuration file you are using?
8
  1. I’m using port 8090 so…

http://localhost:8090/web_asset_store.xml returns this:

<?xml version="1.0" encoding="UTF-8"?>
<urls>
    <url type="read"><![CDATA[http://host.docker.internal:80/fileget]]></url>
    <url type="write"><![CDATA[http://host.docker.internal:80/fileupload]]></url>
    <url type="delete"><![CDATA[http://host.docker.internal:80/filedelete]]></url>
    <url type="getmetadata"><![CDATA[http://host.docker.internal:80/getmetadata]]></url>
    <url type="testkey">http://host.docker.internal:80/testkey</url>
</urls>

http://host.docker.internal/web_asset_store.xml returns the same thing as above.

I’m on Windows so I modified the file C:\Windows\System32\drivers\etc\hosts … here is a copy of it…

# Added by Docker Desktop
132.156.208.198 host.docker.internal
132.156.208.198 gateway.docker.internal

#Added for Specify7 (Docker) by Heryk
127.0.0.1 host.docker.internal
  1. Here is a copy of the nginx config I’m using… I have not modified it.
server {
    listen 80;
    server_name localhost;
    root /usr/share/nginx;

    # serve static files directly
    location /static/ {
        root /volumes;
        rewrite ^/static/config/(.*)$ /specify6/config/$1 break;
        rewrite ^/static/depository/(.*)$ /static-files/depository/$1 break;
        rewrite ^/static/(.*)$ /static-files/frontend-static/$1 break;
    }

    # proxy these urls to the asset server
    location ~ ^/(fileget|fileupload|filedelete|getmetadata|testkey|web_asset_store.xml) {
        client_max_body_size 0;
        resolver 127.0.0.11 valid=30s;
        set $backend "http://asset-server:8080";
        proxy_pass $backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # proxy everything else to specify 7
    location / {
        resolver 127.0.0.11 valid=30s;
        set $backend "http://specify7:8000";
        proxy_pass $backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Thanks for your help!

9

Can the log message help?
$ docker logs specify7-basic-prototype-installation-specify7-1

Updating static files in /volumes/static-files/.
Applying Django migrations.
[27/Sep/2023 17:49:36] [ERROR] [specifyweb.attachment_gw.views:170] Failed fetching asset server configuration
Operations to perform:
  Apply all migrations: accounts, auth, contenttypes, notifications, permissions, sessions, workbench
Running migrations:
  No migrations to apply.
[2023-09-27 17:49:41 +0000] [7] [INFO] Starting gunicorn 20.1.0
[2023-09-27 17:49:41 +0000] [7] [INFO] Listening at: http://0.0.0.0:8000 (7)
[2023-09-27 17:49:41 +0000] [7] [INFO] Using worker: sync
[2023-09-27 17:49:41 +0000] [31] [INFO] Booting worker with pid: 31
[2023-09-27 17:49:41 +0000] [32] [INFO] Booting worker with pid: 32
[2023-09-27 17:49:41 +0000] [33] [INFO] Booting worker with pid: 33
[27/Sep/2023 17:49:43] [ERROR] [specifyweb.attachment_gw.views:170] Failed fetching asset server configuration
[27/Sep/2023 17:49:43] [ERROR] [specifyweb.attachment_gw.views:170] Failed fetching asset server configuration
[27/Sep/2023 17:49:47] [WARNING] [django.request:222] Not Found: /context/app.resource
[27/Sep/2023 17:49:47] [ERROR] [specifyweb.attachment_gw.views:170] Failed fetching asset server configuration
[27/Sep/2023 17:49:47] [WARNING] [django.request:222] Not Found: /context/app.resource

PS. The assets-server logs are empty!

10

They do, the logs point towards specify7 not being able to connect to the asset server!

The asset server logs are still empty if you make a request to them? If you make a request to host.docker.internal/web_asset_store.xml does that show up in asset server logs?

Could you restart just the specify7 container, while keeping the other containers running?

Additionally, keep the asset server logs open (docker logs --follow CONTAINER_ID) while restarting specify7, and check if any requests show up there.

Thank you for being so patient!

11

Hi,
Doing my best to answerer your questions… here goes…

>>> The asset server logs are still empty if you make a request to them?
I think the only way to make a request to it is through the web portal? If so, yes the asset-server logs stay empty.

>>> If you make a request to host.docker.internal/web_asset_store.xml does that show up in asset server logs?
How can I make a request directly to “host.docker.internal” from outside docker? I tried this… http://localhost:8090/web_asset_store.xml and the logs now have content…
Here it is…

$ docker logs specify7-basic-prototype-installation-asset-server-1
Bottle v0.12.18 server starting up (using PasteServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

172.19.0.1 - - [27/Sep/2023:19:08:17 +0000] "GET /web_asset_store.xml HTTP/1.0" 200 455 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.2045.41"

>>> Could you restart just the specify7 container, while keeping the other containers running?
Done. Same error

>>> Additionally, keep the asset server logs open (docker logs --follow CONTAINER_ID) while restarting specify7, and check if any requests show up there.
Done. Nothing more shows up. :frowning:

Thanks
Héryk

12

Can do the following steps, and share the results?

Exec into the specify7 docker container by docker exec -u 0 -it CONTAINER_NAME_OR_ID /bin/bash

  1. Check if the environment variable is correct. Do this by typing env and looking for ASSET_SERVER_URL in the result
  2. type in apt install curl and type in y if it asks for confirmation.
  3. type in curl $ASSET_SERVER_URL.

Doing the above will help us know if the specify docker container can access the asset server correctly or not.

13

Here goes…

  1. ASSET_SERVER_URL=http://192.168.4.45/web_asset_store.xml
  2. Done
root@46940b1c5684:/opt/specify7# curl $ASSET_SERVER_URL
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>

I think I see the issue but not sure how to fix it. The port is missing. Shouldn’t the correct URL be http://192.168.4.45:8090/web_asset_store.xml ?

14

Does going to http://192.168.4.45:8090/web_asset_store.xml from the browser give a result back? It should, while removing the port should not. Same for http://host.docker.internal/web_asset_store.xml (it shouldn’t return anything, unless port is specified - Attachment server unavailable in all-in-one Specify-7 docker-compose setup on Windows - #8 by Heryk – can you check this again, because you mentioned that host.docker.internal/web_asset_store.xml returns result without port)

But yes, the external port that nginx is mapped to, should be used. So, SPECIFY_EXTERNAL_PORTAL_PORT is your case (8090)

15

Additionally, for the asset-server service definition, can you change the port from 80 to instead use SPECIFY_EXTERNAL_PORTAL_PORT? It wouldn’t work otherwise.

16

Yes if I try to access the following URL outside of Docker through a web browser I get a result…

<?xml version="1.0" encoding="UTF-8"?>
<urls>
    <url type="read"><![CDATA[http://192.168.4.45:8080/fileget]]></url>
    <url type="write"><![CDATA[http://192.168.4.45:8080/fileupload]]></url>
    <url type="delete"><![CDATA[http://192.168.4.45:8080/filedelete]]></url>
    <url type="getmetadata"><![CDATA[http://192.168.4.45:8080/getmetadata]]></url>
    <url type="testkey">http://192.168.4.45:8080/testkey</url>
</urls>

And if I remove the port from the URL I get an error message.

Same for host.docker.internal…

This works… http://host.docker.internal:8090/web_asset_store.xml

But this returns an error message… http://host.docker.internal/web_asset_store.xml

17

Awesome! Yes, go ahead and change ASSET_SERVER_URL=http://host.docker.internal/web_asset_store.xml to instead be http://host.docker.internal:${SPECIFY_EXTERNAL_PORTAL_PORT}/web_asset_store.xml.

Additionally, don’t forget to change the asset server port (in asset server definition - from 80 to ${SPECIFY_EXTERNAL_PORTAL_PORT} too, unless you have it explicitly running at port 8080, in which case, use ASSET_SERVER_URL=http://host.docker.internal:8080/web_asset_store.xml for specify service)

18

You are awesome! It worked! :partying_face:
I can now upload attachments! Wow!
Thank you so much!

I will post my complete compose file and .env file below if anyone is having a similar bug and would like check out our configuration! :slight_smile:

docker-compose.yml file:

version: '3.7'
services:

  mariadb:
    restart: unless-stopped
    image: mariadb:10.11
    command: --max_allowed_packet=1073741824
    ports:
      - "${MYSQL_EXTERNAL_PORT}:3306"
    volumes:
      - type: bind
        source: ${DB_VOLUME}
        target: /var/lib/mysql
      - "./seed-database:/docker-entrypoint-initdb.d"
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}

  specify7:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7.7
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://host.docker.internal:${SPECIFY_EXTERNAL_PORTAL_PORT}/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false
    extra_hosts:
      - "host.docker.internal:host-gateway"

  specify7-worker:
    restart: unless-stopped
    image: specifyconsortium/specify7-service:v7.7
    command: ve/bin/celery -A specifyweb worker -l INFO --concurrency=1
    init: true
    volumes:
      - "specify6:/opt/Specify:ro"
      - "static-files:/volumes/static-files"
    environment:
      - DATABASE_HOST=mariadb
      - DATABASE_PORT=3306
      - DATABASE_NAME=${MYSQL_DATABASE}
      - MASTER_NAME=${MYSQL_USER}
      - MASTER_PASSWORD=${MYSQL_PASSWORD}
      - SECRET_KEY=${SPECIFY_SECRET_KEY}
      - ASSET_SERVER_URL=http://host.docker.internal:${SPECIFY_EXTERNAL_PORTAL_PORT}/web_asset_store.xml
      - ASSET_SERVER_KEY=${ASSET_SERVER_KEY}
      - REPORT_RUNNER_HOST=report-runner
      - REPORT_RUNNER_PORT=8080
      - CELERY_BROKER_URL=redis://redis/0
      - CELERY_RESULT_BACKEND=redis://redis/1
      - LOG_LEVEL=WARNING
      - SP7_DEBUG=false

  asset-server:
    restart: unless-stopped
    image: specifyconsortium/specify-asset-service
    init: true
    volumes:
      - type: bind
        source: ${ATTACHMENTS_VOLUME}
        target: /home/specify/attachments
    environment:
      - SERVER_NAME=${YOUR_IP_HERE}
      - SERVER_PORT=${SPECIFY_EXTERNAL_PORTAL_PORT}
      - ATTACHMENT_KEY=${ASSET_SERVER_KEY}
      - DEBUG_MODE=false

  specify6:
    image: specifyconsortium/specify6-service:6.8.02
    volumes:
      - "specify6:/volumes/Specify"

  nginx:
    restart: unless-stopped
    image: nginx
    ports:
      - "${SPECIFY_EXTERNAL_PORTAL_PORT}:80"
    volumes:
      - "static-files:/volumes/static-files:ro"
      - "specify6:/volumes/specify6:ro"
      - "./nginx/specify.conf:/etc/nginx/conf.d/default.conf:ro"

  report-runner:
    restart: unless-stopped
    image: specifyconsortium/report-runner

  redis:
    restart: unless-stopped
    image: redis:6.0

volumes:
  specify6: # provides Specify 6 files to Specify 7 and the web server
  static-files: # provides Specify 7 static files to the web server

.env file:

YOUR_IP_HERE=192.168.4.45

SPECIFY_EXTERNAL_PORTAL_PORT=8090

MYSQL_EXTERNAL_PORT=3307

MYSQL_ROOT_PASSWORD=root

MYSQL_DATABASE=specify

MYSQL_USER=master

MYSQL_PASSWORD=master

SPECIFY_SECRET_KEY=ifindyourlackoffaithdisturbing

ASSET_SERVER_KEY=thiswillbeadaylongremembered

DB_VOLUME=/c/Users/hjulien/DockerMounts/mariadb-specify

ATTACHMENTS_VOLUME=/c/Users/hjulien/DockerMounts/attachments-specify
2 Likes
19

PS. Please let me know if you have any other recommendations to my docker compose configuration. Thanks :slight_smile:

1 Like
20

Very helpful thread! I came across the same problem and solved it by reading this thread.

Can I open the container and see the attachments? Here is the container ID : 8d18aa222175

1 Like
21

Hello @drguiritan,

In the “asset-server” section of the docker-compose file, I defined the volume configuration for the assets folder, specifying both the source and target directories.

    volumes:
      - type: bind
        source: ${ATTACHMENTS_VOLUME}
        target: /home/specify/attachments

The attachment files are actually located outside the container (at the “source” path) but are accessible within the container at the “target” path (/home/specify/attachments). This setup allows you to view the attachment files without needing to open the container, making it much simpler to back up your attachment folder.

The “source” path is specified in the .env file, so you may want to check that file to see where your attachments are saved on your server or computer. In my example, I was running Docker on Windows and saved the files in a directory I created under my username.

ATTACHMENTS_VOLUME=/c/Users/hjulien/DockerMounts/attachments-specify