Moving from Sp6 to Docker Sp7 Any query fails

Get Help
1

I’m on the test phase of migrating NOU collection from Specify 6 to Specify 7, using so far docker composition “just-specify-7”.

Any query, existing or new one (even the dumbest one based on catalog number), systematically fails with same error message:

Error occurred fetching from `http://localhost/stored_query/ephemeral/`
Invalid response code 500. Expected 200.
## ParseError
*not well-formed (invalid token): line 179, column 29
* Content
"None"

I tried on local machine and distant server, with exactly same issue. It feels like I’m missing a simple point.
Thanks for any clue !

2

Hi @pverley,

Can you provide the docker-compose.yml file you are using so we can investigate further?
Have you tried other versions of Specify 7 as well?

3

Please find attached the docker compose file (I’ve just hidden the password). I barely modified it from original github version, except that I commented out the attachment server.
My local mariadb server is configured such as ITUserName and MasterUsername have been granting permissions from any host (I have not mastered docker yet so I do not control how IP is assigned to just-specify7 container, so for testing purpose I opened everything).

MariaDB [virot]> SELECT User, Host FROM mysql.user;
+----------------+-----------+
| User           | Host      |
+----------------+-----------+
| ITUserName     | %         |
| MasterUsername | %         |
| ITUserName     | *         |
| mariadb.sys    | localhost |
| mysql          | localhost |
| root           | localhost |
+----------------+-----------+

Which user is running the queries in Sp7 ?

I have not yet tried any other Docker recipe. I will try asap all-in-one and keep you posted whether it solves the issue.

Thanks for helping out.
Regards
docker-compose_just-specify7.yml (2.3 KB)

1 Like
4

I just gave it a try with docker compose from all-in-one and I’ve got the very same issue with every query.

To initialize all-in-one container I had copied an extract of the NOU database (schema 2.10) in folder all-in-one/seed-database.

5

Hey,

Can you attach the crash report that you get when the query fails?

6

Hi @pverley,

Can you try to deploy version 7.9.0 and inform us if the issue persists?

To do this, replace the image: for the specify7 and specify7-worker sections in the docker-compose.yml file with the following:

    image: specifyconsortium/specify7-service:v7.9.0

Once you have made this change, pull the image and rebuild the containers. If you are still encountering the issue after using this version, please let us know!

7

@pverley,

Could you please download and send us the DataObjFormatters app resource(s) from the App Resources viewer in Specify?

8

Thank you so much for your assistance.

I did update specify7 and specify7-worker services to v7.9.0 as suggested, but the issue persists. Specify 7 Crash Report - Query - 2023-11-29T22 19 43.440Z.txt (1.2 MB)

Interestingly when I tried to open the DataObjFormatters app resource(s), it also crashed with the following StackTrace:

Traceback (most recent call last):
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/specify7/specifyweb/specify/views.py", line 40, in wrapped
    return view(request, *args, **kwargs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/views/decorators/cache.py", line 31, in _cache_controlled
    response = viewfunc(request, *args, **kw)
  File "/opt/specify7/specifyweb/specify/views.py", line 73, in view
    return dispatch_func(request, *args, **kwargs)
  File "/opt/specify7/specifyweb/specify/api.py", line 202, in collection_dispatch
    resp = HttpResponse(toJson(data), content_type='application/json')
  File "/opt/specify7/specifyweb/specify/api.py", line 61, in toJson
    return json.dumps(obj, cls=JsonEncoder)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/specify7/specifyweb/specify/api.py", line 57, in default
    return obj.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 5027: invalid continuation byte

Specify 7 Crash Report - DataObjFormatters - 2023-11-29T22 21 48.742Z.txt (531.9 KB)

Error message reminded my of similar issue on the forum, with special characters encoding issue.

And a quick search on SO confirm that the root of the problem might be accentuated letters (é, è, à, etc., sorry for our French !) in the database ? It feels that we have a lead.

9

Hi there,

I’m unearthing this thread.
I’ve updated the docker-compose.xml to latest for both specify7 and specify7-worker services. I still have systematic encoding error when running queries or when trying to display the DataObjFormatters.
I could do with some guidance to understand in which part of the database do I have encoding issue ? and how to fix it ?
My colleagues aren’t having the issues with the queries with SP6.

Thanks for your help,
Philippe V.

10

Hi @pverley,

Can you send an export of your spappresource and spappresourcedata tables in your database? From there, we can extract the DataObjFormatters resource and investigate if an encoding problem elsewhere may be causing an issue.

11

Thank you. Attached an export of both spappresource and spappresourcedata tables.

specify-nou_spappresource-spappresourcedata_20240515.zip (73.9 KB)

12

Hi @pverley,

Thank you for your patience! I was able to verify that the UTF-8 file was damaged or incorrectly formed, which meant I had to remove the resource manually.

I suspect this resource was formed incorrectly when it was updated. This can happen in extremely rare circumstances when a table’s format or aggregation was edited using the Schema Configuration tool in Specify 6.

First, I made a backup of the resource and opened it in BBEdit. I recovered the contents and created a correctly formatted resource (for use later):

NOU DataObjFormatters.xml (41.4 KB)

After this, I removed the invalid app resource using SQL. Note that this specifically removes the app resource and app resource data for this record only.

DELETE FROM spappresourcedata WHERE SpAppResourceDataID = 7;
DELETE FROM spappresource WHERE SpAppResourceID = 7;

After this, I was able to verify that I could safely search and view all other app resources. I then recreated the DataObjFormatters app resouce by adding it under Botany in the app resource viewer hierarchy.

image
image1240×1332 98.5 KB

Once I created the new Record Formatters app resource, I used the “Load File” button to import the fixed XML I attached earlier in this message.

image
image848×952 82.8 KB

Now the record formatters appear as expected and the issue is resolved! Queries are running successfully and records are formatted as one would expect.

Thank you again for your patience and for sharing a copy of your database with me!

13

It works like a charm, thank you so much !

1 Like