Agent merging "Unknown column 'spmerging.status'"

Hi there,

Starting from an empty database created with Specify 6.8.03, we imported many agents with Workbench Sp7.9.6.2.
Some of them are duplicated from our previous database and when attempting to use the merge records button we get this error:

"OperationalError", "message": "1054", "data": "Unknown column spmerging.status' in 'field list'

Full error message attached Specify 7 Crash Report - 2024-08-01T13_05_00.205Z.txt (546.0 KB)

Our spmerging table does exist though and columns look as follow:

MariaDB [aublet3]> show columns from spmerging;
+--------------------+--------------+------+-----+---------+----------------+
| Field              | Type         | Null | Key | Default | Extra          |
+--------------------+--------------+------+-----+---------+----------------+
| id                 | int(11)      | NO   | PRI | NULL    | auto_increment |
| name               | varchar(256) | NO   |     | NULL    |                |
| taskid             | varchar(256) | NO   |     | NULL    |                |
| mergingstatus      | varchar(256) | NO   |     | NULL    |                |
| response           | longtext     | NO   |     | NULL    |                |
| table              | varchar(256) | NO   |     | NULL    |                |
| newrecordid        | int(11)      | YES  |     | NULL    |                |
| newrecordata       | longtext     | YES  |     | NULL    |                |
| oldrecordids       | longtext     | YES  |     | NULL    |                |
| collection_id      | int(11)      | NO   | MUL | NULL    |                |
| specifyuser_id     | int(11)      | NO   | MUL | NULL    |                |
| timestampcreated   | datetime(6)  | NO   |     | NULL    |                |
| timestampmodified  | datetime(6)  | NO   |     | NULL    |                |
| createdbyagent_id  | int(11)      | YES  | MUL | NULL    |                |
| modifiedbyagent_id | int(11)      | YES  | MUL | NULL    |                |
+--------------------+--------------+------+-----+---------+----------------+

There is indeed no status column. Did we miss any schema or database update on the way?

Cheers,
Philippe

I just got this same message on the demo server when browsing https://sp7demofish.specifycloud.org/specify/view/agent/1514/.

Specify 7 Crash Report - 2024-08-01T22_51_44.015Z.txt (324.8 KB)

Hi @pverley,

Based on the error and the columns that are included in your spmerging table, it looks like an older version of the record merging migrations was run on your database before the final release.

For reference, this is what the table should look like:

Is it possible you were or are using the edge Docker tag for your deployment of Specify 7? If so, you may have pulled our work-in-progress changes on the production branch on GitHub.

This migration issue was likely introduced sometime before the release of Specify 7.9.0 as we solidified which fields were included when that arrived.

As @markp mentioned, we were used the edge Docker tag on our demo instance during that time period which means it is also affected by this issue.

Make sure to backup your database before performing any changes via SQL!

To resolve this on my end for the demo instance, I had to drop the tables that were introduced after the merging (7.9.0) update (including spmerging, uniquenessrule, and uniquenessrule_fields.

Note that I would back these tables up individually before dropping them if they contained data. Because they were empty, I was able to remove them without any issue.

drop table sp7demofish.spmerging;
drop table sp7demofish.uniquenessrule;
drop table sp7demofish.uniquenessrule_fields;

Once I did this, the migrations ran successfully after I ran docker compose up. See the logs from the back-end container (specifycloud-sp7demofish-1) in Docker:

specifycloud-sp7demofish-1         | Updating static files in /volumes/static-files/.
specifycloud-sp7demofish-1         | Applying Django migrations.
specifycloud-sp7demofish-1         | Operations to perform:
specifycloud-sp7demofish-1         |   Apply all migrations: accounts, attachment_gw, auth, businessrules, contenttypes, notifications, permissions, sessions, workbench
specifycloud-sp7demofish-1         | Running migrations:
specifycloud-sp7demofish-1         |   Applying businessrules.0001_initial... OK
specifycloud-sp7demofish-1         |   Applying businessrules.0002_default_unique_rules... OK
specifycloud-sp7demofish-1         |   Applying notifications.0003_spmerging... OK
specifycloud-sp7demofish-1         |   Applying notifications.0004_rename_merge_policy... OK
specifycloud-sp7demofish-1         |   Applying notifications.0005_auto_20240530_1512... OK
specifycloud-sp7demofish-1         |   Applying notifications.0006_localityupdate_localityupdaterowresult... OK
specifycloud-sp7demofish-1         |   Applying permissions.0006_add_dataset_create_recordset_permission... OK
specifycloud-sp7demofish-1         |   Applying permissions.0007_add_stats_edit_permission... OK
specifycloud-sp7demofish-1         |   Applying permissions.0008_attachment_import_role... OK
specifycloud-sp7demofish-1         | [2024-08-02 16:12:51 +0000] [8] [INFO] Starting gunicorn 22.0.0
specifycloud-sp7demofish-1         | [2024-08-02 16:12:51 +0000] [8] [INFO] Listening at: http://0.0.0.0:8000 (8)
specifycloud-sp7demofish-1         | [2024-08-02 16:12:51 +0000] [8] [INFO] Using worker: gthread
specifycloud-sp7demofish-1         | [2024-08-02 16:12:52 +0000] [16] [INFO] Booting worker with pid: 16

The migrations were then re-run properly, ensuring the tables were created correctly. Now, the https://sp7demofish.specifycloud.org/ instance can merge records as expected, and I have updated the backup used for restoring the instance to prevent this issue from occurring again.


@pverley What steps you need to take to resolve this issue depend on the status of your current migrations. Can you either send us a backup of your database or send us the contents of the django_migrations table?

Thank you!

Dear Grant,

Thank you so much for setting me on track for fixing this issue. After close investigation I definitely pulled specify7 images from edge tag around end of 2023, for I was trying out the Weblate process for French language.

Backing up and deleting the three aforementioned table (spmerging, uniquenessrule, uniquenessrule_fields) did not seemed to work in my case, maybe the DB was cluttered with other issues? … (kept getting error like unknow column spmerging.mergingstatus in the backend logs when attempting django migrations).

I decided to start afresh and got rid of every SP7 tables (as if I were just migrating from SP6 to SP7 with current content).
:warning: It might sound like an extreme option, but we have only been using SP7 in production for a few weeks here in CAY herbarium, so we still have a very “young” system that do not rely yet heavily on SP7-only features.

First I did a full backup of my DB. Then I backed up the tables one by one to make it easier to insert them back to the DB one by one later on if needed:

TABLES=("attachmentdataset" "auth_group" "auth_group_permissions" "auth_permission" "continentcodes" "countryinfo" "django_content_type" "django_migrations" "django_session" "geoname" "notifications_message" "spdataset" "splibraryrole" "splibraryrolepolicy" "spmerging" "sprole" "sprolepolicy" "spuserexternalid" "spuserpolicy" "spuserrole" "uniquenessrule" "uniquenessrule_fields")

FOLDER="sp7-tables_$(date '+%Y%m%d')"
DBNAME=aublet3
mkdir $FOLDER

for table in "${TABLES[@]}"; do
        echo $table;
        mariadb-dump -u root -p $(DBNAME) $table > $FOLDER/$table.sql
done

Then I deleted them all but the ones from GEO plugins (continentcodes, countryinfo, geoname do not seem to be added by django migrations)

SET foreign_key_checks = 0;
DROP TABLE IF EXISTS attachmentdataset, auth_group, auth_group_permissions, auth_permission, django_content_type, django_migrations, django_session, notifications_message, spdataset, splibraryrole, splibraryrolepolicy, spmerging, sprole, sprolepolicy, spuserexternalid, spuserpolicy, spuserrole, uniquenessrule, uniquenessrule_fields
SET foreign_key_checks = 1;

Then restarted the system (docker compose up -d). Migrations went smoothly:

specify7-1         | Operations to perform:
specify7-1         |   Apply all migrations: accounts, attachment_gw, auth, businessrules, contenttypes, notifications, permissions, sessions, workbench
specify7-1         | Running migrations:
specify7-1         |   Applying accounts.0001_initial... OK
specify7-1         |   Applying accounts.0002_auto_20211223_1206... OK
specify7-1         |   Applying accounts.0003_auto_20220621_1541... OK
specify7-1         |   Applying attachment_gw.0001_initial... OK
specify7-1         |   Applying contenttypes.0001_initial... OK
specify7-1         |   Applying contenttypes.0002_remove_content_type_name... OK
specify7-1         |   Applying auth.0001_initial... OK
specify7-1         |   Applying auth.0002_alter_permission_name_max_length... OK
specify7-1         |   Applying auth.0003_alter_user_email_max_length... OK
specify7-1         |   Applying auth.0004_alter_user_username_opts... OK
specify7-1         |   Applying auth.0005_alter_user_last_login_null... OK
specify7-1         |   Applying auth.0006_require_contenttypes_0002... OK
specify7-1         |   Applying auth.0007_alter_validators_add_error_messages... OK
specify7-1         |   Applying auth.0008_alter_user_username_max_length... OK
specify7-1         |   Applying auth.0009_alter_user_last_name_max_length... OK
specify7-1         |   Applying auth.0010_alter_group_name_max_length... OK
specify7-1         |   Applying auth.0011_update_proxy_permissions... OK
specify7-1         |   Applying auth.0012_alter_user_first_name_max_length... OK
specify7-1         |   Applying businessrules.0001_initial... OK
specify7-1         |   Applying businessrules.0002_default_unique_rules... OK
specify7-1         |   Applying permissions.0001_initial... OK
specify7-1         |   Applying permissions.0002_role_rolepolicy_userrole... OK
specify7-1         |   Applying permissions.0003_auto_20220321_1445... OK
specify7-1         |   Applying permissions.0004_intialize_defaults... OK
specify7-1         |   Applying permissions.0004_auto_20220407_1927... OK
specify7-1         |   Applying permissions.0005_merge_20220414_1451... OK
specify7-1         |   Applying notifications.0001_initial... OK
specify7-1         |   Applying notifications.0002_message_read... OK
specify7-1         |   Applying notifications.0003_spmerging... OK
specify7-1         |   Applying notifications.0004_rename_merge_policy... OK
specify7-1         |   Applying notifications.0005_auto_20240530_1512... OK
specify7-1         |   Applying notifications.0006_localityupdate_localityupdaterowresult... OK
specify7-1         |   Applying permissions.0006_add_dataset_create_recordset_permission... OK
specify7-1         |   Applying permissions.0007_add_stats_edit_permission... OK
specify7-1         |   Applying permissions.0008_attachment_import_role... OK
specify7-1         |   Applying sessions.0001_initial... OK
specify7-1         |   Applying workbench.0001_initial... OK
specify7-1         |   Applying workbench.0002_spdataset_visualorder... OK
specify7-1         |   Applying workbench.0003_auto_20210218_1256... OK
specify7-1         |   Applying workbench.0004_auto_20210219_1131... OK
specify7-1         |   Applying workbench.0005_auto_20210428_1634... OK

Copied back the spdataset table:

mariadb -u root -p $(DBNAME) < sp7-tables_20241003/spdataset.sql

Et voilĂ  !

:warning: Again I’m certainly not recommending what I did. I’m just mentioning it for the record in the knowledge base. @Grant please issue a red card :red_square: if this is too much of a dangerous option, for later readers.

Thanks again for your help, and thanks to @markp for backing up this issue :+1:

Cheers,
Philippe V.

1 Like

@pverley In case it was unintentional, the reply above contains the value for the mariadb password.

Thanks for your watchful eye :hugs:

Actually that is the database name, not the password. But in any case it is better to post generic code, and nothing too specific from our system.

mariadb -u user_name -p db_name

The -p just says that the password will be asked in the command line (unsafe though according to mariadb manual)

Ah yes, my mistake, I didn’t know that it handled the space in a specific way (was reading the docs after posting). Thanks for teaching me something new!

Hehe, now we know the password credentials for you is “ubc_specify” or something :sweat_smile: /s

Time to change the database name to password123! to really confuse people!

1 Like