Specify7 crashes with certain edits to Preparation type subview def

[fedoras]

Hi there,

I am trying to edit the collectionobject data entry form for Ichthyology and have been getting some strange errors here.

The names of preparation types in question can be quite long, so I am trying to rearrange the sub viewdef of Preparation table to accommodate:

First I tried to increase the column span for the field in question like below:

  <cell type="field" id="7" name="prepType" uitype="combobox" picklist="PrepType" colspan="3"/>

But for some reason I get a 500 error message, whenever I tried to save:

I then tried to increase the width of column 3 (currently now 200 px), this is working, but the elements to the far right are getting pushed out of view if I increase more (which I need to). So I wanted to move these to a separate row instead:

              <row>
                 <cell type="label" labelfor="7"/>
                 <cell type="field" id="7" name="prepType" uitype="combobox" picklist="PrepType"/>
                 <cell type="label" labelfor="1"/>
                 <cell type="field" id="1" uitype="plugin" name="this" initialize="name=PartialDateUI;df=preparedDate;tp=preparedDatePrecision" uifieldformatter="Date"/>
                 <cell type="field" id="isOnLoan" name="isOnLoan" uitype="checkbox" label="IS_ON_LOAN" readonly="true"/>
             </row>
             <row>
                 <cell type="label" labelfor="3"/>
                 <cell type="field" id="3" name="countAmt" uitype="text"/>
                 <cell type="label" labelfor="2"/>
                 <cell type="field" id="2" name="preparedByAgent" uitype="querycbx" initialize="name=Agent"/>
             </row>
             <row> <!-- newly added rows with elements to the far right move here -->
                 <cell type="label" labelfor="4"/>
                 <cell type="field" id="4" name="storage" uitype="querycbx" initialize="name=Storage"/>
                 <cell type="command" id="ShowLoansBtn" name="ShowLoansBtn" label="SHOW_INTERACTIONS" ignore="true"/>
                 <cell type="subview" id="attachments" viewname="ObjectAttachment" name="preparationAttachments" initialize="align=right;btn=true;icon=PreparationAttachment"/>
             </row>

The strange thing is that I am then getting the exact same error response.

Looking at the server logs it says the following:

 [Tue Feb 27 06:10:54.399608 2024] [wsgi:error] [pid 27767] [remote 10.175.116.37:54130]   File "/home/specify/currentversion/specify7/ve/lib64/python3.6/site-packages/MySQLdb/connections.py", line 276, in query
 [Tue Feb 27 06:10:54.399610 2024] [wsgi:error] [pid 27767] [remote 10.175.116.37:54130]     _mysql.connection.query(self, query)
 [Tue Feb 27 06:10:54.399613 2024] [wsgi:error] [pid 27767] [remote 10.175.116.37:54130] django.db.utils.DataError: (1406, "Data too long for column 'NewValue' at row 1")

Why is this happening and how do I achieve my goals now?

This is my complete view file:
fish.views.xml (72.3 KB)

Specify 7 Version: v7.7.5

[Grant]

Hi @fedoras,

First, thank you so much for providing your form definitions, screenshots, and the server logs! It always makes diagnosing issues like these much easier and we really appreciate it.

The problem you are facing is caused by a known issue with the audit log implementation.

"Data too long for column 'NewValue' at row 1"

The issue is with truncation of field values when saving them to the audit log, particularly when adding a unicode character. Truncation in Python counts characters, but the MySQL field limit is based on the number of bytes. This discrepancy can lead to truncated values exceeding the field size due to unicode characters being represented by multiple bytes.

Here’s the issue that we are tracking on GitHub:

Saving LARGE app resources causes exception due to auditing. (`"Data too long for column 'NewValue' at row 1"`)

Now for the difficult part– the workaround requires that you either disable the audit log entirely, split the form definition into several files, or that you delete and create a new form definition each time you make changes.

1. Delete the form definition, create a new one, and use the ‘Load File’ function to import it in Specify 7 every time you want to change the form definition. When using this approach, it no longer prevents you from saving since it doesn’t attempt to save the value to the audit log table!

2. Split the form definition file into several files so that it no longer exceeds the maximum allowed field value in the audit log when updated.

3. (Not recommended) Disable auditing by editing the ‘Remote Preferences’ at the global level and setting auditing.do_audits=true to auditing.do_audits=false. You can stop auditing just for the duration of the time you are doing form edits.

In the future, we are planning expand those fields to handle larger app resource data values. This will most likely require a schema update to resolve. I apologize for the trouble and we sincerely appreciate your report!

[fedoras]

Hi there and thanks for your many interesting suggestions!

Strangely enough I am still having issues…

@Grant

3. (Not recommended) Disable auditing by editing the ‘Remote Preferences’ at the global level and setting auditing.do_audits=true to auditing.do_audits=false. You can stop auditing just for the duration of the time you are doing form edits.

This is the first thing I tried (outside of office hours to be sure i did not interrupt general audit logging), but for some reason the errors persist… I have including a more extensive logging of the error here:
error.log (17.4 KB)

@Grant

1. Delete the form definition, create a new one, and use the ‘Load File’ function to import it in Specify 7 every time you want to change the form definition. When using this approach, it no longer prevents you from saving since it doesn’t attempt to save the value to the audit log table!

I tried this in the test environment and I’m glad I did because I seem to be unable to use the “Load File” function when recreating the views file after deleting it. For some reason, it won’t save the recreated file.

My steps are:

1. Delete the fish.views file so now there is none for the collection under viewsets:

2. Create a new Resource file under viewsets called “fish.views”

3. Click Load file and browse to the file in question:

4. After opening the file, its contents appear in the editing pane:

5. Pressing “Save” and the button successfully greys out:

6. When I refresh the page, however, the pane is empty again:

7. If I instead click on another resource file, e.g. “exhibits.views” it warns me that changes have not been saved:

This is very peculiar…

The remaining option is 2, and I am willing to try, but unsure how to go about this. What parts would I put where? How small must I make the largest possible file?

[Grant]

Hi @fedoras,

This issue was recently encountered and I was able to use the “Load File” solution without any issue on v7.9.3. I imagine the discrepancy in behavior could be due to the difference in Specify 7 versions between your installation and mine.

Upon further review, restarting the Specify 7 instance may be necessary to change the auditing behavior. I guess that ultimately reinforces why that approach is not recommended. Thank you for providing the logs so we can investigate further!

After completing step 6, could you please try using the “Load File” function since the app resource has been created?

As for Option 2, the maximum value depends on the NewValue field in the spauditlogfield table. This field is of data type text with a maximum value of 65,535 bytes. Unicode characters can consist of multiple bytes so the truncated value (done in Python) can exceed that value. An XML form definition under 63 KB should be safe to import, but I cannot say definitely.