OSU’s Specify instances have been getting a lot of bot traffic that we’d like to find a way to work around. The problem is that we really like having the anonymous guest user account enabled so that people don’t need to sign in to see our data. This is important to our stakeholders for accessibility of data.
Is there a way at present to limit the API access of an account but not the manual ability to export CSVs? We’d be fine requiring an account for API access for powerusers. But given the impact that bot traffic is having on our instance’s performance, we might have to disable the anonymous user entirely if there isn’t a way to do this at present.
You might want to consider using simple user agent/IP blocks and run fail2ban to ban repeat offenders; or use a WAF/Cloudflare challenge for stronger bot mitigation. We use Cloudflare for our hosted instances to prevent our servers from being hammered by bot traffic!
I’ll run these by our IT folks. I do know that simple IP blocks only work for short time periods as the entities attacking an institution as big as OSU often just start right back up again from a new IP once blocked. I wonder if we could set up something that blocks any IP that makes more than, say, 100 API calls as the public user, and then just offer to make accounts with API access for powerusers.
Good to know that CloudFlare works well for the hosted instances!
An institutional firewall should be able to kick out bots like CloudFlare does, but that is not what it is made for. However, I am surprised any institution would allow anonymous access to something as important as a Collection Management System, as it provides a very wide attack surface for hackers. Any little bug in Specify or a library it uses might enable a hacker to elevate themselves from a guest user to a root user and gain full access to your data. If you want to share data with the public, send it to GBIF, set up or join a Symbiota portal, or use the Specify Web Portal.