Excessive 404 errors causing IP address to be blocked

On our shared Linux cPanel servers, generating excessive "404: File does not exist" errors will result in the IP address of the visitor generating the errors to get temporarily blocked. The threshold is current configured for 200 errors within 300 seconds, with a ban period of 3600 seconds. So if a visitor generates more than 200 404: File does not exist errors within 5 minutes, their IP address gets blocked for 1 hour.

The purpose for this temporary block is for performance and manageability of the server and your site. Consider this: Assume that one visitor generates the full 200 404: File does not exist errors within 5 minutes and the site in question has 100 visitors over the course of the day. That is 20,000 errors in one day getting logged. So over the course of a month that is over 600,000 additional errors for the site being logged. And this is just the threshold of what is allowed. With no restrictions, a busy site with bad code generating multiple 404 errors could very well generate millions of errors being logged in a very short amount of time. Now multiply that by the number of sites on the server. As you can imagine, this can quickly cause the server error log to swell to an unmanageable size.

This is why it is important to correct any 404: File does not exist, and why we must have thresholds in place to limit the number of those being generated. While on the surface, it may appear that our thresholds are too strict. But when you figure how many errors one user would be capable of generating before any temporary ban gets enforced, the thresholds are actually quite lenient.