Controlaccess.botBlocker

A robot or a client downloading a whole website can have a big impact on a dynamic website because the load is much higher than average. One way to deal with this is to keep a running moving total of the recents hits from any one IP and to return an http error code if the number of hits exceeds a magic number threshhold value. If the error code is a 503 service unavailable, well behaved clients are expected to retry the url after a small period of time. It remains to be seen if there are many, indeed any, well behaved clients .

Since this routine is called for every hit through mainresponder, its important to keep the overhead down, This is an interesting challenge - my solution is a table structure within temp with a subtable entry for each second of the day which in turn contains an array indexed by IP number counting the number of hits from one IP address in one second.

Computing the moving total then means accessing at most interval subtable entries. Since we are interested in a small time interval, we garbage collect the main table as we go by discarding subtables whose age is too great. Hopefuly this distributes that overhead evenly.

The script, which can be found at frontierTailor.data.installAddress.config.mainresponder.callbacks.controlaccess.botBlocker, must be called from a controlaccess table entry, which you need to create. Two globals at config.bayly.mainresponderCallbacks.maxHitsInterval and config.bayly.mainresponderCallbacks.maxHitsPerInterval also need to be manually defined. I started with values of max of 30 hits in 10 seconds, but will tune this as I monitor the scripts behaviour.

There are lots of ways to extend this idea.

• Don't return an error, simply have the thread wait for small period of time. This might slow robots on their first pass, but not work well if they are doing a verification pass.