Skip to main content

Solution: Managing Bot Traffic

  • Updated

Bot traffic describes machine-based traffic to your website. Bot traffic takes many forms, some good and some bad:

  • Good Traffic:
    • Search engine data gathering
    • SEO application data gathering
    • Script based/API initiated automation

  • Bad Traffic:
    • Malicious scripted attacks (DDos)
    • Web scraping bots 
    • Script based/API initiated automation
    • Malicious penetration testing 

Bot traffic is necessary for many search engines to operate and sites to function optimally. It's not possible–nor desirable–to eliminate bot traffic entirely. However, you can use Noibu to detect bot traffic and take steps to reduce or deter malicious activity.

Detecting Bot Traffic

While it's difficult to detect bot traffic through a session video, Noibu has assembled a list of detection methods that might identify a bot-controlled session. This can help you discern human-run sessions from bot-run sessions. When reviewing a session, look out for the following anomalies:

 

Issue Information:

Session Information:

  • Large amounts of sessions with single page visits (1 page count)
  • Large amounts of sessions with extremely outdated browsers (e.g. Chrome v85) and/or OS (e.g. Windows 7)
  • Large amounts of Linux sessions
  • IP addresses that map to a "Data center", "Proxy server", "VPN server", etc.

Session Recording:

  • The cursor is at the furthest point in the top-left corner of the screen
  • Mouse movements are linear, with no curvature 
  • Mouse movements are instantaneous, moving rapidly from point to point
  • Mouse clicks are positioned at the direct centre of each element
  • Multiple identical sessions

Managing Bot Traffic

While you can't get rid of all bot traffic, consult the suggestions below to minimize malicious activity on your website:

  • Limit the number of requests allowed
  • Use a robots.txt file
  • Block IP addresses from accessing resources
  • Use third-party resources to help limit traffic

Learn more about iFrame Tracking and Whitelisting the Noibu IP Address.

Articles in this section