MotoMiner Traffic Cop

One of the ways we ensure our MotoMinerBot is a respectful search engine crawler is via The MotoMiner Traffic Cop. Our traffic cop monitors all outbound requests and evenly disperses and throttles them based on defined thresholds. This technology ensures we are not placing undue load on 3rd-party networks and data centers. MotoMiner desires a healthy relationship with dealer website providers and the traffic cop is needed to achieve this.

The MotoMiner Traffic Cop has the ability to govern outbound requests several ways. The most common way is by IP address. We can set how many maximum outbound requests can be sent to an IP address down to the millisecond. The Traffic Cop will enforce the rule and ensure outbound requests are evenly dispersed/throttled so network spikes do not occur. Think of it as a self-enforced crawl delay for targeted networks.

For an example, let's say a provider states that they do not want MotoMiner to exceed 12,000 requests per hour, MotoMiner can deliver. Calculations are performed down to the millisecond. 12,000 requests per hour is a request every 300 milliseconds. The traffic cop will not authorize any requests which occur before the 300 millisecond threshold. This ensures MotoMiner stays within the confines set forth by the website provider and it also serves to evenly disperse outbound requests.

Without Traffic Cop:

Notice that without the traffic cop, outbound requests for this IP address would spike causing unnecessary load on the 3rd-party network/data center.

---

With Traffic Cop:

With the traffic cop, you’ll see that spikes have been eliminated and outbound traffic is more evenly dispersed.

Website Providers:

Your dealers want to be indexed by MotoMiner. If you would like to learn more or would like to setup request thresholds for your data center, drop us a line via our online contact form.