Today, we are excited to introduce a new component to our MotoMiner architecture: The MotoMiner Traffic Cop.
As a reminder, our MotoMinerBot is a respectable search engine crawler. The MotoMiner Traffic Cop is another component we have employed to ensure that we are respectful of 3rd-party networks and data centers. The traffic cop monitors all outbound requests and evenly disperses and throttles them. While this news may not directly apply to our shoppers, it is a vital piece of the puzzle when it comes to a healthy relationship between MotoMiner and dealer website providers.
The MotoMiner Traffic Cop has the ability to govern outbound requests several ways. The most common way is by IP address. We can set how many maximum outbound requests can be sent to a particular IP address every minute. The Traffic Cop will enforce the rule and ensure outbound requests are evenly dispersed and throttled, so network spikes do not occur. Think of it as a self-enforced crawl delay for targeted networks. Check out the before and after charts below. You’ll notice that before the traffic cop, outbound requests would spike, causing unnecessary load on this 3rd-party network/data center. After the traffic cop, you’ll see that outbound request spikes have been eliminated and that traffic is more evenly dispersed.
Your dealers want to be indexed by MotoMiner. If you would like to learn more or would like to setup request thresholds for your data center, drop us a line via our online contact form.