What are bots?
Bots are tools that automatically visit websites to complete specific tasks. Some bots are helpful, while others are not.
“Bad bots” can generate fake traffic to trigger ads and earn money dishonestly. This activity can get your site flagged or even blocked by advertisers.
But not all bots are bad. Many play a key role in keeping the web running smoothly. For example, some “good bots” help search engines understand your content better to rank your site appropriately. Other “good bots” can ensure your site is safe and relevant for advertisers. Blocking these “good bots” could lead to missed revenue opportunities for your site.
The important part is knowing which “good bots” to allow.
What do "good bots" do?
These bots may review your site to:
- Make sure your content is safe for advertisers
- Categorize your content accurately
- Match advertisers to your content so they can bid on your ad space
These “good bots” are a very important part of the web ecosystem. Without access, your site might be miscategorized or marked as high risk, which are two things that can keep advertisers (and their budgets) away.
Make sure your host allows “good bots”
Sometimes, hosting companies will block bots automatically, even the helpful ones. To avoid that, ask your host to allow the “good bots” by allowing the following user agents and IP addresses:
User agents to allow:
- Leikibot
- DoubleVerifyBot
- DVBot
- Amazonbot
- AmazonAdBot
- AdsBot-Google
- CriteoBot
- Gumgum
- Peer39_crawler
- Proximic
- PubMatic Crawler Bot
- TTD-Content
- Bidswitchbot
- SinceraSyntheticUser
- IAS_crawler (http://integralads.com/site-indexing-policy/)
- IAS_wombles
- IAS_admantx user-agent (https://www.admantx.com/service-fetcher.html)
- Grapeshot
- Opebot-v
IP addresses to allow:
- 3.217.168.199 (IAS)
- 18.214.43.70 (IAS)
- 3.226.51.67 (IAS)
- 52.45.0.188 (1plusX)
We also recommend having your host allow the following list of IP addresses for DoubleVerify specifically:
- 34.138.190.128/28
- 34.138.22.128/28
- 34.94.132.97/32
- 34.107.106.182/32
- 34.87.69.201/32
- 35.198.204.243/32
- 35.246.237.77/32
- 35.236.14.138/32
- 34.84.80.99/32
- 35.246.238.37/32
- 34.102.65.14/32
- 35.231.101.249/32
- 34.138.127.213/32
- 34.239.194.235
- 52.0.62.67
- 54.161.75.128
- 35.211.248.201
- 35.211.197.238
- 35.211.55.96
- 204.154.110.0/23
- 38.64.81.0/24
- 63.251.109.0/24
- 69.174.120.0/24
- 69.174.125.72/29
- 95.183.5.56/29
- 108.60.134.208/29
- 116.51.17.56/30
- 144.121.119.248/29
- 154.59.97.0/24
- 158.106.219.188/30
- 158.106.221.240/29
- 206.72.210.0/23
- 209.51.163.228/30
- 213.198.83.212/30
- 213.254.216.8/29
- 213.254.244.0/24
- 2001:218:4000:5000::f4/126
- 2001:668:0:3:ffff:1:0:600/126
- 2001:668:0:3:ffff:3:0:ec/126
- 2001:728:0:5000::150c/127
Ensure these bots are not blocked in your robots.txt file
Your robots.txt file helps advertisers analyze each post/page for content-specific advertising opportunities. These opportunities are an easy way to boost your site’s ad revenue. Blocking “good bots” here can prevent advertisers from properly analyzing your content, which could hurt your ad performance.
If your website generally blocks crawlers, please update your robots.txt file on each of your domains to allow us access to the “good bots” listed above.