Many websites would not deliver you content if you came presenting as a wrong user agent. In PHP there are ways to present yourself as someone else if you are going to crawl somebody’s website.
In this snippet we show you how to use PHP user agent settings in cURL, and trick server to think it was visited by Google bot.
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)'; curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
Of course, you might want to present as some other user agent in your PHP script, so here is the list of some popular search engine user agents:
- Google ” Googlebot/2.1 ( http://www.googlebot.com/bot.html)
- Google Image ” Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
- MSN Live ” msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
- Yahoo ” Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
More user agents you can find at: user-agents.org.