web crawlers

基本解释

Once the Sitemap XML is completed, you can make it available to Web crawlers by submitting a URL to the sitemap to those sites that support the protocol.
完成 Sitemap XML 文件后，便可以向支持这一协议的站点提交站点地图的 URL，这样 web 爬虫程序便可以使用该文件了。
Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.
Sitemaps 0.90为 Web 站点管理员提供了简单而且直观的方法，使他们可以向 web 爬虫程序提供正确的信息，从而使爬虫程序能够高效地爬行某个 Web 站点。
Also known as "invisible web", the term "deep web" refers to a vast repository of underlying content, such as documents in online databases that general-purpose web crawlers cannot reach.
“深网”也被称做“隐形网”，指的是网络中存储数量浩繁的基本信息的空间，例如存储于在线数据库中的文件，一般用途的网络搜索器是找不到这些文件的。

"Think algorithms and models and different software and Web crawlers, " says Christine Elliott, an American Express spokesperson.
FORBES: Magazine Article
Industry specific web crawlers or vulnerability discovery bots, network scans, and resource exhaustion attacks against web servers are rampant.
FORBES: Clean That Network Traffic
Through the use of proprietary Web crawlers, this news organization had breached the firewall to provide its readers with an upper hand.
FORBES: Data Security: A Shareholder Protection Imperative

网络爬虫
Web page lifetime analysis from users perspective is important to enhance the performance of web crawlers and search engines,and to improve the efficiency of web advertising.
从用户的角度对网页的生命周期进行分析可以提高网络爬虫和搜索引擎的性能,改善网络广告的效果。