An open source .NET web crawler written in C# using SQL 2005/2008. Arachnode.net is a complete and comprehensive .NET web crawler for downloading, indexing and storing Internet content including e-mail addresses, files, hyperlinks, images, and Web pages. Its features include
  <UL>
	<LI>.NET architecture</LI>
	<LI>Configurable Rules and Actions</LI>
	<LI>Lucene.NET Integration</LI>
	<LI>SQL Server 2008 and full-text indexing</LI>
	<LI>.DOC/.PDF/.PPT/.XLS Indexing</LI>
	<LI>HTML to XML and XHTML</LI>
	<LI>Multi-threading and Throttling</LI>
	<LI>Respectful Crawling</LI>
	<LI>Analysis Services</LI>
	<LI>SQL Server 2008 and SSIS</LI>
	<LI>EXIF data extraction</LI>
 </UL>

An open source .NET web crawler written in C# using SQL 2005/2008. Arachnode.net is a complete and comprehensive .NET web crawler for downloading, indexing and storing Internet content including e-mail addresses, files, hyperlinks, images, and Web pages.

Arachnode.net

Grub Next Generation is distributed web crawling system (clients/servers) which helps to build and maintain index of the Web. It is client-server architecture where client crawls the web and updates the server. The peer-to-peer grubclient software crawls during computer idle time.

Grub

Discover open source projects across all platforms

Projects

Arachnode.net

Grub

Tagcloud

License

Suggested keywords:

Projects

Arachnode.net

Grub

Tagcloud

License