Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Scrapy - Web crawling & scraping framework for Python

Open Search Server is both a modern crawler and search engine and a suite of high-powered full text search algorithms. Built using the best open source technologies like lucene, zkoss, tomcat, poi, tagsoup. Open Search Server is a stable, high-performance piece of software.
   <UL>
	<LI>Multi-languages indexing</LI>
	<LI>The crawlers go through web sites and file systems to rapidly and easily build your index.</LI>
	<LI>Numerous document formats are supported, such as XML, HTML/XHTML, Adobe™ PDF, Microsoft™ Word™, PowerPoint™, OpenOffice™, etc</LI>
	<LI>Quick integration thanks to an XML interface via HTTP queries (XML over HTTP) and PHP classes</LI>
	<LI>The web interface is built around the power offered by the Zkoss framework. It runs with the main Ajax browsers. This RIA-type interface is as comfortable to use as that of a heavy client</LI>
   </UL>

Open Search Server is both a modern crawler and search engine and a suite of high-powered full text search algorithms. Built using the best open source technologies like lucene, zkoss, tomcat, poi, tagsoup. Open Search Server is a stable, high-performance piece of software.

Open Search Server

Grub Next Generation is distributed web crawling system (clients/servers) which helps to build and maintain index of the Web. It is client-server architecture where client crawls the web and updates the server. The peer-to-peer grubclient software crawls during computer idle time.

Grub

Gigablast is one of the remaining four search engines in the United States that maintains its own searchable index of over a billion pages. It is scalable to thousands of servers. Has scaled to over 12 billion web pages on over 200 servers. It supports Distributed web crawler, Document conversion, Automated data corruption detection and repair,  Can cluster results from same site, Synonym search, Spell checker and lot more.  

Gigablast - Web and Enterprise search engine in C++

Yioop is an open source, PHP search engine capable of crawling, index, and providing search results for hundred of millions of pages on relatively low end hardware. It can index a variety of text formats HTML, RSS, PDF, RTF, DOC and images GIF, JPEG, PNG, etc. It can import data from ARC, WARC, Media-Wiki, Open Directory RDF. It is easily localized to many languages. It has built-in support for new feeds, discussion groups, blogs, and wikis. It also supports mixing indexes to create mash ups.

Yioop - Open Source Search Engine Software

Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repositoriy of your choice (e.g. a search engine). It very flexible, powerful, easy to extend, and portable.

Norconex HTTP Collector - Enterprise Web Crawler

Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set. Pavuk is a multifunctional open source web grabber with slow but continous development. Its features include
  <UL>
	<LI>recursive downloading based on links inside HTML documents</LI>
	<LI>transformation of Gopher and FTP directories into HTML document </LI>
	<LI>supports proxy servers (HTTP, FTP, SSL, HTTP gateway for FTP, HTTP gateway for Gopher, SOCKS 4/5) </LI>
	<LI>supports authentication against HTTP servers and proxy HTTP servers </LI>
	<LI>does restart of transfer after program break, link down, timeout or some other error </LI>
	<LI>can be run on a terminal or inside an X windows window</LI>
	<LI>have Native Language Support based on GNU gettext</LI>
	<LI>FTP over SSL</LI>
	<LI>multiple round-robin used HTTP proxies </LI>
 </UL>

Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set.

Pavuk

ASPseek is an Internet search engine software developed by SWsoft.ASPseek consists of an indexing robot, a search daemon, and a CGI search frontend. It can index as many as a few million URLs and search for words and phrases, use wildcards, and do a Boolean search. Search results can be limited to time period given, site or Web space (set of sites) and sorted by relevance (PageRank is used) or date. <BR><BR>ASPseek is optimized for multiple sites (threaded index, async DNS lookups, grouping results by site, Web spaces), but can be used for searching one site as well. ASPseek can work with multiple languages/encodings at once (including multibyte encodings such as Chinese) due to Unicode storage mode. Other features include stopwords and ispell support, a charset and language guesser, HTML templates for search results, excerpts, and query words highlighting.

ASPseek is an Internet search engine software developed by SWsoft.ASPseek consists of an indexing robot, a search daemon, and a CGI search frontend. It can index as many as a few million URLs and search for words and phrases, use wildcards, and do a Boolean search. Search results can be limited to time period given, site or Web space (set of sites) and sorted by relevance (PageRank is used) or date.

ASPseek

Discover open source projects across all platforms

Projects