Infrastructure/Search

= Points of Contact =

Project Sponsor
Name: Mike McGrath Fedora Account Name: mmcgrath Group: Infrastructure Infrastructure Sponsor: mmcgrath

Secondary Contact info
Name: Allen Kistler Fedora Account Name: akistler Group: Infrastructure

Name: Huzaifa Sidhpurwala Fedora Account Name: huzaifas Group: Infrastructure

= Project Info = Project Name: Search Engine Enhancement Target Audience: All users of Fedora web sites Expiration/Delivery Date (required): F13

Description/Summary
Fedora needs a search engine

Requirements

 * Crawl the web sites (wiki and non-wiki)
 * Search the web sites (wiki and non-wiki)
 * Java, if any, must be the GCJ/OpenJDK versions in RHEL5; Sun/IBM/BEA Java is not acceptable

Preferences

 * Python-based
 * Programmable keywords to have control over what pages get displayed for certain keywords
 * XML or library interface so other applications can use it
 * Support multiple-language text (i.e., Unicode)

Project Plan

 * 1) Investigate and evaluate existing open source search engines
 * 2) Select candidate software
 * 3) Create public test instances of candidate software
 * 4) Test for functionality, performance, and impact (re-evaluate, if necessary)
 * 5) Create capacity and deployment plans
 * 6) Deploy

Resources Needed

 * Public Test for testing candidate software
 * Permanent home(s) for deployment
 * Web server(s)
 * Database server(s) (maybe)

In Progress

 * DataparkSearch - akistler examined
 * Description
 * written in C
 * forked from mnoGoSearch in 2003 when mnoGoSearch went semi-commercial
 * Indices are stored in a database; Supported databases include MySQL and PostgreSQL (SQLite not advertised, but listed in documentation)
 * Support for all SBCS and some DBCS
 * Search tool supports Booleans (AND, OR, NOT, NEAR)
 * Evaluation
 * Evaluated after mnoGoSearch.
 * Database modes are single and multi, like mnoGoSearch, but include hashed modes, which does not support string and substring searches, and cached mode, which stores only URI indices in the database, but stores word data in disk files through an additional daemon. The absence of blob mode from mnoGoSearch, but the presence of cached mode in DataparkSearch, appears to be a major difference between the two.
 * As written, the setup expects that the application process logs in to the database as the schema owner, however with additional manual steps it can be made to work as not the schema owner. The call handler setup and custom stored procedure language definition present in mnoGoSearch is commented out in the setup of DataparkSearch, so PostgreSQL superuser is not required, as written. (Their presence in mnoGoSearch is questionable, anyway.) In general, DataparkSearch does appear to be a more slowly developed version of mnoGoSearch.
 * Multiple character set support is not the default, but is specified explicitly before compilation.
 * Bugs in create.multi.sql
 * 185: ERROR: relation "cachedchk2" already exists
 * 186: ERROR: column "url_id" does not exist
 * Caused by duplicate CREATE statements in the script. Comment out one set to resolve the bug.
 * Bugs in drop.multi.sql
 * 16: ERROR: sequence "url_rec_id_seq" does not exist
 * 17: ERROR: sequence "categories_rec_id_seq" does not exist
 * 18: ERROR: sequence "qtrack_rec_id_seq" does not exist
 * 19: ERROR: sequence "server" does not exist
 * Caused by extraneous DROP statements. Comment them out or ignore them. They're harmless.
 * Extended search mode appears broken. Only the first result is returned. All other results are lost. This might be only an error in the search form, in which case it can be easily debugged and fixed.
 * Requirements
 * buildrequires
 * gcc make
 * postgresql-devel (for PostgreSQL support)
 * zlib-devel
 * requires
 * postgresql-libs (for PostgreSQL support)
 * httpd
 * zlib
 * others as desired to index documents (pdf, etc.)
 * Setup Notes
 * The tarball was compiled into /opt/dpsearch, although it should be possible to create an RPM for more conventional locations (/bin, /etc, /sbin, etc.)
 * The code compiles and runs fine on x86_64 (compare to mnoGoSearch)
 * Create database user (dp/search) different from database owner (dbowner/dbowner) different from superuser (postgres)
 * Run create.multi.sql as dbowner, then grant privileges to dp
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE cachedchk TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE cachedchk2 TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE categories TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE cookies   TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE crossdict TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict      TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict10    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict11    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict12    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict16    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict2     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict3     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict32    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict4     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict5     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict6     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict7     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict8     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict9     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE links     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE qinfo     TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE qtrack    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE robots    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE server    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE srvinfo   TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE storedchk TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE url       TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE urlinfo   TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE categories_rec_id_seq TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE qtrack_rec_id_seq    TO dp;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE url_rec_id_seq       TO dp;
 * cp /opt/dpsearch/etc/indexer.conf-dist /opt/dpsearch/etc/indexer.conf
 * chmod and chgrp to protect passwords!
 * vi /opt/dpsearch/etc/indexer.conf
 * set DBAddr, including mode=multi
 * set LocalCharset to UTF-8 for full Unicode support
 * set MaxHops to the maximum depth (default is 256), if desired/required
 * add CrawlDelay, if desired/required
 * add "Server Disallow http://fedoraproject.org/w/ " and "Server Disallow http://fedoraproject.org/wikiold/ "
 * set Server to http://fedoraproject.org/ (the URI to be crawled and indexed)
 * cp /opt/dpsearch/etc/stopwords.conf-dist /opt/dpsearch/etc/stopwords.conf
 * Note that there are no Chinese stopwords, so don't uncomment the line to configure them
 * cp /opt/dpsearch/etc/langmap.conf-dist /opt/dpsearch/etc/langmap.conf
 * vi /opt/dpsearch/etc/langmap.conf
 * Comment out the lines for ca.latin1.lit.lm, ga.latin1.lit.lm, ko.utf8.lit.lm, and pt-pt.latin1.lm, because the files for them don't exist
 * Uncomment the lines for ja.euc-jp.lm, ja.sjis.lm, ta.tscii.lm, zh.big5.lm, and zh.gb2312.lm, because the files for them do exist
 * cp /opt/dpsearch/etc/sections.conf-dist /opt/dpsearch/etc/sections.conf
 * cp /opt/dpsearch/etc/search.htm-dist /opt/dpsearch/etc/search.htm (htm name must not change; compare to mnoGoSearch)
 * chmod and chgrp to protect passwords!
 * vi /opt/dpsearch/etc/dpsearch.htm
 * set DBAddr and LocalCharset to correspond to /opt/dpsearch/etc/indexer.conf
 * vi /opt/dpsearch/etc/dpsearch.htm
 * set DBAddr and LocalCharset to correspond to /opt/dpsearch/etc/indexer.conf


 * Egothor
 * Description
 * written in Java


 * Ferret
 * Description
 * Ruby port of Lucene
 * KinoSearch and Ferret intend to merge as Lucy


 * Indri
 * Description
 * Written in C/C++
 * Indexes text and HTML, can be extended with custom parsers
 * Provides Java, PHP, and C++ APIs and, if desired, a SOAP interface
 * Search tool is a local Java application or a PHP CGI application


 * KinoSearch - akistler examined
 * Description
 * Perl/C port of Lucene
 * in Fedora already
 * maintainer Rectangular Research appears to be just one guy, who considers KinoSearch to be alpha software
 * KinoSearch and Ferret intend to merge as Lucy
 * Evaluation
 * Search engine library with sample indexer and search page rather than fully-functional application. Stores indices in Berkeley DB files with JSON interfaces.  Allows custom-designed indices, including categories (exact match) to fulfill "programmable keywords" requirement.  Each document index on each document source is a single write-once file collection (BDB and JSON) in a unique directory.  Rerunning the indexer creates a new directory, obsoleting the old directory if all the old documents are included.  The old directory then needs to be cleaned up.  Postings can, however, be deleted from an index.  Additionally, only the new documents can be indexed, but that's not efficient.
 * Requirements
 * buildrequires
 * gcc
 * (EPEL) perl-Module-Build
 * requires
 * (EPEL) perl-JSON-XS
 * Problem: Desires 1.53, but EPEL has 1.43
 * Note: http://web.archive.org/web/20071122035408/search.cpan.org/src/MLEHMANN/JSON-XS-1.53/
 * Note: works with 1.43, anyway
 * (EPEL) perl-Lingua-Stem-Snowball
 * (EPEL) perl-Lingua-StopWords
 * (EPEL) perl-Parse-RecDescent
 * sample indexer reads files from the file system and requires
 * (EPEL) perl-HTML-Tree
 * sample cgi search script requires
 * (CPAN) Data::Pageset (which requires Data::Page)
 * (EPEL) perl-Test-Exception
 * (EPEL) perl-Class-Accessor-Chained


 * mnoGoSearch - akistler examined
 * Description
 * written in C
 * UNIX/Linux source code is GPL; Windows binaries are commercial, likely based on the GPL UNIX/Linux code, and lag a few versions behind
 * Indices are stored in a database; Supported databases include MySQL, PostgreSQL, and SQLite (among others)
 * HTTP, FTP, and NNTP crawling
 * C, PHP, and Perl APIs (advertised, apparently only C API really included)
 * SBCS and most MBCS supported, including most eastern Asian languages
 * Evaluation
 * The supplied install.pl script generates a configure command, but does not support SQLite.
 * Adding --with-sqlite3 to the generated command adds SQLite support. An empty database must be created manually.  A URI in the indexer.conf file specifies the location of the database.  According to the documentation, sqlite:/path/to/db/file should work, but doesn't.  According to the message boards on mnoGoSearch.org, sqlite://localhost/path/to/db/file should work, but doesn't.
 * No problems compiling with PostgreSQL support. Configure and Makefile may need significant rewriting to make good RPMs. There is no support for cross-compiling to 32-bit architectures from 64-bit machines.
 * As written, the setup expects that the application process logs in to the database as a PostgreSQL superuser and schema owner, however with additional manual steps it can be made to work as neither the superuser nor the schema owner. Several schemata are available (single, multi, and blob). LavTech recommends blob for sites indexing more than 50k documents. The crawler is very flexible with quite a complex configuration file. The CGI search page also has nice features for "advanced" searching, although it can be customized to suit each site. Tags are labels configured within the crawler, usually by URI server component. Categories are numerical hierarchies, up to 6 levels deep, also specified in the crawler configuration.
 * Bugs in pgsql/drop.blob.sql
 * drop function clean_srvinfo; (the is omitted, but needs to be included)
 * DROP LANGUAGE plpgsql; (missing)
 * DROP FUNCTION plpgsql_call_handler; (missing, has to be run twice, once for postgres, once for dbowner?)
 * Does this definition in postgres (dangerously) shortcut anything inherent for other DBs on the same server?
 * Requirements
 * buildrequires
 * gcc make
 * sqlite-devel (for SQLite support)
 * postgresql-devel (for PostgreSQL support)
 * zlib-devel
 * requires
 * sqlite (for SQLite support)
 * postgresql-libs (for PostgreSQL support)
 * httpd
 * zlib
 * others as desired to index documents (pdf, etc.)
 * Setup Notes
 * The tarball was compiled into /opt/mnoGoSearch, although it should be possible to create an RPM for more conventional locations (/bin, /etc, /sbin, etc.)
 * RPM for Fedora currently awaiting review
 * On x86_64 architecture, the x86_64 binary fails when indexing a crawl with the error "indexer[21272]: PQexecPrepared: ERROR: incorrect binary data format in bind parameter 2." The tarball refuses to cross-compile to 32-bit architecture, despite tweaking the ./configure options. Compiling on a 32-bit machine and moving the binaries to the 64-bit machine works.
 * Create database user (mno/search) different from database owner (dbowner/dbowner) different from superuser (postgres)
 * /opt/mnoGoSearch/sbin/indexer -Ecreate, to create the tables, etc., and /opt/mnoGoSearch/sbin/indexer -Edrop, to drop tables, etc., just runs a script from /opt/mnoGoSearch/share//, but needs to be run as postgres
 * Run create.blob.sql as postgres, change owners to dbowner, and grant privileges to mno
 * ALTER TABLE bdict     OWNER TO dbowner;
 * ALTER TABLE bdicti    OWNER TO dbowner;
 * ALTER TABLE categories OWNER TO dbowner;
 * ALTER TABLE crossdict OWNER TO dbowner;
 * ALTER TABLE dict      OWNER TO dbowner;
 * ALTER TABLE links     OWNER TO dbowner;
 * ALTER TABLE qcache    OWNER TO dbowner;
 * ALTER TABLE qinfo     OWNER TO dbowner;
 * ALTER TABLE qtrack    OWNER TO dbowner;
 * ALTER TABLE server    OWNER TO dbowner;
 * ALTER TABLE srvinfo   OWNER TO dbowner;
 * ALTER TABLE url       OWNER TO dbowner;
 * ALTER TABLE urlinfo   OWNER TO dbowner;
 * ALTER TABLE wrdstat   OWNER TO dbowner;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE bdict     TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE bdicti    TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE categories TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE crossdict TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE dict      TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE links     TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE qcache    TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE qinfo     TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE qtrack    TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE server    TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE srvinfo   TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE url       TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE urlinfo   TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE wrdstat   TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE categories_rec_id_seq TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE qtrack_rec_id_seq    TO mno;
 * GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE url_rec_id_seq       TO mno;
 * cp /opt/mnoGoSearch/etc/indexer.conf-dist /opt/mnoGoSearch/etc/indexer.conf
 * chmod and chgrp to protect passwords!
 * vi /opt/mnoGoSearch/etc/indexer.conf
 * set DBAddr, including mode=blob for > 50k documents
 * set LocalCharset to UTF-8 for full Unicode support
 * set MaxHops to the maximum depth (default is 256), if desired/required
 * add CrawlDelay, if desired/required
 * add "Server Disallow http://fedoraproject.org/w/ " and "Server Disallow http://fedoraproject.org/wikiold/ "
 * set Server to http://fedoraproject.org/ (the URI to be crawled and indexed)
 * cp /opt/mnoGoSearch/etc/search.htm-dist /opt/mnoGoSearch/etc/mnoGoSearch.htm (htm name must match cgi name below)
 * chmod and chgrp to protect passwords!
 * vi /opt/mnoGoSearch/etc/mnoGoSearch.htm
 * set DBAddr and LocalCharset to correspond to /opt/mnoGoSearch/etc/indexer.conf
 * cp /opt/mnoGoSearch/bin/search.cgi to /var/www/cgi-bin/mnoGoSearch
 * To crawl the sites configured, /opt/mnoGoSearch/sbin/indexer
 * To index the data collected, /opt/mnoGoSearch/sbin/indexer -Eblob
 * To display database statistics, /opt/mnoGoSearch/sbin/indexer -S
 * To clear the database, /opt/mnoGoSearch/sbin/indexer -C
 * To display database statistics, /opt/mnoGoSearch/sbin/indexer -S
 * To clear the database, /opt/mnoGoSearch/sbin/indexer -C


 * Nutch - akistler examined
 * Description
 * written in Java
 * based on Lucene
 * the crawler/indexer is a Java command line application; the default depth is 5; the default number of threads is 10
 * the search tool runs in a Java servelet container, e.g., Tomcat
 * Evaluation
 * There's nothing to build. Simply configure the crawler (which is actually the indexer, too) and deploy/configure the searcher.  The crawler caches the pages it indexes, making the cache available to the search tool.  The search interface is extremely simple and is multi-lingual, but is almost entirely an advertisement for the Nutch project.  It doesn't look particularly easy to rebrand.  Overall, the polish of the finished product means it's less flexible to custom modifications, like programmable keywords.  After creating a new index (i.e., after a new crawl), the search application must be reloaded in Tomcat manager.  The crawler is more flexible than a brief investigation could reveal.  The official documentation leaves a lot to be desired.  Searches are for single terms only, no multiple terms or +/- Booleans.
 * Requirements
 * java-1.6.0-openjdk, java-1.6.0-openjdk-devel (explicit specification short-circuits yum's attempt to install BEA for Tomcat)
 * tomcat5, tomcat5-webapps, tomcat5-admin-webapps (tomcat5-admin-webapps is probably not actually required, but is very handy)
 * Set-up Notes
 * The best "tutorial" is http://wiki.apache.org/nutch/NutchTutorial
 * Create /etc/profile.d/java.sh for "export JAVA_HOME=/etc/alternatives/jre" (required for the crawler, architecture independent definition)
 * In /opt (in this example, though anyplace will do) "tar xzf /path/to/tar/file/nutch-1.0.tar.gz"
 * Set http.agent.name in /opt/nutch-1.0/conf/nutch-site.xml (e.g., FedoraProject)
 * Create a flat file of starting URLs in /opt/nutch-1.0/urls/crawl-start.txt (it can be any file name in the directory)
 * Edit /opt/nutch-1.0/conf/crawl-urlfilter.txt to set regular expressions which keep or discard links for processing (e.g., the domain of the servers crawled)
 * If not running the crawler as root (Why would you?), create directories /opt/nutch-1.0/crawl and /opt/nutch-1.0/logs writable by whatever uid/gid used to crawl
 * Note: The crawler also needs to be able to create temporary files in its working directory
 * Run the crawler, putting the database in /opt/nutch-1.0/crawl
 * Deploy the WAR file from /opt/nutch-1.0/nutch-1.0.war as /nutch in Tomcat manager
 * Set searcher.dir in /var/lib/tomcat5/webapps/nutch/WEB-INF/classes/nutch-site.xml to /opt/nutch-1.0/crawl


 * Swish-e - akistler examined
 * Description
 * written in C
 * Note: Swish++ is a rewrite in C++ (not evaluated here)
 * Evaluation
 * Search engine with a built-in web crawler, a built-in file system crawler, and an interface for an external crawler. The distribution includes sample search pages which use the Perl API.  There is also a C API.  The index is not customizable, but does include a facility for including metawords (exact match) and the path in the index for each document.  The documentation acknowledges that the software only supports ASCII, but some MBCS may also work.
 * Requirements
 * buildrequires
 * gcc
 * make
 * libxml2-devel
 * zlib-devel
 * requires
 * libxml2
 * zlib
 * perl-libwww-perl (for the built-in spider)
 * others as desired to index documents (pdf, etc.)


 * Xapian - akistler examined
 * Description
 * written in C++
 * bindings to Python, Ruby, and Perl XS
 * xapian-core, xapian-bindings, and perl-Search-Xapian in Fedora already; xapian-omega is not
 * additional bindings to PHP, Java, and more (?)
 * Omega provides a Xapian front-end for indexing (via script) and searching (command line or CGI)
 * Omega provides glue scripts for ht://Dig, mbox files, and perl DBI
 * Flax is another search engine built on top of Xapian and CherryPy
 * Evaluation
 * Xapian is a search engine library. Omega adds functionality on top of Xapian.  The Xapian database is very flexible, supporting an entirely user-designed schema.  Usage through Omega loses very little, if any, of that flexibility, however the supplied Omega CGI is extremely rudimentary.  The supplied Omega CGI also requires the database to be named "default," although that can be changed.  Database columns are of type field or index.  Fields are stored verbatim (e.g., URL, date, MIME type, keywords).  Indices are input as blocks of text or other content to be indexed, but not stored (e.g., the corpus of a file or web page).  The Omega scriptindex utility can be combined with an external web crawler for HTML.  Making Omega work with Apache requires relabeling /var/lib/omega as httpd_sys_content, or moving /var/lib/omega to /var/www/omega and using the default context there.  In this evaluation, /var/lib/omega was moved to /var/www/omega.  Xapian only works with UTF-8.
 * Requirements
 * xapian-core buildrequires
 * gcc gcc-c++
 * make
 * zlib-devel
 * xapian-bindings buildrequires (not including gcc gcc-c++ make)
 * python python-devel
 * ruby ruby-devel
 * xapian-core-devel
 * perl-Search-Xapian buildrequires (not including gcc gcc-c++ make)
 * perl
 * xapian-core-devel
 * xapian-omega buildrequires (not including gcc gcc-c++ make)
 * libtool
 * xapian-core-devel
 * xapian-core requires
 * xapian-core-libs
 * xapian-bindings requires
 * coreutils
 * python
 * xapian-core-libs
 * perl-Search-Xapian requries
 * perl
 * xapian-core-libs
 * xapian-omega requires
 * httpd
 * perl
 * perl-DBI
 * xapian-core-libs
 * Set-up Notes
 * Using custom crawler that requires perl-libwww-perl
 * Updated Omega rpm .spec to move /var/lib/omega to /var/www/omega, including updating /etc/omega.conf
 * Use default database at /var/www/omega/data/default
 * /var/www/omega/data/default.index is
 * URI : field boolean=Q unique=Q
 * ContentType : field=MIMEtype index=T
 * LastModified : field=Date date=unix
 * Title : field index=S
 * Content : index
 * To index, run (as someone who can write to /var/www/omega/data/default)
 * ./crawl.pl http://fedoraproject.org | scriptindex /var/www/omega/data/default /var/www/omega/data/default.index
 * Note that "Content : unhtml index" would be preferable in the index, but unhtml apparently has bugs

Eliminated from Consideration

 * CLucene
 * Reason for elimination
 * It has no crawling/spidering facility. It is a library toolkit (API) only.
 * Description
 * C++ port of Lucene
 * in Fedora already
 * described as beta by the developers


 * Isearch
 * Reason for elimination
 * It has no crawling/spidering facility. It is intended for local file indexing only.
 * The web site appears to have fallen out of maintenance.
 * Description
 * written in C++


 * Lucene - akistler examined
 * Reason for elimination
 * It has no crawling/spidering facility. It has no user query interface.  There are no samples.
 * Description
 * written in Java, but ported to others
 * Requires/uses GCJ
 * in Fedora already
 * PyLucene is a Python wrapper around Java Lucene
 * Evaluation
 * Search engine library meant to be integrated into applications
 * Requirements
 * buildrequires (based on 1.4.3-f7)
 * ant
 * ant-junit
 * java-1.4.2-gcj-compat-devel
 * javacc
 * jpackage-utils
 * junit
 * make
 * requires (based on 1.4.3-f7)
 * java-1.4.2-gcj-compat


 * Namazu
 * Reason for elimination
 * It has no crawling/spidering facility. It indexes local documents only.
 * Description
 * written in Perl
 * in Fedora already


 * Solr - akistler examined
 * Reason for elimination
 * It has no crawling/spidering facility. It has no user query interface.  There are no samples.
 * Description
 * written in Java
 * based on Lucene
 * Evaluation
 * The documentation describes installing Sun Java to run Solr, but OpenJDK 1.5 or later is fine. Solr needs a Java servlet container in which to run.  It comes with Jetty, but other containers should work, as well (e.g., Tomcat).  Currently only supports UTF-8 characters.
 * Basically Solr provides an HTTP admin GUI for a search engine that uses a superset of the Lucene query syntax. The schema is very flexible.  Set-up is essentially entirely through XML files.  Applications can query the servlet port and get XML or JSON responses.
 * Requirements
 * buildrequires
 * ant (note that ant currently pulls in java-gcj-compat, too, but it appears not to be a problem)
 * ant-junit
 * java-1.6.0-openjdk-devel
 * junit
 * requires
 * java-1.6.0-openjdk


 * Terrier (TERabyte RetrIEveR)
 * Reason for elimination
 * It has no crawler or user search tool. It does not run as a service (as provided), only interactively.
 * Description
 * written in Java
 * runs from the command line (i.e., not a Tomcat servlet)


 * YaCy - huzaifa examined
 * Reason for elimination
 * Requires Sun Java
 * written in Java
 * well maintained
 * support for peer search engine database exchanges
 * customized search parameters
 * fast indexing and web interface for querying the back-end db


 * Zettair
 * Reason for elimination
 * No crawling capability, only indexes local documents
 * User search/retrieval tool is command-line only, no web interface
 * Description
 * written in C

Never Considered

 * Gonzui
 * written in Ruby
 * specializes in source code search
 * not actively maintained


 * Grub
 * written in C#


 * Heritrix
 * written in Java
 * archives content rather than indexing it


 * ht://Dig
 * written in C++
 * not actively maintained


 * HtdigSearch
 * It's just a MediaWiki plugin, not suitable for searching non-wiki sites


 * MWSearch
 * Requires EzMwLucene (Java) to be running on the servers to be searched
 * EzMwLucene is wiki-only, therefore MWSearch is wiki-only


 * OpenFTS
 * written in Perl or TCL on top of PostgreSQL
 * Python interface available
 * not actively maintained


 * Plucene
 * Perl port of Lucene
 * not actively maintained


 * RigorousSearch
 * Crawls the MediaWiki database, not the web site
 * Doesn't work for non-MediaWiki web sites, including any non-wiki web site


 * Sphinx
 * written in C++
 * designed to index SQL tables, not web pages


 * SphinxSearch
 * Written in C++
 * MediaWiki plug-in, so it's wiki-only


 * Whoosh
 * written in Python
 * inspired by Lucene, but closer to a Python port of parts of KinoSearch combined with some features of Terrier
 * toolkit only, not even sample crawlers and user interfaces

Public Testing
Public Testing is taking place on publictest3.

Search Engines

 * DataparkSearch ( http://publictest3.fedoraproject.org/cgi-bin/dpsearch )
 * PostgreSQL installed (postgresql, postgresql-devel, postgresql-libs, postgresql-server)
 * SELinux (not present on publictest3, but needed eventually) needs:
 * "setsebool -P httpd_can_network_connect=1" to connect to PostgreSQL
 * Crawling trials (with database cleared each time, i.e., not incremental)
 * Memory and CPU utilization are modest, less than 10% each. Most CPU time is spent in I/O Wait for the database.
 * Depth=4, 2.5 hrs crawling, 2k documents, db = 320M (700M with clone detection off)
 * Depth=5, 16.5 hrs crawling, 12k documents, db = 1.1G


 * mnoGoSearch ( http://publictest3.fedoraproject.org/cgi-bin/mnoGoSearch )
 * PostgreSQL installed (postgresql, postgresql-devel, postgresql-libs, postgresql-server)
 * SELinux (not present on publictest3, but needed eventually) needs:
 * "setsebool -P httpd_can_network_connect=1" to connect to PostgreSQL
 * Crawling trials (with database cleared each time, i.e., not incremental)
 * Memory and CPU utilization are quite modest, about 1% each. Most CPU time is spent in I/O Wait for the database.
 * Depth=4, 2 hrs crawling, 1.5 min indexing, 11k documents
 * Depth=5, 4.5 hrs crawling, 12 min indexing, 25k documents
 * Depth=6, 6.5 hrs crawling, 16 min indexing, 34k documents
 * Depth=7, 12 hrs crawling, 23 min indexing, 40k documents, db = 2.6G


 * Nutch ( http://publictest3.fedoraproject.org/nutch )
 * The Nutch tarball was unpacked in /opt/nutch-1.0, just as in preliminary local testing
 * Tomcat is reverse proxied through Apache (see notes below)
 * Nutch's definition/conception of depth appears to be unusual. The crawler must be directed to spider much more deeply than should be necessary.
 * Crawls are executed as (e.g.) "/opt/nutch-1.0/bin/nutch crawl /opt/nutch-1.0/urls -dir /opt/nutch-1.0/crawl -depth 5 -threads 2"
 * Crawling trials
 * java process uses about 18% of 6G of memory (4G RAM, 2G swap), regardless of depth
 * Depth=4, 2 threads, 1.5k documents
 * Depth=5, 2 threads, 3 hrs, 8k documents
 * Depth=6, 1 thread, 8.5 hrs, 23k documents
 * Depth=7, 1 thread, 14.5 hrs, 37k documents, db = 400M
 * Depth=8, 1 thread, 16.5 hrs, 44k documents, db = 440M


 * Xapian ( http://publictest3.fedoraproject.org/cgi-bin/omega )
 * At this time, only installed xapian-core-libs, xapian-core, and xapian-omega (i.e., no xapian-bindings or perl-Search-xapian)
 * Enabled cgi-bin in /etc/httpd/conf.d/cgi-bin.conf (see notes below)
 * Omega bombs on http://fedoraproject.org/wiki/Overview (and would possibly on others later) with "unhtml index"
 * Resolution: Don't use "unhtml"
 * Omega bombs on long URIs (longer than 244 chars)
 * Example: http://fedoraproject.org/wiki/Special:WhatLinksHere/Ru_RU/%D0%9F%D0%BB%D0%B0%D0%BD_%D1%80%D0%B0%D0%B1%D0%BE%D1%82%D1%8B_%D0%BF%D0%BE_%D0%BF%D0%B5%D1%80%D0%B5%D0%B2%D0%BE%D0%B4%D1%83_web-%D1%81%D0%B0%D0%B9%D1%82%D0%B0_%D0%BF%D1%80%D0%BE%D0%B5%D0%BA%D1%82%D0%B0_Fedora
 * Resolution: Enhanced custom crawler to filter URIs better (fedoraproject.org/w/ from the wiki); added capability to discard URIs that are too long (mostly hex URIs translated from other DBCS)
 * Perl custom crawler prints warnings for (and refuses to translate) URIs with Unicode characters outside the Latin 1 range
 * Resolution: None. This issue is known for URI.pm.
 * Crawling trials
 * Failed, Depth=5, scriptindex used 70% of 4G of memory (2G RAM, 2G swap, 0% free)
 * terminated, system sluggish with swap I/O
 * Depth=4, 4 hrs, 15218 documents, index is about 500M on disk, scriptindex used 40% of 4G of memory (2G RAM, 2G swap)
 * Depth=5, 8 hrs, 41171 documents, index is about 1G on disk, scriptindex used 20% of 6G of memory (4G RAM, 2G swap)

Apache Configuration Notes

 * CGI for Xapian Omega and mnoGoSearch
 * In /etc/httpd/conf.d/cgi-bin.conf, just use the default configuration normally commented out in httpd.conf.

# ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" # #  AllowOverride None Options None Order allow,deny Allow from all 
 * 1) ScriptAlias: This controls which directories contain server scripts.
 * 2) ScriptAliases are essentially the same as Aliases, except that
 * 3) documents in the realname directory are treated as applications and
 * 4) run by the server when requested rather than as documents sent to the client.
 * 5) The same rules about trailing "/" apply to ScriptAlias directives as to
 * 6) Alias.
 * 1) "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
 * 2) CGI directory exists, if you have that configured.


 * Reverse proxy for Tomcat
 * In /etc/httpd/conf.d/tomcat5.conf:

 Order Allow,Deny Allow from ...  ProxyPass       /admin   http://localhost:8082/admin ProxyPassReverse /admin  http://localhost:8082/admin #  Order Allow,Deny Allow from ...  ProxyPass       /manager http://localhost:8082/manager ProxyPassReverse /manager http://localhost:8082/manager # ProxyPass       /nutch   http://localhost:8082/nutch ProxyPassReverse /nutch  http://localhost:8082/nutch

Tomcat Configuration Notes

 * In /etc/tomcat5/server.xml:
 * comment out the AJP connector on port 8009
 * comment out the HTTP connector on port 8080
 * uncomment the proxied HTTP connector on 8082
 * add proxyName to the HTTP connector on 8082
 * could alternatively define proxyName and proxyPort and undefine redirectPort in the HTTP connector on port 8080


 * SELinux (not present on publictest3, but needed eventually)
 * For port 8082, SELinux needs "setsebool -P httpd_can_network_connect=1"
 * Alternatively, for port 8080, SELinux needs "setsebool -P httpd_can_network_relay=1"


 * manager and admin (in fact, all users) defined in /etc/tomcat5/tomcat-users.xml


 * Recommended, but not done here, change the shutdown password in server.xml (default is SHUTDOWN)

Deployment Plan
= References =