help centre
For more info visit status.atmail.com

How can we help?


Search our knowledge base for answers to
common questions and latest updates.



My activities New request

Follow

Installing Apache Solr with Dovecot for fulltext search results

Nathan Salt -

PROBLEM

I want to perform full text search

ENVIRONMENT

  • On-Premise Server + WebMail Installations: Version 6.0 > Current Version
  • Webmail Only Installations: Version 6.0 > Current Version

CAUSE

I want to perform full text search

RESOLUTION

Tutorial on installing Fulltext search with Dovecot + Apache Sol + Lucene + Atmail 6 ( CentOS/Fedora/Linux )

So, your a modern day mail user - You have several folders, each with over 5,000+ messages. Searching your mail for previous clients, customers, friends and family is a pain, slow, and search results are not what you want. The default IMAP search feature just can't scale to modern day usage.
Using the fulltext-search feature of Dovecot ( FTS ) you can index your mailbox with fulltext search capabilities, with 10x speed improvement, and search results that are relevant! This will enable you to login via the Atmail 6 Webmail interface and data-mine your mail better.
The FTS/Solr feature of Dovecot enables the Text/Body keywords on the IMAP search to call the index server.

Download

First, download java for your environment. Java 1.5+ is required

Download from Sun and install under /opt/

http://java.sun.com/javase/downloads/5u21/jdk

cd /opt/ ; wget [large-url-from-sun]
chmod 755 filename.bin
./filename.bin

This will install and extract Java out into the /opt/jdk* directory.

Next, configure the alternatives command to find the new java location.

# alternatives --install /usr/bin/java java /opt/jdk1.5.0_21/bin/java 2
# alternatives --config java
There are 2 programs which provide 'java'.

Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.4.2-gcj/bin/java
2           /opt/jdk1.5.0_21/bin/java

Enter to keep the current selection[+], or type selection number: 2

Next, run

# java -version
java version "1.5.0_21"

And you should see the version string configured via the new Java

Install Ant

The Apache Ant project is required to build the Solr server, download the latest Ant version and install under /opt/

http://ant.apache.org/bindownload.cgi

cd /opt/ ; tar xfvz  apache-ant-1.7.1-bin.tar.gz

Next, update your PATH to reflect the ant/bin directory

Example using bash

vi ~/.bash_profile

set the path to Ant

Replace with your real Ant directory/version

PATH=$PATH:$HOME/bin:/opt/apache-ant-1.7.1/bin/

Then, for the current session run

# export PATH=$PATH:$HOME/bin:/opt/apache-ant-1.7.1/bin/

Then, check ant is available to execute in the path

# ant -v
Apache Ant version 1.7.0 compiled on December 13 2006

Download Solr

http://lucene.apache.org/solr/

Download Apache solar and install under /opt/

# cd /opt/ ; wget http://apacheurl/

# tar xfvz apache-solr*

# cd /opt/apache-solr-1.4.0/example/

Copy the Dovecot XML file as the default schema

cp /usr/local/atmail/server_source/dovecot/doc/solr-schema.xml  solr/conf/schema.xml

Next, run the daemon

# cd /opt/apache-solr-1.4.0/

# ant example

Then you should receive various output, and

BUILD SUCCESSFUL
Total time: 34 seconds

Next, you need to start the Solr daemon

cd example

nohup java -jar start.jar &

You should see a successful bind

2009-11-29 19:38:53.961::INFO:  Started SocketConnector @ 0.0.0.0:8983

Setup firewall access

By default, Solr should only be accessible via the localhost interface, to the local server. Do not open the service to the world for security issues

Verify service working

# telnet localhost 8983
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
GET /solr/

...Welcome to Solr..

Next, using an outside host, make sure the connection to port 8983 cannot be made - You want only the localhost to access Solr for security.

Congratulations - Solr is installed!

Startup daemon
You could write your own start shell script, or a simple approach is to add to /etc/rc.local to startup on boot.

/etc/rc.local

# Start Solr
cd /opt/apache-solr-1.4.0/example/
nohup java -jar start.jar &

Dovecot recompile
The next step required dovecot to be recompiled with the solr plugin install

# service atmailserver stop

# cd /usr/local/atmail/server_source/dovecot/

# ./configure --prefix=/usr/local/atmail/mailserver/ --with-mysql --with-docs=no --with-solr

# make install

Dovecot binary will be replaced, verify the FTS plugin is available

# ls -l /usr/local/atmail/mailserver/lib/dovecot/ | grep solr
-rw-r--r-- 1 root   root   167408 Nov 29 19:48 lib21_fts_solr_plugin.a
-rwxr-xr-x 1 root   root     1195 Nov 29 19:48 lib21_fts_solr_plugin.la
-rwxr-xr-x 1 root   root   108649 Nov 29 19:48 lib21_fts_solr_plugin.so

Edit the Dovecot config for FTS
Edit

/usr/local/atmail/mailserver/etc/dovecot.conf

Find

protocol imap {
mail_plugins = quota imap_quota

Replace with

protocol imap {
mail_plugins = quota imap_quota fts fts_solr

Next find

# Maildir configuration. Quota checks are done via the maildirsize - faster
plugin {
quota = maildir:user
quota_rule = *:storage=5GB
}

Add

# Maildir configuration. Quota checks are done via the maildirsize - faster
plugin {
fts = solr
fts_solr = break-imap-search url=http://localhost:8983/solr/
quota = maildir:user
quota_rule = *:storage=5GB
}

Note the line

fts_solr = break-imap-search debug url=http://localhost:8983/solr/

Since the Solr server supports context sensitive and lucense search terms, this is not "technically" RFC compliant - However end users will only benefit from this new feature, not be hindered.

Next, restart services

# service atmailserver restart

Test index

# telnet localhost imap
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Atmail IMAP/POP3 server ready
1 login user@domain password
2 select Inbox
3 SEARCH text "keyword"

The first search will take longer, this is the first index of the mailbox, dovecot will send the results via HTTP/XML to Solr to store and retrieve.

* OK Indexed 11% of the mailbox, ETA 1:12
* OK Indexed 30% of the mailbox, ETA 1:07
* OK Indexed 49% of the mailbox, ETA 0:52
* OK Indexed 61% of the mailbox, ETA 0:43
* OK Indexed 80% of the mailbox, ETA 0:22
* OK Mailbox indexing finished
* SEARCH 857 1486 1526 1544 1589 1590 1591 1608 1737 1753 2660 2864 3967 4124 4127 4129 4159 4235 4254 4273 4682 4683 4684 4685 4770 5132 5467 5470 5735 5788 5838 5903 5904 5905 5906 5907 5996 6024 6113 6144
1 OK Search completed (114.650 secs)

Run the same query again:

1 search text "sync"
* SEARCH 857 1486 1526 1544 1589 1590 1591 1608 1737 1753 2660 2864 3967 4124 4127 4129 4159 4235 4254 4273 4682 4683 4684 4685 4770 5132 5467 5470 5735 5788 5838 5903 5904 5905 5906 5907 5996 6024 6113 6144
1 OK Search completed (0.000 secs).

Reduced from 114secs to 0 secs for a 10,000 message folder running on a P4 with 1GB RAM - Thank you Solr !

Each time you receive new mail and issue a search, dovecot will index the "new" messages to Solr to update the index accordingly.

Now, kick back and data-mine your IMAP folders for your valuable information. Launch Atmail 6 and use the new Webmail interface to search, and enjoy the new results!

Have more questions? Submit a request

Comments


Contact our support team


+61 (7) 5357 6605       support@atmail.com