I’ve been using cyrus imap for about 15 years. I’m probably the last user. Carnegie Mellon recently announced that they were abandoning cyrus-imap. I never tried to get any help from them anyway, so I guess that isn’t a big deal, but it did make clear that I was using an out of date product. I also knew the product to be fragile and brittle, and tools for repair were not really available. Also, I had some things wrong with my cyrus files that were nagging. Overall it was past time to move on.

I didn’t look far for a replacement, in fact I didn’t do much research at all. Dovecot seemed the place to go. So after doing some reading I set about to convert. My plan was to convert first on oregano, my local development machine, and get it working there. I get almost no mail there. Then, once I thought I knew what I was doing, I would convert to it on one of the client websites I maintain where, again, there is very little mail, but there is some, and there are two or three accounts only, and the mail is mostly error things. Not very important. Finally, after those two, I would convert the mail on tarragon, where there is some 13GB of mail for about a dozen or so users.

About the mail spools.

Cyrus has:

/var/lib/imap – the cyrus-imap control directory, where they keep track of stuff, and keep the cyrus databases, and

/var/spool/imap – the cyrus mail spool.

But I have all this on a btrfs subvolume called mail, symlinked by /mail. So for me

/var/lib/imap -> /mail/lib which is the cyrus imap control stuff, and

/var/spool/imap -> /mail/spool which is where the maildirs are.

Mail directory (mail_location) and Home directory (mail_home).

I am creating a directory /var/vmail for dovecot which is symlinked to /mail/vmail, so as before the mail spools are on the btrfs mail subvolume.

When Dovecot refers to “~” it means the “user’s home” directory – which is NOT the /home/user system location (it could be but doesn’t have to be). Dovecot has the idea of the “home directory” as “a per-user directory where user state is retained”. They call it the “home” directory, and they use “~” to refer to it, but that “home” directory may be /var/vmail/domain/user.

Dovecot refers to both the “home” directory and the “mail” directory. “home” is where the state stuff is stored. “Mail” is where the mail spool is, the maildir. They can be totally separate, or one could be inside the other, or they can theoretically be the same – but that can cause issues. I prefer mail inside of home. So home=/var/vmail/<domain>/<user>/ and inside it mail=/var/vmail/<domain>/<user>/mail so the parameters, and where set, are:

mail_home = /var/mail/%d/%n (returned with userdb lookup, see auth-sql.conf.ext)

mail_location = maildir:~/mail (set in 10-mail.conf)

Password database and User databases

A password database (passdb), which could be simply a file, is where Dovecot authenticates a user. There can be more than one – so you can authenticate virtual users differently than system users. There are many kinds, some are yes/no (like pam), and some are lookup (sql).

A user database (userdb) is consulted after authentication to provide uid, gid, home (mail_home) and some other options – but basically is the data needed to figure out where the mail should be put.

These databases can actually be the same, and they also can be sql or files or ldap or other stuff.

I am going to use a sql database for passdb, in addition to the /etc/passwd for system users. I have the wmb_auth database already, which has userid_key, userid, mailpass. Mailpass is md5 there.

You describe these with the userdb { } and passdb { } entries, and in there you stipulate a ‘driver’ for what kind of access, like file, sql. I will use sql for passdb, but it makes sense to use ‘static’ for the userdb driver, since I will have uid=vmail, gid=vmail and home=/var/mail/%d/%n. No real need to do any lookup. But see comment under static user database about what happens if there is no such user.

There is a feature called prefetch which is useful when userdb and passdb queries come from the same database, but since I’m using static there is no reason to use prefetch.

The file dovecot-sql.conf.ext was copied from usr/share/doc/dovecot/example-config, and contains the query, adjusted as follows for the wmb_auth database.

password_query = select userid as username, mailpass as password, maildomain as domain from userids where userid=’%n’ and domain=’%d’

Setup steps

useradd -M -r -d /var/vmail -s /sbin/nologin vmail

mkdir /mail/vmail

ln -s /mail/vmail /var/vmail

chown vmail:vmail /var/vmail

The dovecot conf.d files I have updated

10-auth.conf realms, auth mech, disable plaintext, include auth-sql.conf.ext

10-logging.conf where and what to log, including debugging

10-mail.conf mail_location, inbox

10-master.conf what services/protocols to run, lmtp socket, auth socket

10-ssl.conf keys

15-lda.conf empty – no special settings

15-mailboxes.conf set up Trash, Junk, Sent, Drafts

20-imap.conf empty – no special settings

20-lmtp.conf empty – no special settings

20-pop3.conf commented out

auth-sql.conf.ext passdb and userdb specs, drivers

dovecot.sql.conf.ext included by auth-sql, mysql connect, password query, password scheme

Handling users and domain

There are some users who could (but under cyrus don’t) have multiple mail accounts. including my own: dee@wmbuck.net, dee@wmbuck.com, dee@thegraygeek.com. I won’t list the others on this public forum. I don’t currently have multiple accounts set up in wmb_auth for use by cyrus-imapd. Each user in wmb_auth actually only appears once, with one of the mail accounts. Now I have added a field “maildomain” to wmb_auth, which will be used by dovecot for the “%d” in /var/vmail/%d/%n, and this will permit having multiple entries for those users, so the information can be kept for different mail accounts. I’ve done this on oregano and gotten it working.

But, I cannot change the wmb_auth on the wmbuck.net website to include the additional accounts yet. There is a problem with having multiple entries in the database with the same username (“%n”, e.g. “dee”) under pam_mysql – specifically, if pam_mysql gets multiple records on the username lookup he barfs and none of the user’s accounts work. If I put in ralph@ralphhome.com along with ralph@ralphbusiness.com then both of them stop working.

Postfix and authenticating access to smtpd

Postfix wants to validate users and passwords for access to smtpd, which is done with the smtpd_sasl_auth_enable and related directives. But that is cyrus sasl. I have this set up that postfix checks /etc/sasl2/smtpd.conf which says to use saslauthd. Saslauthd in turn is told (in sysconfig) that he should use MECH=pam, and in /etc/pam.d/mail I have setup pam_mysql.so to look up in the wmb_auth database. Pretty convoluted, but that’s how you do it with cyrus-imap.

With dovecot, I won’t be using cyrus_sasl, but I can tell postfix to use dovecot to authenticate external users with:

smtpd_sasl_type = dovecot

smtpd_sasl_path = private/auth

with matching service auth { unix_listener /var/spool/postfix/private/auth { } } in dovecot.conf

smtpd_sasl_auth_enable = yes

So users can connect to send mail through wmbuck.net and they can enter a username and password, and postfix will give it to dovecot, who will check the password just as he does for imap login, and tell postfix if it is ok.

And I can get rid of pam_mysql.

Changes in postfix (on oregano)

1. Change the sockets for mailbox_transport and virtual_transport from /var/lib/imap/socket/lmtp to private/dovecot-lmtp.

2. SMTPD: change smtpd_sasl_type to dovecot, sasl_path to private/auth.

3. SMTP: no change

No substantive changes in master.cf (changed msa to submission).

Dovecot and ssl

Dovecot documentation has good treatment of the difference between SSL and TLS, and they go on to say that they use the term ssl somewhat indifferently to mean encryption of the channel, whether with ssl or tls. But they then proceed to do something weird that I don’t think they describe well. Maybe they think this is obvious, but it is absolutely not clear to me.

If you set ssl=on or ssl=required on the inet listener for the imap-login service (which will run on 143) it will invoke “SSL” action (as opposed to STARTTLS action) – i.e. it will automatically begin ssl handshaking as soon as the connection is made. And this will completely prevent the MUA from logging in if the MUA thinks we are doing STARTTLS. WTF?

If you set “disallow_plaintext_auth = yes” AND you do NOT set ssl=on/required on the inet_listener for the imap port within the imap_login service, then it will do STARTTLS action. Go figure.

I personally think this is very counter-intuitive.

Conversion using cyrus2dovecot

I cloned this from github into my own /home/dee/git/cyrus2dovecot. I’ve made various changes to it. For using it, I need to feed it a list of user mailboxes, and to generate that list I’ve created a script called /root/cyrustodovecot.sh

In specifying paths:

%h is first character of username

%g is first character of domain name

%d is domain name

%u is username, which is probably full email address

%U is username without the domain name

The mail spool files on Tarragon are in /var/spool/imap/* –> /mail/spool/*.

For system users: under /mail/spool/%h the only one with any actual mail is /mail/spool/d/user/dee. No other system user has mail on tarragon.

For virtual domain users: under /mail/spool/domain/%g/%d/%h/user/%U are the mail spools, e.g. /mail/spool/domain/b/blogs.com/f/user/fred

/mail/spool/domain/s/smith.net/m/user/mary

The backed up tarragon mailboxes are on oregano in /backup/Tarragon/mail/spool so for cyrus2dovecot (at least while we are still practicing).

dovecot_inbox = /var/vmail/%d/%U

For system users (only dee) I will use:

cyrus_inbox = /backup/Tarragon/mail/spool/%h/user/%u

cyrus_sub = /backup/Tarragon/mail/lib/user/%h/%u.sub

cyrus_seen = /backup/Tarragon/mail/lib/user/%h/%u.seen

for the domain users I will use:

cyrus_inbox => /backup/Tarragon/mail/spool/domain/%g/%d/user/%h/%U

cyrus_sub => /backup/Tarragon/mail/lib/%g/%d/user/%h/%U.sub

cyrus_seen => /backup/Tarragon/mail/lib/%g/%d/user/%h/%U.seen

I wrote a script cyrus2dovecot.sh which automates finding the usernames and constucting these strings, and invoked cyrus2dovecot command with these settings for each user.

Problems in the existing cyrus files

Once I got it set up, cyrus2dovecot gave me lots of errors for messages for which it could not find entries in the index. I thought initially this was a problem in cyrus2dovecot, but when I dug into it I find that there are literally thousands of message files that are not in the index, and do not appear to me with my imap client. In a lot of these cases, these are messages which I know to have been moved from an old folder to a new folder. In fact, many of them are in folders that don’t even exist as far a I am concerned – folders I eliminated when moving stuff around. I conclude that what is in the index files is what is showing to me in thunderbird, which is what I want, and the stuff that is being complained about is stuff that is removed, or moved, and which should have been deleted.

It looks to me like cyrus2dovecot is going to copy this stuff, and create old folders, even through there are no index files. To prevent this, I’ve modified cyrus2dovecot so that it skips mail files which don’t appear in the index.

On a test run just completed, cyrus2dovecot says it processed 65674 messages, but the number of “skipped because not in the index” messages is 107322. This is only for user dee, which of course is by far the biggest.

Problems with INBOX

One of the reasons I would like to migrate, and get a fresh start with dovecot is that I know that my mailboxes are screwed up.

One problem is that after setting up a user, sometimes – in some way I don’t understand, somebody decides there should be a mailbox created called INBOX, which should be placed within the outermost root (which I think is actually the INBOX). It then shows up inside Thunderbird as INBOX/INBOX, and often inside the inner one is Trash. This is true for user dee, and also for some of the domain users.

Also some time long ago, the cyrus imap mailboxes for the system user dee got set up in some ancient way with a personal namespace of INBOX. I have never really understood this, I don’t understand about personal namespaces, and the use of INBOX in the cyrus files, instead of user/dee/…

I am certain that all these problems are fundamentally owing to some mistake on my part in setting up cyrus-imap, or perhaps in migrating from UW imap which I did probably more than 15 years ago.

Unsubscribed folders

There is a similar, and maybe worse problem involving subfolders. Not only is it the case that messages which have been removed are still present – it is also the case that entire folders which have been moved or renamed are still present. For example, an old folder for a friend, like Melody, was a long time ago moved to within a folder called Friends (i.e. Friends/Melody). Later on, I decided to replace folder Friends in favor of a folder called People (so it has subfolder People/Melody).

After processing the mail database from cyrus, the new dovecot folders have entries for a folder Melody, a folder Friends/Melody and a folder called People/Melody, but only the folder People/Melody is marked as subscribed. If I tell Thunderbird to subscribe to all three, the folder Melody has 544 messages going from 4/13/2008 up to 8/25/2011, the folder Friends/Melody has 583 messages going from 4/13/2008 up to 1/2/2012, and the folder People/Melody has 1578 messages, going from 4/13/2008 up to 7/24/2017.

Doing Tarragon

Getting ready to do tarragon on a new instance. That is why I am updating this document. The git repository I had created on oregano for the cyrus2dovecot has been cloned into /home/dee/.git on tarragon so it will be available on the new system.

Results on Tarragon

There were only 2 real problem that any of the users had. The easy one was that some of their folders were not subscribed. I had to walk them through how to resubscribe. The more difficult was that some of the subfolders were not picked up, and I had to manually edit the “Subscriptions” and put some of them back in.