User Tools

Site Tools


info:mail

This is an old revision of the document!


Email server using postfix and dovecot

For a small-scale mail server, it is best to use OS users for authentication. In this setup, users have a home directory that stores the mail directory (~/mail/).

Address Mapping

To map email addresses to OS users, postfix uses the file /etc/postfix/virtual, which is specified in main.cf, property virtual_alias_maps. It contains mappings like the following:

address@domain.org target_user1, target_user2, forward@address.org

The domain of the incoming address must be one of the domains listed in main.cf property virtual_alias_domains. User aliases can be defined in /etc/aliases.

After editing the file, run the following commands:

# postmap /etc/postfix/virtual
# postfix reload

Sieve

Dovecot allows to process incoming mails with sieve scripts. This is very useful to file messages into folders with skipping the inbox. By default, there is a global sieve script that files all spam mails into the trash folder. It has the following content:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" ["YES"] {
  fileinto "spam";
  stop;
}

To add custom sieve scripts, I recommend installing the Thunderbird Sieve extension. It allows the creation and syntax checking of sieve scripts directly on the server. On Debian, it is available as xul-ext-sieve package.

For specific information about managing sieve on antiguru.de look at moritz:antiguru.de.

Mail aggregation aka unified inbox (proposal)

Warning: NYI

The setup described does now fulfill the expectations. Amavisd does now scan and decorate mails from fetchmail. But it has not yet been implemented on the 'real' server.

After lots of trials which didn't work, I decided to move away from the production server. On my trusted Sparcstation, I set up a testbed system, and seemingly, this works.

First, you've got to configure postfix. To the end of /etc/postfix/main.cf, add (but only if the stanza is not there already, in our conguration, it was!):

mailbox_command = /usr/lib/dovecot/deliver

To the end of /etc/postfix/master.cf, I added

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================

amavisfeed unix    -       -       n       -       2     smtp
     -o smtp_data_done_timeout=1200
     -o smtp_send_xforward_command=yes
     -o disable_dns_lookups=yes
     -o max_use=20
# this sets the name and default options for the amavis smtp feed

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================

127.0.0.1:10025 inet n    -       n       -       -     smtpd
     -o content_filter=
     -o smtpd_delay_reject=no
     -o smtpd_client_restrictions=permit_mynetworks,reject
     -o smtpd_helo_restrictions=
     -o smtpd_sender_restrictions=
     -o smtpd_recipient_restrictions=permit_mynetworks,reject
     -o smtpd_data_restrictions=reject_unauth_pipelining
     -o smtpd_end_of_data_restrictions=
     -o smtpd_restriction_classes=
     -o mynetworks=127.0.0.0/8
     -o smtpd_error_sleep_time=0
     -o smtpd_soft_error_limit=1001
     -o smtpd_hard_error_limit=1000
     -o smtpd_client_connection_count_limit=0
     -o smtpd_client_connection_rate_limit=0
     -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
     -o local_header_rewrite_clients=

# this establishes the smtp listener to take back processed messages from amavis

# ==========================================================================
# content filters to match the policies of amavis
# ==========================================================================
# regular incoming mail, originating from anywhere (usually from outside)
# the MX record (or backup mailers) should point to this IP address
# set to your external IP address
192.168.100.233:smtp inet  n  -  n  -  -  smtpd
  -o content_filter=amavisfeed:[127.0.0.1]:10040

# ==========================================================================
# incoming mail from fetchmail, considered externally originating
# (add 'smtphost localhost/2345' to the poll section in .fetchmailrc)
127.0.0.1:2345 inet  n  -  n  -  -  smtpd
  -o content_filter=amavisfeed:[127.0.0.1]:10041
  -o smtpd_client_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8

# ==========================================================================
# IP address to be used by internal hosts for mail submission
127.0.0.1:smtp inet  n  -  n  -  -  smtpd
  -o content_filter=amavisfeed:[127.0.0.1]:10042
  -o mynetworks=127.0.0.0/8
  -o smtpd_client_restrictions=permit_mynetworks,reject

# ==========================================================================
# locally originating mail submitted on this host through a sendmail binary
pickup     fifo  n  -  n  60  1  pickup
  -o content_filter=amavisfeed:[127.0.0.1]:10042
# ==========================================================================

Important: There may not be whitespace before the first lines of the rules in master.cf, otherwise they'll be considered a continuation of the last rule starting at the first character, even if they don't make sense, without error message.

Also important: further up in master,cf, there might be duplicate entries like

#smtp      inet  n       -       -       -       -       smtpd
#pickup    fifo  n       -       -       60      1       pickup

You need to disable them as shown here. Otherwise, at least when you forget to comment out the smtp line, postfix cannot start ('port already in use').

So, what does this configuration achieve? We set up postfix to take back mails (from amavis) on port 10025, and we prepare postfix to

  • listen on the external IP address as smtp server and hand over mail from there to amavis at port 10040
  • listen on the local loopback email address at port 2345 as smtp server and hand over mail from there to amavis at port 10041
  • listen on the local loopback email address as smtp server and hand over mail from there to amavis at port 10042

Backward compatibility: before, postfix was setup to listen to the smtp port on all addresses (in the smtp service line we commented out, see above). Now, to differentiate, we set up listeners on the loopback address and on the external address. So, in this setup, there's no universal local smtp listener (for all local IP addresses). If you need such listeners, you'll have to add a rule for each of them to the postfix configuration, because the preceding rules don't allow you to adress the smtp port globally any more in a rule.

Of course, this setup has to be matched by a corresponding configuration for amavisd-new. Apart from uncommenting the spam filter lines in /etc/amavis/15-content-filter-mode and adding “.localhost” to the @local_domains_acl array in /etc/amavis/05-domain_id, (is this needed? will have to find out) all other changes could be made to /etc/amavis/conf.d/50-user. I added:

# be more verbose
log_level =2 ;

# ports we listen on
$inet_socket_port = [10040,10041,10042];

# policies for these ports
# cf. /etc/postfix/master.cf for the corresponding setup

$interface_policy{'10040'} = 'EXT';
$interface_policy{'10041'} = 'EXT-FM';
$interface_policy{'10042'} = 'INT-HOST';

# some global settings
# do not quarantine spam
$virus_quarantine_to = $QUARANTINEDIR;
$spam_quarantine_to = undef;
# Spam levels
 # always tag
$sa_tag_level_deflt  = undef;
$sa_tag2_level_deflt = 5;
$sa_kill_level_deflt = 20;


### POLICIES ###
# regular incoming mail, originating from anywhere (usually from outside)
$policy_bank{'EXT'} = {
  # just use global settings, no special overrides
};

# incoming mail from fetchmail, considered externally originating
$policy_bank{'EXT-FM'} = {
     log_level => 2, 
    # no bounces for spam, not even for score below spam_dsn_cutoff_level_maps:
  final_spam_destiny => D_DISCARD,
};
# mail locally submitted on the host on which MTA runs
$policy_bank{'INT-HOST'} = {
    # NOTE: this is just an example; ignoring internally generated spam
    # may not be such a good idea, consider zombified infected local PCs
  bypass_spam_checks_maps   => [ 1 ],
  bypass_banned_checks_maps => [ 1 ],
  final_spam_destiny   => D_PASS,
  final_banned_destiny => D_PASS,
};

Source: http://www.amavis.org/amavisd-new-docs.html#pbanks

So far, so good. I've only got one question: how does amavisd detect which mail messages are to be considered local/inbound? By exploiting the xforward option of postfix? The correct setting of local_domains_maps is also important:

“If recipient matches @local_domains_maps it goes to an inside recipient (inbound or all-internal), otherwise mail is outbound.” [from the amavisd mailing list]

If the automatic decision if a mail is considered local does not work as wanted, we'll have to investigate the originating flag of amavisd-new. If set to 1, this means that mail is considered to be outbound (originating from the server), and should not be processed. This flag could be added to the policy banks laid out above.

As last step, you'll have to add the port for the fetchmail rule above to the user's .fetchmailrc:

poll pop.gmx.net protocol pop3:
 username "XXXXXX" password "..........", is someuser here smtphost localhost/2345
  # use secure connection relying on CA certificates
  ssl
  # do not delete from server (for testing)
  keep
  # get all messages, not just the ones that arrived after the last poll (use this after commenting out keep)
  #fetchall

Note: if you've got an active firewall and experience timeouts running fetchmail, you'll have to open the outgoing port for pop3s.

Result: this will process mails from fetchmail through the EXT-FM policy and these mails will be decorated with spam markings.

The next step will now be to move this system to the production machine.

Details:

To poll mail from external POP3 accounts and make it accessible by IMAP here, the simple method is to use fetchmail to check out the messages from the server and then directly hand them to dovecot's deliver. This means we'll not use an MTA (Mail Transfer Agent) like postfix (which basically means sending an email between the different programs), but just use an MDA (Mail Delivery Agent), basically copying or piping the message from one program to the other.

So, we'll use fetchmail in conjunction with dovecot's deliver as MDA.

Because running fetchmail system-wide (as user fetchmail/group nobody) creates lots of seemingly unsolvable permission problems when attempting to use the MDA to deliver to user's inboxes, we instead choose to run everything for the individual user. (It would be possible to run the MDA as root user, either setuid or via sudo, but this is a dangerous kludge, and unnecessary, as the method described here works.)

So, after installing fetchmail, we have to create a user-specific config and adjust the permissions so that fetchmail will find them acceptable. In the user's directory, logged in as that user, execute

$ touch ~/.fetchmailrc
$ chmod 0600 /etc/fetchmailrc

Fetchmail is ssl-enabled and can either use one of the root CA certificates or check if the remote ssl fingerprint matches the one stored locally (for self-signed certificates). For the usual freemail providers, the first method should work.

Fetchmail can be set up to use dovecot's deliver as local delivery agent (aka lda) to deliver the retrieved mail to the local mail directories. Compared to other solutions like the popular procmail, this is the preferred solution as it will update dovecot's folder indices and allow filtering via the sieve plugin (as mentioned above). Using procmail will not update indices and only allow filtering via procmail's own mechanism.

So, we edit our ~/.fetchmailrc like this:

set syslog

poll pop.gmx.net protocol pop3:
 username "XXXXXXX" with password "........." is someuser here
 # use secure connection relying on CA certificates
 ssl
 # do not delete from server (for testing)
 keep
 # get all messages, not just the ones that arrived after the last poll (use this after commenting out keep)
 #fetchall
 # dovecot lda (Debian location)
 mda "/usr/lib/dovecot/deliver"

Now, start fetchmail:

$ fetchmail -v

You can look (with the correct permissions) at /var/log/mail.log[err,&c] for messages about errors or success or failure.

If you don't see error messages here, set up a cron job to have fetchmail poll the POP3 server(s) regularly:

$ crontab -e

Add a line like this to your crontab:

*/5 * * * * /usr/bin/fetchmail &> /dev/null

(this will poll every 5 minutes). Then, save and exit the editor. Now, the fetchmail job will run unattended as set in the crontab.

Once this works, one could filter mail using the sieve method described above.

Spam checker

While this method is very simple and easy to set up, integrating spam and antivirus filters takes some more work.

If you only want spam filtering, this is quite easy. After installing and configuring spamassassin (on which I won't elaborate), change the mda line in your .fetchmailrc like

 mda "spamc -e /usr/lib/dovecot/deliver"

This will filter your mail through spamassassin, which will 'decorate' it with headers showing how spammy spamassassin thinks the mail message is. The filtering will then be handled by sieve, as shown above.

While this works on my test machine, it fails on the production server, because there, users do not have permission to access spamc/spamd. As I wanted to move on to the setup described above, I didn't innvestigate how to change this.

Additional AV filter

Regarding antivirus filtering, the picture changes, though.

Apparently, amavisd-new, which is installed here, does not offer to pipe mail messages through its scanning system. So the method employed till now has to be abandoned. It's no longer possible to just use an MDA, we'll have to switch to our MTA (postfix) to process to mail messages.

This turns out to be less complicated than it sounds. (Of course, I assume that the basic mail processing system already works correctly and is set uo to send and receive mail, scan received mail for viruses and spam, and deliver to dovecot's inboxes.)

The setup would then look like this: fetchmail → postfix → amavisd & spamassassin → lda → dovecot.

How could this be implemented?

  • omitting the mda line from .fetchmailrc leads to mail messages being processed by the local mail system (i.e. postfix)
  • postfix is already configured to pass mails through amavisd
  • Amavisd calls spamassassin

While this basically works, it turned out that amavisd, when configured to ignore local mails, also ignores all mails polled by fetchmail. At first, it seemed that there was no sane way to tell amavisd to behave differently. After digging through lots of stupid blog posts of the “I'm so cool” type, I finally discovered the way how this might be achieved. The key is to have several ports listening for mail and define matching policies for amavisd. See at the beginning of this section.

Q&A

Q: Which variant should I choose?

A: For a LAN aggregator, I'd choose the first variant, maybe enhanced by adding spamassassin as shown. That way, there's no need to run postfix. You could then scan for viruses in your MUA (email client).

For a server with a public IP, the last variant is the right one. That way, you'll get a unified inbox for all accounts polled by fetchmail, with mails scanned for viruses and spam, reachable from wherever you can freely access the internet.

Q: Don't I have to use procmail? This can be read all over the Internet?

A: While one could use procmail, this has lots of disadvantages in conjunction with dovecot. Procmail does not update the folder indices when delivering mail, and there's no way to run sieve scripts on the incoming mail. So, use deliver (dovecot's lda) instead, which does all this.

Q: Do you have to change dovecot's configuration to make this work?

A: On a fresh install, the lda section should be uncommented, and maybe you'll need to set the postmaster_address (IIRC). On our setup, this had been done already.

Q: How to test this without wreaking havoc? Make a backup copy of ~/mail/ and move it back in case things break?

A: Use your old Sparcstation ;-) (but the other method might work, too).

info/mail.1422008985.txt.gz · Last modified: 2015/01/23 11:29 by hartmut