Exim is designed to work efficiently on systems that are permanently connected to the Internet and are handling a general mix of mail. In such circumstances, most messages can be delivered immediately. Consequently, Exim does not maintain independent queues of messages for specific domains or hosts, though it does try to send several messages in a single SMTP connection after a host has been down, and it also maintains per-host retry information.
Policy controls are now an important feature of MTAs that are connected to the Internet. Perhaps their most important job is to stop MTAs being abused as ``open relays'' by misguided individuals who send out vast amounts of unsolicited junk, and want to disguise its source. Exim provides flexible facilities for specifying policy controls on incoming mail:
Exim 4 (unlike previous versions of Exim) implements policy controls on incoming SMTP mail by means of Access Control Lists (ACLs). Each list is a series of statements that may either grant or deny access. They are used at two places in the SMTP dialogue while receiving a message: after each RCPT command, and at the very end of the message. The sysadmin can specify conditions for accepting or rejecting individual recipients or the entire message (see chapter 37). Denial of access results in an SMTP error code.
When a message has been received, either from a remote host or from the local host, but before the final acknowledgement has been sent, a locally supplied C function can be run to inspect the message and decide whether to accept it or not (see chapter 38). If the message is accepted, the list of recipients can be modified by the function.
After a message has been accepted, a further checking mechanism is available in the form of the system filter (see chapter 39). This runs at the start of every delivery process.
Every message handled by Exim is given a message id which is sixteen characters long. It is divided into three parts, separated by hyphens, for example 16VDhn-0001bo-00. Each part is a sequence of letters and digits, normally representing a number in base 62. However, in the Darwin operating system (Mac OS X) and when Exim is compiled to run under Cygwin, base 36 is used instead, because the names of files in those systems are not case-sensitive.
The first six characters are the time the message was received, as a number in seconds - the normal Unix way of representing a time of day. If the clock goes backwards (due to resetting) in a process that is receiving more than one message, the later time is retained.
After the first hyphen, the next six characters are the id of the process that received the message.
The final two characters, after the second hyphen, are used to ensure uniqueness of the id. There are two different formats:
If the localhost_number option is not set, uniqueness is required only within the local host. This portion of the id is 00 except when a process receives more than one message in a single second, when the number is incremented for each additional message.
If the localhost_number option is set, uniqueness among a set of hosts is required. This portion of the id is set to the base 62 encoding of
<sequence number> * 256 + <host number>
where <sequence number> is the count of messages received by the current process within the current second. As the maximum value of the host number is 255, this allows for a maximum value of 14 for the sequence number. If this limit is reached, a delay of one second is imposed before reading the next message, in order to allow the clock to tick and the sequence number to get reset.
The only way Exim can receive mail from a remote host is using SMTP over TCP/IP, in which case the sender and recipient addresses are tranferred using SMTP commands. However, from a locally running process (such as a user's MUA), there are several possibilities:
If the process runs Exim with the -bm option, the message is read non-interactively (usually via a pipe), with the recipients taken from the command line, or from the body of the message if -t is also used.
If the process runs Exim with the -bS option, the message is also read non-interactively, but in this case the recipients are listed at the start of the message in a series of SMTP RCPT commands, terminated by a DATA command. This is so-called ``batch SMTP'' format.
If the process runs Exim with the -bs option, the message is read interactively, using the SMTP protocol. A two-way pipe is normally used for passing data between the local process and the Exim process.
A local process may also make a TCP/IP call to the host's loopback address (127.0.0.1) or any other of its IP addresses. Exim does not treat the loopback address specially. It treats all such connections in the same way as connections from other hosts.
In the three cases that do not involve TCP/IP, the sender address is constructed from the login name of the user that called Exim and a default qualification domain (which can be set by the qualify_domain configuration option). For local or batch SMTP, a sender address that is passed using the SMTP MAIL command is ignored. However, the system administrator may allow certain users to specify a different sender address unconditionally, or all users to specify certain forms of different sender address. The -f option or the SMTP MAIL command is used to specify these different addresses. See section 5.2 for details of trusted users, and the untrusted_set_sender option for a way of allowing untrusted users to change sender addresses.
Messages received by either of the non-interactive mechanisms are always accepted, provided they are syntactically valid. However, when a message is being received using the SMTP protocol (either over TCP/IP, or interacting with a local process), policy controls can be applied at the time of reception, and the message can be rejected if it does not accord with local policy.
Exim can be configured not to start a delivery process when a message is received; this can be unconditional, or depend on the number of incoming SMTP connections or the system load. In these situations, new messages wait on the queue until a queue-runner process picks them up. However, in standard configurations under normal conditions, delivery is started as soon as a message is received.
When Exim accepts a message, it writes two files in its spool directory. The first contains the envelope information, the current status of the message, and the headers, and the second contains the body of the message. The names of the two spool files consist of the message id, followed by -H for the file containing the envelope and headers, and -D for the data file.
By default all these message files are held in a single directory called input inside the general Exim spool directory. Some operating systems do not perform very well if the number of files in a directory gets very large; to improve performance in such cases, the split_spool_directory option can be used. This causes Exim to split up the input files into 62 sub-directories whose names are single letters or digits.
The envelope information consists of the address of the message's sender and the address(es) of the recipient(s). This information is entirely separate from any addresses contained in the headers. The status of the message includes a list of recipients who have already received the message. The format of the first spool file is described in chapter 48.
Address rewriting that is specified in the rewrite section of the configuration (see chapter 30) is done once and for all on incoming addresses, both in the header and the envelope, at the time the message is accepted. If during the course of delivery additional addresses are generated (for example, via aliasing), these new addresses get rewritten as soon as they are generated. At the time a message is actually delivered (transported) further rewriting can take place; because this is a transport option, it can be different for different forms of delivery. It is also possible to specify the addition or removal of certain header lines at the time the message is delivered (see chapters 14 and 23).
A message remains in the spool directory until it is completely delivered to its recipients or to an error address, or until it is deleted by an administrator or by the user who originally created it. In cases when delivery cannot proceed - for example, when a message can neither be delivered to its recipients nor returned to its sender, the message is marked ``frozen'' on the spool, and no more deliveries are attempted.
An administrator can ``thaw'' such messages when the problem has been corrected, and can also freeze individual messages by hand if necessary. In addition, an administrator can force a delivery error, causing a bounce message to be sent.
There is an option called auto_thaw, which can be used to cause Exim to retry frozen messages after a certain time. When this is set, no message will remain on the queue for ever, because the delivery timeout will eventually be reached. Delivery failure reports that reach this timeout are discarded. There is also an option called timeout_frozen_after, which simply discards frozen messages after a certain time.
While Exim is working on a message, it writes information about each delivery attempt to the main log file. The includes successful, unsuccessful, and delayed deliveries for each recipient (see chapter 44). The log lines are also written to a separate message log file for each message. These logs are solely for the benefit of the administrator, and are normally deleted along with the spool files when processing of a message is complete.
All the information Exim itself needs to set up a delivery is kept in the first spool file with the headers. When a successful delivery occurs, the address is immediately written at the end of a journal file, whose name is the message id followed by -J. At the end of a delivery run, if there are some addresses left to be tried again later, the first spool file (the -H file) is updated to indicate which these are, and the journal file is then deleted. Updating the spool file is done by writing a new file and renaming it, to minimize the possibility of data loss.
Should the system or the program crash after a successful delivery but before the spool file has been updated, the journal is left lying around. The next time Exim attempts to deliver the message, it reads the journal file and updates the spool file before proceeding. This minimizes the chances of double deliveries caused by crashes.
The main delivery processing elements of Exim are called routers and transports, and collectively these are known as drivers. Code for a number of them is provided in the source distribution, and compile-time options specify which ones are included in the binary. Run time options specify which ones are actually used for delivering messages.
A router is a driver that operates on an address, either determining how its delivery should happen, by routing it to a specific transport, or converting the address into one or more new addresses (for example, via an alias file). A router may also explicitly choose to fail an address, causing it to be bounced.
A transport is a driver that transmits a copy of the message from Exim's spool to some destination. There are two kinds of transport: for a local transport, the destination is a file or a pipe on the local host, whereas for a remote transport the destination is some other host. A message is passed to a specific transport as a result of successful routing. If a message has several recipients, it may be passed to a number of different transports.
An address is processed by passing it to each configured router in turn, subject to certain pre-conditions, until one accepts it or specifies that it should be bounced. We'll describe this process in more detail shortly. As a simple example, the diagram below illustrates how each recipient address in a message is processed in a small configuration of three routers that are configured in various ways.
To make this a more concrete example, we'll describe it in terms of some actual routers, but remember, this is only an example. You can configure Exim's routers in many different ways, and there may be any number of routers in a configuration.
The first router in a configuration is often one that handles addresses in domains that are not recognized specially by the local host. These are typically addresses for arbitrary domains on the Internet. A pre-condition is set up which looks for the special domains known to the host (for example, its own domain name), and the router is run for addresses that do not match. Typically, this is a router that looks up domains in the DNS in order to find the hosts to which this address routes. If it succeeds, the address is queued for a suitable SMTP transport; it it does not succeed, the router is configured to fail the address.
In the example, the second and third routers can only be run for domains that are special to the local host, for which the pre-condition for the first router is not met. The second router does redirection - also known as aliasing and forwarding. When it generates one or more new addresses from the original, each of them is routed independently from the start. Otherwise, the router may cause an address to fail, or it may simply decline to handle the address, in which case it is passed on to the next router.
The final router in many configurations is one that checks to see if the address belongs to a local mailbox. The pre-condition may involve a check to see if the local part is the name of a login account, or it may look up the local part in a file or a database. If its pre-condition is not met, or if the router declines, we have reached the end of the routers. When this happens, the address is bounced.
As explained in the example above, a number of conditions are checked before running a router. If any of these pre-conditions are not met, the router is skipped, and the address is passed to the next router. Sometimes you want to fail a delivery when some conditions are met but others are not, instead of passing the address on for further routing. You can do this by having a second router that explicitly fails the delivery when the relevant conditions are met.
When all the conditions on a router are met, the router is run. What happens next depends on the outcome, which is one of the following:
accept: The router accepts the address, and either queues it for a transport, or generates one or more ``child'' addresses. Processing the original address ceases, unless the unseen option is set on the router. This option can be used to set up multiple deliveries with different routing (for example, for keeping archive copies of messages). When unseen is set, the address is passed to the next router. Normally, however, an accept return marks the end of routing.
If child addresses are generated, Exim checks to see whether they are duplicates of any existing recipient addresses. During this check, local parts are treated as case-sensitive. Duplicate addresses are discarded. Each of the remaining child addresses is then processed independently, starting with the first router by default. It is possible to change this by setting the redirect_router option to specify which router to start at for child addresses. Unlike pass_router (see below) the router specified by redirect_router may be anywhere in the router configuration.
pass: The router recognizes the address, but cannot handle it itself. It requests that the address be passed to another router. By default the address is passed to the next router, but this can be changed by setting the pass_router option. However, (unlike redirect_router) the named router must be below the current router (to avoid loops).
decline: The router declines to accept the address because it does not recognize it at all. By default, the address is passed to the next router, but this can be prevented by setting the no_more option. When no_more is set, all the remaining routers are skipped.
fail: The router determines that the address should fail, and queues it for the generation of a bounce message. There is no further processing of the original address unless unseen is set on the router.
defer: The router cannot handle the address at the present time. (A database may be offline, or a DNS lookup may have timed out.) No further processing of the address happens in this delivery attempt. It is tried again next time the message is considered for delivery.
error: There is some error in the router (for example, a syntax error in its configuration). The action is as for defer.
If an address reaches the end of the routers without having been accepted by any of them, it is bounced as unrouteable.
The pre-conditions that are tested for each router are listed below, in the order in which they are tested. The individual configuration options are described in more detail in chapter 14.
The local_part_prefix and local_part_suffix options can specify that the local parts handled by the router must have certain prefixes or suffixes. These conditions are tested first. When they are met, the prefix and/or suffix is removed from the local part before further processing, including the evaluation of any other conditions.
Certain routers can be designated for use only when verifying an address, as opposed to routing it for delivery, and similarly, certain routers can be designated for use only when not verifying. (See the verify_only and verify options).
Certain routers can be explicitly skipped when running the routers to check an address given in the SMTP EXPN command (see the expn option).
If the domains option is set, the domain of the address must be in the set of domains that it defines.
If the local_parts option is set, the local part of the address must be in the set of local parts that it defines. If local_part_prefix and/or local_part_suffix is in use, the prefix and/or suffix is removed from the local part before this check. If you want to do pre-condition tests on local parts that include affixes, you can do so by using a condition option (see below) that uses the variables $local_part, $local_part_prefix, and $local_part_suffix as necessary.
If the check_local_user option is set, the local part must be the name of an account on the local host.
If the senders option is set, the envelope sender address must be in the set of addresses that it defines.
If the require_files option is set, the existence or non-existence of specified files is tested.
If the condition option is set, it is evaluated and tested. This option uses an expanded string to allow you to set up your own custom pre-conditions. Expanded strings are described in chapter 11.
Note that require_files comes near the end of the list, so you cannot use it to check for the existence of a file in which to lookup up a domain, local part, or sender. However, as these options are all expanded, you can use the exists expansion condition to make such tests within each condition. The require_files option is intended for checking files that the router may be going to use internally, or which are needed by a specific transport (for example, .procmailrc).
When a message is to be delivered, the sequence of events is roughly as follows:
If a system-wide filter file is specified, the message is passed to it. The filter may add recipients to the message, replace the recipients, discard the message, cause a new message to be generated, or cause the message delivery to fail. The format of the filter file is the same as for user filter files, described in the separate document entitled Exim's interface to mail filtering. Some additional features are available in system filters - see chapter 39 for details. Note that a message is passed to the system filter only once per delivery attempt, however many recipients it has. However, if there are several delivery attempts because one or more addresses could not be immediately delivered, the system filter is run each time. The filter condition first_delivery can be used to detect this.
Each recipient address is offered to each configured router in turn, subject to its pre-conditions, until one is able to handle it. If no router can handle the address, that is, if they all decline, the address is failed. Because routers can be targeted at particular domains, several locally handled domains can be processed entirely independently of each other.
A router that accepts an address may set up a local or a remote transport for it. However, the transport is not run at this time. Instead, the address is placed on a queue for the particular transport, to be run later. Alternatively, the router may generate one or more new addresses (typically from alias, forward, or filter files). New addresses are fed back into this process from the top, but in order to avoid loops, a router ignores any address which has an identically-named ancestor that was processed by itself.
When all the routing has been done, addresses that have been successfully handled are passed to their assigned transports. When local transports are doing real local deliveries, they handle only one address at a time, but if a local transport is being used as a pseudo-remote transport (for example, to collect batched SMTP messages for transmission by some other means) multiple addresses can be handled. Remote transports can always handle more than one address at a time, but can be configured not to do so, or to restrict multiple addresses to the same domain.
Each local delivery runs in a separate process under a non-privileged uid, and they are run in sequence. Remote deliveries also run in separate processes, normally under a uid that is private to Exim (``the Exim user''), but in this case, several remote deliveries can be run in parallel. The maximum number of simultaneous remote deliveries for any one message is set by the remote_max_parallel option.
When it encounters a local delivery during a queue run, Exim checks its retry database to see if there has been a previous temporary delivery failure for the address before running the local transport. If there was a previous failure, Exim does not attempt a new delivery until the retry time for the address is reached. However, this happens only for delivery attempts that are part of a queue run. Local deliveries are always attempted when delivery immediately follows message reception, even if retry times are set for them. This makes for better behaviour if one particular message is causing problems (for example, causing quota overflow, or provoking an error in a filter file).
Remote transports do their own retry handling, since an address may be deliverable to one of a number of hosts, each of which may have a different retry time. If there have been previous temporary failures and no host has reached its retry time, no delivery is attempted, whether in a queue run or not. See chapter 31 for details of retry strategies.
If there were any permanent errors, a bounce message is returned to an appropriate address (the sender in the common case), with details of the error for each failing address. Exim can be configured to send copies of bounce messages to other addresses.
If one or more addresses suffered a temporary failure, the message is left on the queue, to be tried again later. Delivery is said to be deferred.
When all the recipient addresses have either been delivered or bounced, handling of the message is complete. The spool files and message log are deleted, though the message log can optionally be preserved if required.
Exim's mechanism for retrying messages that fail to get delivered at the first attempt is the queue-runner process. You must either run an Exim daemon that uses the -q option with a time interval to start queue-runners at regular intervals, or use some other means (such as cron) to start them. If you do not arrange for queue-runners to be run, messages that fail at the first attempt will remain on your queue for ever. A queue-runner process works it way through the queue, one message at a time, trying each delivery that has passed its retry time.
Exim uses a set of configured rules to determine when next to retry the failing address (see chapter 31). These rules also specify when Exim should give up trying to deliver to the address, at which point it generates a bounce message. If no retry rules are set for a particular host, address, and error combination, no retries are attempted, and temporary errors are treated as permanent.
There are many reasons why a message may not be immediately deliverable to a particular address. Failure to connect to a remote machine (because it, or the connection to it, is down) is one of the most common. Temporary failures may be detected during routing as well as during the transport stage of delivery. Local deliveries may be delayed if NFS files are unavailable, or if a mailbox is on a file system where the user is over quota. Exim can be configured to impose its own quotas on local mailboxes; where system quotas are set they will also apply.
If a host is unreachable for a period of time, a number of messages may be waiting for it by the time it recovers, and sending them in a single SMTP connection is clearly beneficial. Whenever a delivery to a remote host is deferred, Exim makes a note in its hints database, and whenever a successful SMTP delivery has happened, it looks to see if any other messages are waiting for the same host. If any are found, they are sent over the same SMTP connection, subject to a configuration limit as to the maximum number in any one connection.
When a message cannot be delivered to some or all of its intended recipients, a bounce message is generated. Temporary delivery failures turn into permanent errors when their timeout expires. All the addresses that fail in a given delivery attempt are listed in a single message. If the original message has many recipients, it is possible for some addresses to fail in one delivery attempt and others to fail subsequently, giving rise to more than one bounce message. The wording of bounce messages can be customized by the administrator. See chapter 40 for details.
Bounce messages contain an X-Failed-Recipients: header line which lists the failed addresses, for the benefit of programs that try to analyse such messages automatically.
A bounce message is normally sent to the sender of the original message, as obtained from the message's envelope. For incoming SMTP messages, this is the address given in the MAIL command. However, when an address is expanded via a forward or alias file, an alternative address can be specified for delivery failures of the generated addresses. For a mailing list expansion (see section 41.2) it is common to direct bounce messages to the manager of the list.
If a bounce message (either locally generated or received from a remote host) itself suffers a permanent delivery failure, the message is left on the queue, but it is frozen, awaiting the attention of an administrator. There are options which can be used to make Exim discard such failed messages, or to keep them for only a short time (see timeout_frozen_after and ignore_bounce_errors_after).