Previous   Next   Contents       (Exim 4.00 Specification)

3. How Exim receives and delivers mail

3.1. Overall philosophy

Exim is designed to work efficiently on systems that are permanently connected to the Internet and are handling a general mix of mail. In such circumstances, most messages can be delivered immediately. Consequently, Exim does not maintain independent queues of messages for specific domains or hosts, though it does try to send several messages in a single SMTP connection after a host has been down, and it also maintains per-host retry information.

3.2. Policy control

Policy controls are now an important feature of MTAs that are connected to the Internet. Perhaps their most important job is to stop MTAs being abused as ``open relays'' by misguided individuals who send out vast amounts of unsolicited junk, and want to disguise its source. Exim provides flexible facilities for specifying policy controls on incoming mail:

3.3. Message identification

Every message handled by Exim is given a message id which is sixteen characters long. It is divided into three parts, separated by hyphens, for example 16VDhn-0001bo-00. Each part is a sequence of letters and digits, representing a number in base 62:

3.4. Receiving mail

The only way Exim can receive mail from a remote host is using SMTP over TCP/IP, in which case the sender and recipient addresses are tranferred using SMTP commands. However, from a locally running process (such as a user's MUA), there are several possibilities:

In the three cases that do not involve TCP/IP, the sender address is constructed from the login name of the user that called Exim and a default qualification domain (which can be set by the qualify_domain configuration option). For local or batch SMTP, a sender address that is passed using the SMTP MAIL command is ignored. However, the system administrator may allow certain users to specify a different sender address unconditionally, or all users to specify certain forms of different sender address. The -f option or the SMTP MAIL command is used to specify these different addresses. See section 5.2 for details of trusted users, and the untrusted_set_sender option for a way of allowing untrusted users to change sender addresses.

Messages received by either of the non-interactive mechanisms are always accepted, provided they are syntactically valid. However, when a message is being received using the SMTP protocol (either over TCP/IP, or interacting with a local process), policy controls can be applied at the time of reception, and the message can be rejected if it does not accord with local policy.

Exim can be configured not to start a delivery process when a message is received; this can be unconditional, or depend on the number of incoming SMTP connections or the system load. In these situations, new messages wait on the queue until a queue-runner process picks them up. However, in standard configurations under normal conditions, delivery is started as soon as a message is received.

3.5. Handling an incoming message

When Exim accepts a message, it writes two files in its spool directory. The first contains the envelope information, the current status of the message, and the headers, and the second contains the body of the message. The names of the two spool files consist of the message id, followed by -H for the file containing the envelope and headers, and -D for the data file.

By default all these message files are held in a single directory called input inside the general Exim spool directory. Some operating systems do not perform very well if the number of files in a directory gets very large; to improve performance in such cases, the split_spool_directory option can be used. This causes Exim to split up the input files into 62 sub-directories whose names are single letters or digits.

The envelope information consists of the address of the message's sender and the address(es) of the recipient(s). This information is entirely separate from any addresses contained in the headers. The status of the message includes a list of recipients who have already received the message. The format of the first spool file is described in chapter 48.

Address rewriting that is specified in the rewrite section of the configuration (see chapter 30) is done once and for all on incoming addresses, both in the header and the envelope, at the time the message is accepted. If during the course of delivery additional addresses are generated (for example, via aliasing), these new addresses get rewritten as soon as they are generated. At the time a message is actually delivered (transported) further rewriting can take place; because this is a transport option, it can be different for different forms of delivery. It is also possible to specify the addition or removal of certain header lines at the time the message is delivered (see chapters 14 and 23).

3.6. Life of a message

A message remains in the spool directory until it is completely delivered to its recipients or to an error address, or until it is deleted by an administrator or by the user who originally created it. In cases when delivery cannot proceed - for example, when a message can neither be delivered to its recipients nor returned to its sender, the message is marked ``frozen'' on the spool, and no more deliveries are attempted.

An administrator can ``thaw'' such messages when the problem has been corrected, and can also freeze individual messages by hand if necessary. In addition, an administrator can force a delivery error, causing a bounce message to be sent.

There is an option called auto_thaw, which can be used to cause Exim to retry frozen messages after a certain time. When this is set, no message will remain on the queue for ever, because the delivery timeout will eventually be reached. Delivery failure reports that reach this timeout are discarded. There is also an option called timeout_frozen_after, which simply discards frozen messages after a certain time.

While Exim is working on a message, it writes information about each delivery attempt to the main log file. The includes successful, unsuccessful, and delayed deliveries for each recipient (see chapter 44). The log lines are also written to a separate message log file for each message. These logs are solely for the benefit of the administrator, and are normally deleted along with the spool files when processing of a message is complete.

All the information Exim itself needs to set up a delivery is kept in the first spool file with the headers. When a successful delivery occurs, the address is immediately written at the end of a journal file, whose name is the message id followed by -J. At the end of a delivery run, if there are some addresses left to be tried again later, the first spool file (the -H file) is updated to indicate which these are, and the journal file is then deleted. Updating the spool file is done by writing a new file and renaming it, to minimize the possibility of data loss.

Should the system or the program crash after a successful delivery but before the spool file has been updated, the journal is left lying around. The next time Exim attempts to deliver the message, it reads the journal file and updates the spool file before proceeding. This minimizes the chances of double deliveries caused by crashes.

3.7. Processing an address for delivery

The main delivery processing elements of Exim are called routers and transports, and collectively these are known as drivers. Code for a number of them is provided in the source distribution, and compile-time options specify which ones are included in the binary. Run time options specify which ones are actually used for delivering messages.

A router is a driver that operates on an address, either determining how its delivery should happen, by routing it to a specific transport, or converting the address into one or more new addresses (for example, via an alias file). A router may also explicitly choose to fail an address, causing it to be bounced.

A transport is a driver that transmits a copy of the message from Exim's spool to some destination. There are two kinds of transport: for a local transport, the destination is a file or a pipe on the local host, whereas for a remote transport the destination is some other host. A message is passed to a specific transport as a result of successful routing. If a message has several recipients, it may be passed to a number of different transports.

An address is processed by passing it to each configured router in turn, subject to certain pre-conditions, until one accepts it or specifies that it should be bounced. We'll describe this process in more detail shortly. As a simple example, the diagram below illustrates how each recipient address in a message is processed in a small configuration of three routers that are configured in various ways.

Routing an address

To make this a more concrete example, we'll describe it in terms of some actual routers, but remember, this is only an example. You can configure Exim's routers in many different ways, and there may be any number of routers in a configuration.

The first router in a configuration is often one that handles addresses in domains that are not recognized specially by the local host. These are typically addresses for arbitrary domains on the Internet. A pre-condition is set up which looks for the special domains known to the host (for example, its own domain name), and the router is run for addresses that do not match. Typically, this is a router that looks up domains in the DNS in order to find the hosts to which this address routes. If it succeeds, the address is queued for a suitable SMTP transport; it it does not succeed, the router is configured to fail the address.

In the example, the second and third routers can only be run for domains that are special to the local host, for which the pre-condition for the first router is not met. The second router does redirection - also known as aliasing and forwarding. When it generates one or more new addresses from the original, each of them is routed independently from the start. Otherwise, the router may cause an address to fail, or it may simply decline to handle the address, in which case it is passed on to the next router.

The final router in many configurations is one that checks to see if the address belongs to a local mailbox. The pre-condition may involve a check to see if the local part is the name of a login account, or it may look up the local part in a file or a database. If its pre-condition is not met, or if the router declines, we have reached the end of the routers. When this happens, the address is bounced.

3.8. Running an individual router

As explained in the example above, a number of conditions are checked before running a router. If any of these pre-conditions are not met, the router is skipped, and the address is passed to the next router. Sometimes you want to fail a delivery when some conditions are met but others are not, instead of passing the address on for further routing. You can do this by having a second router that explicitly fails the delivery when the relevant conditions are met.

When all the conditions on a router are met, the router is run. What happens next depends on the outcome, which is one of the following:

If an address reaches the end of the routers without having been accepted by any of them, it is bounced as unrouteable.

3.9. Router pre-conditions

The pre-conditions that are tested for each router are listed below, in the order in which they are tested. The individual configuration options are described in more detail in chapter 14.

Note that require_files comes near the end of the list, so you cannot use it to check for the existence of a file in which to lookup up a domain, local part, or sender. However, as these options are all expanded, you can use the exists expansion condition to make such tests within each condition. The require_files option is intended for checking files that the router may be going to use internally, or which are needed by a specific transport (for example, .procmailrc).

3.10. Delivery in detail

When a message is to be delivered, the sequence of events is roughly as follows:

3.11. Retry mechanism

Exim's mechanism for retrying messages that fail to get delivered at the first attempt is the queue-runner process. You must either run an Exim daemon that uses the -q option with a time interval to start queue-runners at regular intervals, or use some other means (such as cron) to start them. If you do not arrange for queue-runners to be run, messages that fail at the first attempt will remain on your queue for ever. A queue-runner process works it way through the queue, one message at a time, trying each delivery that has passed its retry time.

Exim uses a set of configured rules to determine when next to retry the failing address (see chapter 31). These rules also specify when Exim should give up trying to deliver to the address, at which point it generates a bounce message. If no retry rules are set for a particular host, address, and error combination, no retries are attempted, and temporary errors are treated as permanent.

3.12. Temporary delivery failure

There are many reasons why a message may not be immediately deliverable to a particular address. Failure to connect to a remote machine (because it, or the connection to it, is down) is one of the most common. Temporary failures may be detected during routing as well as during the transport stage of delivery. Local deliveries may be delayed if NFS files are unavailable, or if a mailbox is on a file system where the user is over quota. Exim can be configured to impose its own quotas on local mailboxes; where system quotas are set they will also apply.

If a host is unreachable for a period of time, a number of messages may be waiting for it by the time it recovers, and sending them in a single SMTP connection is clearly beneficial. Whenever a delivery to a remote host is deferred, Exim makes a note in its hints database, and whenever a successful SMTP delivery has happened, it looks to see if any other messages are waiting for the same host. If any are found, they are sent over the same SMTP connection, subject to a configuration limit as to the maximum number in any one connection.

3.13. Permanent delivery failure

When a message cannot be delivered to some or all of its intended recipients, a bounce message is generated. Temporary delivery failures turn into permanent errors when their timeout expires. All the addresses that fail in a given delivery attempt are listed in a single message. If the original message has many recipients, it is possible for some addresses to fail in one delivery attempt and others to fail subsequently, giving rise to more than one bounce message. The wording of bounce messages can be customized by the administrator. See chapter 40 for details.

Bounce messages contain an X-Failed-Recipients: header line which lists the failed addresses, for the benefit of programs that try to analyse such messages automatically.

A bounce message is normally sent to the sender of the original message, as obtained from the message's envelope. For incoming SMTP messages, this is the address given in the MAIL command. However, when an address is expanded via a forward or alias file, an alternative address can be specified for delivery failures of the generated addresses. For a mailing list expansion (see section 41.2) it is common to direct bounce messages to the manager of the list.

3.14. Failures to deliver bounce messages

If a bounce message (either locally generated or received from a remote host) itself suffers a permanent delivery failure, the message is left on the queue, but it is frozen, awaiting the attention of an administrator. There are options which can be used to make Exim discard such failed messages, or to keep them for only a short time (see timeout_frozen_after and ignore_bounce_errors_after).


Previous  Next  Contents       (Exim 4.00 Specification)