Previous   Next   Contents       (Exim 4.40 Specification)

44. Message processing

Exim performs various transformations on the sender and recipient addresses of all messages that it handles, and also on the messages' header lines. Some of these are optional and configurable, while others always take place. All of this processing, except rewriting as a result of routing, and the addition or removal of header lines while delivering, happens when a message is received, before it is placed on Exim's queue.

Some of the automatic processing takes place by default only for “locally-originated” messages. This adjective is used to describe messages that are not received over TCP/IP, but instead are passed to an Exim process on its standard input. This includes the interactive “local SMTP” case that is set up by the -bs command line option. Note: messages received over TCP/IP on the loopback interface ( or ::1) are not considered to be locally-originated. Exim does not treat the loopback interface specially in any way.

Processing that happens automatically for locally-originated messages can also be requested for other messages. This is done by obeying the modifier

  control = submission

in one of the ACLs that are run for an incoming message (see section 38.11). This makes Exim treat the message as a local submission, and is normally used when the source of the message is known to be an MUA running on a client host (as opposed to an MTA). In the descriptions below, the term “submission mode” is used to describe this state.

When a From: or Sender: header is generated in submission mode, the value of qualify_domain is used by default. However, it is possible to specify another domain by a setting such as

  control = submission/domain=some.other.domain

44.1. Line endings

RFC 2821 specifies that CRLF (two characters: carriage-return, followed by linefeed) is the line ending for messages transmitted over the Internet using SMTP over TCP/IP. However, within individual operating systems, different conventions are used. For example, Unix-like systems use just LF, but others use CRLF or just CR.

Exim was designed for Unix-like systems, and internally, it stores messages using the system's convention of a single LF as a line terminator. When receiving a message, all line endings are translated to this standard format. Originally, it was thought that programs that passed messages directly to an MTA within an operating system would use that system's convention. Experience has shown that this is not the case; for example, there are Unix applications that use CRLF in this circumstance. For this reason, and for compatibility with other MTAs, the way Exim handles line endings for all messages is now as follows:

44.2. Unqualified addresses

By default, Exim expects every address it receives from an external host to be fully qualified. Unqualified addresses cause negative responses to SMTP commands. However, because SMTP is used as a means of transporting messages from MUAs running on personal workstations, there is sometimes a requirement to accept unqualified addresses from specific hosts or IP networks.

Exim has two options that separately control which hosts may send unqualified sender or receipient addresses in SMTP commands, namely sender_unqualified_hosts and recipient_unqualified_hosts. In both cases, if an unqualified address is accepted, it is qualified by adding the value of qualify_domain or qualify_recipient, as appropriate.

44.3. The UUCP From line

Messages that have come from UUCP (and some other applications) often begin with a line containing the envelope sender and a timestamp, following the word “From”. Examples of two common formats are:

  From a.oakley@berlin.mus Fri Jan  5 12:35 GMT 1996
  From f.butler@berlin.mus Fri, 7 Jan 97 14:00:00 GMT

This line precedes the RFC 2822 header lines. For compatibility with Sendmail, Exim recognizes such lines at the start of messages that are submitted to it via the command line (that is, on the standard input). It does not recognize such lines in incoming SMTP messages, unless the sending host matches ignore_fromline_hosts or the -bs option was used for a local message and ignore_fromline_local is set. The recognition is controlled by a regular expression that is defined by the uucp_from_pattern option, whose default value matches the two common cases shown above and puts the address that follows “From” into $1.

When the caller of Exim for a non-SMTP message that contains a “From” line is a trusted user, the message's sender address is constructed by expanding the contents of uucp_sender_address, whose default value is “$1”. This is then parsed as an RFC 2822 address. If there is no domain, the local part is qualified with qualify_domain unless it is the empty string. However, if the command line -f option is used, it overrides the “From” line.

If the caller of Exim is not trusted, the “From” line is recognized, but the sender address is not changed. This is also the case for incoming SMTP messages that are permitted to contain “From” lines.

Only one “From” line is recognized. If there is more than one, the second is treated as a data line that starts the body of the message, as it is not valid as a header line. This also happens if a “From” line is present in an incoming SMTP message from a source that is not permitted to send them.

44.4. Resent- header lines

RFC 2822 makes provision for sets of header lines starting with the string Resent- to be added to a message when it is resent by the original recipient to somebody else. These headers are Resent-Date:, Resent-From:, Resent-Sender:, Resent-To:, Resent-Cc:, Resent-Bcc: and Resent-Message-ID:. The RFC says:

Resent fields are strictly informational. They MUST NOT be used in the normal processing of replies or other such automatic actions on messages.

This leaves things a bit vague as far as other processing actions such as address rewriting are concerned. Exim treats Resent- header lines as follows:

44.5. The Auto-Submitted: header line

Whenever Exim generates a bounce or a delay warning message, it includes the header line

  Auto-Submitted: auto-generated

44.6. The Bcc: header line

If Exim is called with the -t option, to take recipient addresses from a message's header, it removes any Bcc: header line that may exist (after extracting its addresses). If -t is not present on the command line, any existing Bcc: is not removed.

44.7. The Date: header line

If a locally-generated or submission-mode message has no Date: header line, Exim adds one, using the current date and time.

44.8. The Delivery-date: header line

Delivery-date: header lines are not part of the standard RFC 2822 header set. Exim can be configured to add them to the final delivery of messages. (See the generic delivery_date_add transport option.) They should not be present in messages in transit. If the delivery_date_remove configuration option is set (the default), Exim removes Delivery-date: header lines from incoming messages.

44.9. The Envelope-to: header line

Envelope-to: header lines are not part of the standard RFC 2822 header set. Exim can be configured to add them to the final delivery of messages. (See the generic envelope_to_add transport option.) They should not be present in messages in transit. If the envelope_to_remove configuration option is set (the default), Exim removes Envelope-to: header lines from incoming messages.

44.10. The From: header line

If a submission-mode message does not contain a From: header line, Exim adds one if either of the following conditions is true:

  1. The envelope sender address is not empty (that is, this is not a bounce message); the added header line copies the envelope sender address.

  2. The SMTP session is authenticated and $authenticated_id is not empty; the added header's local part is $authenticated_id and the domain is the domain specified on the submission control, or $qualify_domain if that is not set.

A non-empty envelope sender takes precedence.

If a locally-generated incoming message does not contain a From: header line, Exim adds one containing the sender's address. The calling user's login name and full name are used to construct the address, as described in section 44.16. They are obtained from the password data by calling getpwuid() (but see the unknown_login configuration option). The address is qualified with qualify_domain.

For compatibility with Sendmail, if an incoming, non-SMTP message has a From: header line containing just the unqualified login name of the calling user, this is replaced by an address containing the user's login name and full name as described in section 44.16.

44.11. The Message-ID: header line

If a locally-generated or submission-mode incoming message does not contain a Message-ID: or Resent-Message-ID: header line, Exim adds one to the message. If there are any Resent-: headers in the message, it creates Resent-Message-ID:. The id is constructed from Exim's internal message id, preceded by the letter E to ensure it starts with a letter, and followed by @ and the primary host name. Additional information can be included in this header line by setting the message_id_header_text and/or message_id_header_domain options.

44.12. The Received: header line

A Received: header line is added at the start of every message. The contents are defined by the received_header_text configuration option, and Exim automatically adds a semicolon and a timestamp to the configured string.

The Received: header is generated as soon as the message's header lines have been received. At this stage, the timestamp in the Received: header line is the time that the message started to be received. This is the value that is seen by the DATA ACL and by the local_scan() function.

Once a message is accepted, the timestamp in the Received: header line is changed to the time of acceptance, which is (apart from a small delay while the -H spool file is written) the earliest time at which delivery could start.

44.13. The Return-path: header line

Return-path: header lines are defined as something an MTA may insert when it does the final delivery of messages. (See the generic return_path_add transport option.) Therefore, they should not be present in messages in transit. If the return_path_remove configuration option is set (the default), Exim removes Return-path: header lines from incoming messages.

44.14. The Sender: header line

For a locally-originated message from an untrusted user, Exim may remove an existing Sender: header line, and it may add a new one. You can modify these actions by setting local_sender_retain true or local_from_check false.

When a local message is received from an untrusted user and local_from_check is true (the default), a check is made to see if the address given in the From: header line is the correct (local) sender of the message. The address that is expected has the login name as the local part and the value of qualify_domain as the domain. Prefixes and suffixes for the local part can be permitted by setting local_from_prefix and local_from_suffix appropriately. If From: does not contain the correct sender, a Sender: line is added to the message.

If you set local_from_check false, this checking does not occur. However, the removal of an existing Sender: line still happens, unless you also set local_sender_retain to be true. It is not possible to set both of these options true at the same time.

By default, no processing of Sender: header lines is done for messages received by TCP/IP or for messages submitted by trusted users. However, when a message is received over TCP/IP in submission mode, Sender: header lines are always removed. If the SMTP session is authenticated, and $authenticated_id is not empty, a sender address is created with $authenticated_id as the local part and either the domain specified in the submission control or, if that is not specified, $qualify_domain as the domain. This is compared with the address in the From: header line. If they are different, a Sender: header line is added. Prefixes and suffixes for the local part in From: can be permitted by setting local_from_prefix and local_from_suffix appropriately.

44.15. Adding and removing header lines

When a message is delivered, the addition and removal of header lines can be specified on any of the routers and transports, and also in the system filter. Changes specified in the system filter affect all deliveries of a message.

Header changes specified on a router affect all addresses handled by that router, and also any new addresses it generates. If an address passes through several routers, the changes are cumulative. When a message is processed by a transport, the message's original set of header lines is output, except for those named in any headers_remove options that the address has encountered as it was processed, and any in the transport's own headers_remove option. Then the new header lines from headers_add options are output.

44.16. Constructed addresses

When Exim constructs a sender address for a locally-generated message, it uses the form

  <user name> <$<login>>@<qualify_domain>>

For example:

  Zaphod Beeblebrox <zaphod@end.univ.example>

The user name is obtained from the -F command line option if set, or otherwise by looking up the calling user by getpwuid() and extracting the “gecos” field from the password entry. If the “gecos” field contains an ampersand character, this is replaced by the login name with the first letter upper cased, as is conventional in a number of operating systems. See the gecos_name option for a way to tailor the handling of the “gecos” field. The unknown_username option can be used to specify user names in cases when there is no password file entry.

In all cases, the user name is made to conform to RFC 2822 by quoting all or parts of it if necessary. In addition, if it contains any non-printing characters, it is encoded as described in RFC 2047, which defines a way of including non-ASCII characters in header lines. The value of the headers_charset option specifies the name of the encoding that is used (the characters are assumed to be in this encoding). The setting of print_topbitchars controls whether characters with the top bit set (that is, with codes greater than 127) count as printing characters or not.

44.17. Case of local parts

RFC 2822 states that the case of letters in the local parts of addresses cannot be assumed to be non-significant. Exim preserves the case of local parts of addresses, but by default it uses a lower-cased form when it is routing, because on most Unix systems, usernames are in lower case and case-insensitive routing is required. However, any particular router can be made to use the original case for local parts by setting the caseful_local_part generic router option.

If you must have mixed-case user names on your system, the best way to proceed, assuming you want case-independent handling of incoming email, is to set up your first router to convert incoming local parts in your domains to the correct case by means of a file lookup. For example:

    driver = redirect
    domains = +local_domains
    data = ${lookup{$local_part}cdb\

For this router, the local part is forced to lower case by the default action (caseful_local_part is not set). The lower-cased local part is used to look up a new local part in the correct case. If you then set caseful_local_part on any subsequent routers which process your domains, they will operate on local parts with the correct case in a case-sensitive manner.

44.18. Dots in local parts

RFC 2822 forbids empty components in local parts. That is, an unquoted local part may not begin or end with a dot, nor have two consecutive dots in the middle. However, it seems that many MTAs do not enforce this, so Exim permits empty components for compatibility.

44.19. Rewriting addresses

Rewriting of sender and recipient addresses, and addresses in headers, can happen automatically, or as the result of configuration options, as described in chapter 31. The headers that may be affected by this are Bcc:, Cc:, From:, Reply-To:, Sender:, and To:.

Automatic rewriting includes qualification, as mentioned above. The other case in which it can happen is when an incomplete non-local domain is given. The routing process may cause this to be expanded into the full domain name. For example, a header such as

  To: hare@teaparty

might get rewritten as

  To: hare@teaparty.wonderland.fict.example

Rewriting as a result of routing is the one kind of message processing that does not happen at input time, as it cannot be done until the address has been routed.

Strictly, one should not do any deliveries of a message until all its addresses have been routed, in case any of the headers get changed as a result of routing. However, doing this in practice would hold up many deliveries for unreasonable amounts of time, just because one address could not immediately be routed. Exim therefore does not delay other deliveries when routing of one or more addresses is deferred.

Previous  Next  Contents       (Exim 4.40 Specification)