Sendmail Lecture v1.0 Index page

Sendmail v8.7+

What is Sendmail ?
Where to get Sendmail.
How Sendmail processes mail.
Autopsy of the Rulesets.
The emails path through the rulesets.
Mailer Definitions.
m4 macros.
A walk through of a typical m4 macro file.
Masquerading and Sendmail for the SOHO.
New v8.10 features.
Q'n'A.

What is Sendmail ?

Sendmail is a mail transport agent, MTA

	MTA	Mail Transport Agent
		Exim, Sendmail
	MUA	Mail Users Agent
		Pine, Netscape, Outlook, Emacs, etc..
	MDA	Mail Delivery Agent
		POP, Imap, Procmail, uux ,etc...

Sendmail only transports email between systems letting the systems themselves sort out final dilervery.

Sendmail must be told how to communiate with the MDAs

Very popular as an MTA.

Well supported, comes as standard on most Linux distributions; Redhat, SuSE etc.

Communicate with SMTP, ESTMP mainly over TVP/IP networks, although it can use other protocols

How Sendmail processes mail

Sendmail separates the email message into 3 parts

Envelope
Header
Body

The envelope is used in the SMTP comunications e.g.

To: wibble@morpheus.ednet.co.uk To: faye@scotweb.ltd.uk

Mar 26 15:52:21 morpheus sendmail[22829]: PAA22828: to=<wibble@morpheus.ednet.co.uk>, ctladdr=<wibble@morpheux.org> (507/100), delay=00:00:00, xdelay=00:00:00, mailer=local, stat=Sent

Mar 26 15:54:25 morpheus sendmail[22841]: PAA22840: to=<faye@scotweb.ltd.uk>, ctladdr=<wibble@morpheux.org>(507/100), delay=00:00:30, xdelay=00:00:30, mailer=relay, relay=mail.ednet.co.uk [195.89.132.6], stat=Deferred: Network is unreachable

The header is contained in the email itself often generated by the MUA and added to by sendmail. It is used by sendmail internaly to decide how is should deal with each email, and provides sendmail with a place to put accounting information.

`Return-Path:`
`Received:`	`from morpheux.org (IDENT:faye@eros.morpheus.ednet.co.uk [192.168.42.2]) by morpheus.morpheus.ednet.co.uk (8.9.3/8.9.3) with ESMTP id HAA02277; Thu, 30 Mar 2000 07:04:58 +0100`
`Sender:`	`faye@morpheus.ednet.co.uk`
`Message-ID:`	`<38E2EE6F.3D12A4FC@morpheux.org>`
`Date:`	`Thu, 30 Mar 2000 07:04:31 +0100`
`From:`	`Faye Gibbins`
`Organization:`	`Morpheus Private Networks`
`X-Mailer:`	`Mozilla 4.7 [en] (X11; I; Linux 2.2.14 i686)`
`X-Accept-Language:`	`en`
`MIME-Version:`	`1.0`
`To:`	`Faye Gibbins , wibble@morpheux.org`
`Subject:`	`test`
`Content-Type:`	`text/plain; charset=us-ascii`
`Content-Transfer-Encoding:`	`7bit`
`Status:`
`X-Mozilla-Status:`	`8001`
`X-Mozilla-Status2:`	`00000000`
`X-UIDL:`	`3877292000002827`

Header names and syntax are defined in RFC822 and RFC1123.
Custom headers can start with x- .
The form of the header is defined in the sendmail.cf file.
Internally sendmail stores the header separately from the body.
The header ends with a blank line, after the line is considered the body.

Sendmail's configuration file can be complex because of the diverse nature of email address, vis.

bob!jane!jon
host.domain!user
user%second@first
node::user	DECnet
abe,george,andrew	old style
abe george andrew	new style
/RN=MS.USER/O=CORP/ PRMD=CORP/ADMD=TELE/C=US	X.400
/RN=MS.USER/O=CORP/ PRMD=CORP/ADMD=TELE/C=US@X.400.gateway.here	X.400

Source - Sendmail, O'Reilly

Autopsy of the Rulesets

Macros

To cope with the range of email addresses Sendmail uses Macros, Class macros, Database macros,Subrutines and Regular Expresions to manipulate the email addresses into a single standard form which sendmail can then use.

Macros, or string macros, hold a single string value, some macros are predefined by sendmail, and some are set in the configuration file manually. This is simular to PERLS stings $! or $. and my $string = "wibble" e.g.

$b	The current date in RFC822 format
${client_addr}	The connecting host's canonical name (V8.8 and above)
Source - Sendmail, O'Reilly, C31

To define a macro on the command line use the syntax-

-MXtext

To define a macro on the configuration file use the syntax-

DXtext or
D{somemacrco}text

If you want a macro name to have more than one character in it, then it must be surrounded by curly braces. The 'D' tells sendmail that we want to define a macro. The PERL eqiv would be-
my $somemacro = "text";

Macros can be expaned using this syntax-

$macro_name

This will expand the macro in the configuration file

A Macro is expanded when Sendmail is started up, you can't define them on the fly.
Macros are expanded recursively, i.e.

D{some_macro}${macro_two} D{macro_two}wibble

Will be expanded thus;

${some_macro}

becomes

${macro_two}

Which then becomes

wibble

To test if a macro has been given a value , the if-then statement can be used -

$?{some_macro} wibble $| wobble $.

Which reads, if ($?) the macro {some_macro} is defined then return wibble else ($|) return wobble endif ($.).

The perl equiv would be

if(defined $some_macro){return "wibble"} else{return "wobble"}

Sendmail if-then statements can be nested just like the PERL equivalent. But there is no elsif equivalent.

To stop a macro being expanded prefix it with $&, ie

$&{some_macro}

Class Macros

Classes allow the use of arrays in the configuration file.

Class macros are set when Sendmail is started up, and are set by the syntax

C{some_array} hosts1.domain host2.domain ....

F{some_array} file

F{some_array} |program

Attempts to override classes result in appending of the list to the previous definition

The delimiter if defined by isspace(3), this closely relates to PERL's \s (with newlines).

There are two syntaxes used to test a value against the class macro.

Define a macro.

D{some_macro}tester

Define a class.

C{some_class}tester not_test other_value

then if ${some_macro} is passed to the workspace

$={some_class}

returns true, the value and

$~{some_class}

returns false.

This is some what analigous to the PERL's grep statement used in scalar context.

Database Macros

Database macros allows sendmail to look up and match over large volumes of data.

However they are more difficult to setup.

Unlike other Macros a change to the database is immediately available to sendmail.

To make a d.b. for sendmail use the makemap command.

makemap switches class file

Switches
-d Allow the database to conatin dulpicate keys
-f Preserve uppercase
-N Append null byte to all keys
-o Append to d.b
-r Replace(silently) duplicate keys

Source - Sendmail, O'Reilly

The class depends upon the syle of the database you want to create and what database formats have been compiled into sendmail. The command 'strings /usr/lib/sendmail | grep map_open' returns the db info for sendmail.

faye@morpheus faye]$ strings /usr/lib/sendmail | grep map_open

bt_map_open(%s, %s, %d) hash_map_open(%s, %s, %d) db_map_open: cannot pre-open database %s db_map_open(%s): file changed after pre-open db_map_open(%s): cannot fstat pre-opened file db_map_open: cannot lock %s db_map_open(%s): file changed after open nis_map_open(%s, %s, %d) nis_map_open: yp_match(@, %s, %s) => %s text_map_open(%s, %s, %d) text_map_open(%s, %s): delimiter = stab_map_open(%s, %s, %d) impl_map_open(%s, %s, %d) user_map_open(%s, %d) switch_map_open(%s, %s, %d)

Database macros are defined in the sendmail.cf file useing the 'K' command, i.e.

K my_database class switched file_or_map

Class	Version	Description
btree	8.1+	Berkeley's db form old database
bestmx	8.7+	Look up the best MX record for a host
dbm	8.1+	Really ndbm supplied with most verions of UNIX
dequote	8.1+	A pseudo map for removing quotation marks
hash	8.1+	Berkeley's db form od daabase
hesiod	8.7+	MIT network user authentication services
host	8.1+	Internal table to store and look up hostnames
implicit	8.1+	Search for an aliases database file
ldapx	8.8+	The Lightweight directory Access Protocol
netinfo	8.7+	NeXT Computers network information services
nis	8.1+	Sun's Network Inormation Services (NIS)
nisplus	8.7+	Sun's newer Network Information Services (NIS+)
null	8.7+	Provide a never found service
program	8.7+	Run an external program to look up the key
sequence	8.7+	Search a series of maps
stab	8.1+	Internally load aliases into the symbol table
switch	8.7+	Internal hook to auto-build sequences
text	8.7+	Look up in flat text files
userdb	8.7+	Look up in the User Database
user	8.7+	Look up local passwd information

Source - Sendmail, O'Reilly

Switch	Version	Description
-A	8.7+	Append values for duplicate keys
-a	8.1+	Append tag on successful match
-f	8.1+	Don't fold keys to lowercase
-k	8.7+	Specify column for key
-m	8.1+	Suppress replacement on match
-N	8.1+	Append a null byte to all keys
-O	8.2+	Adaptive versus never add a null
-o	8.1+	The database file is optional
-q	8.7+	Don't strip quotes from key
-s	8.7+	Space replacement character
-v	8.7+	Specify the value's column
-z	8.7+	Specify the column delimiter

Source - Sendmail, O'Reilly

Sendmail uses database searches with the $( $) or $[ $] syntax

$( name key $@$1 $@text2 $:default $)

This checks for the map to 'key' in the defined database 'name'. If it returns a value , that value is rewritten by '$@$1 $@text2'. If the key does not match in the database then the value 'default' is return. Database lookups can get very complex!

The $[ ]$ is special syntax used for resolving hosts and IP addresses.

More database definitions are added to the distribution every release, newer one include the regex, for matching regular expressions.

Rules

Rules use macros, strings and special $ operators to rewrite information in a simular way to PERL's 's///' operator.

They are defined by the R operator and have the syntax -

Rleft_hand_side right_hand_side comment

Separated by tabs.
The LHS and RHS are mandatory, the comment is optional.

The rules operate on the 'workspace' analagous to PERL's '$_'.

Any workspace that matches the pattern on the LHS is rewriten by RHS, it is then passed back to the LHS again.

file test.cf

D{some_macro}wibble R${some_macro} wobble

Has this result -

[faye@eros tmp]$ echo =S0 | /usr/lib/sendmail -bt -Ctest.cf

Warning: .cf file is out of date: sendmail 8.9.3 supports version 8, .cf file is version 0
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter

> Rwibble wobble

'String' macros, or text may appear on the LHS and RHS.

Class Macros appear on the LHS.

Database macros appear on the RHS.

Sendmail provides other operators to help with the pattern match on the LHS.

Operator	Description or Use
$*	Match zero or more tokens
$+	Match one or more tokens
$-	Match exacly one token
$@	Match exactly zero tokens
$=	Match any token in a class
$~	Match any singkle token not in a class

Source - Sendmail, O'Reilly

Other tools are provided on the RHS.

Operator	Description or Use
$digit	Copy by position
$:	Rewrite once (prefix)
$@	Rewrite and return (prefix)
$>set	Rewrite through another rule set
$#	Specify a delivery agent
$[ $]	Canonicalize hostname
$( $)	Database lookup

Source - Sendmail, O'Reilly

Rule Sets

All the rewrite rules in the sendmail.cf file are organised into rule sets.

Rule sets start with an S operator. End finish with the start of the next ruleset, e.g.

################################################ ### Ruleset 96 -- bottom half of ruleset 3 ### ################################################

S96

# handle special cases for local names

R$* < @ localhost > $*	$: $1 < @ $j . > $2	no domain at all
R$* < @ localhost . $m > $*	$: $1 < @ $j . > $2	local domain
R$* < @ localhost . UUCP > $*	$: $1 < @ $j . > $2	.UUCP domain
R$* < @ [ $+ ] > $*	$: $1 < @@ [ $2 ] > $3	mark [a.b.c.d]
R$* < @@ $=w > $*	$: $1 < @ $j . > $3	self-literal
R$* < @@ $+ > $*	$@ $1 < @ $2 > $3	canon IP addr

$j is the offical canonical hostname.
$=w ids the list of this machines name.
$m is the domain part of the local host'd fully qualified name.

The emails path through the rule sets

Sendmail has an internal order it uses to process the mail with with rule sets, it also uses serveral rule sets to determine if the mail even reaches the machine. These later sets can be use to stop spam.

Source - Sendmail, O'Reilly

Rule set 5 is used to further process the recipient address, if it has gone through the 'alias' process unchanged.

Sendmail also uses several other rule sets to authenticate the mail and the user.

check_mail
Validates the sender-envelope address given to the SMTP MAIL command.

check_rcpt
Validates the recipient-envelope address given to the SMTP RCPT command.

check_relay
Validates the host initiating the SMTP connection.

check_compat
Compares or contrasts each envelope sender and envelope recipient pairs of addresses just before delivery, and validate based on the result.

Source - Sendmail, O'Reilly

Mailer Definitions

Mailer Definitions tell sendmail how to use the local programs to deliver mail.

Each mailer definition also sets which rule set the sender and recipient email address get further processed by.

Mailer definitions are set with the 'M' operator.

Procmail	Mprocmail, P=/usr/bin/procmail, F=DFMSPhnu9, S=11/31, R=21/31, T=DNS/RFC822/X-Unix,A=procmail -Y -m $h $f $u
SMTP	Msmtp, P=[IPC], F=mDFMuX, S=11/31, R=21, E=\r\n, L=990, T=DNS/RFC822/SMTP, A=IPC $h
Local	Mlocal, P=/usr/bin/procmail, F=lsDFMAw5:/\|@qSPfhn9, S=10/30, R=20/40, T=DNS/RFC822/X-Unix, A=procmail -Y -a $h -d $u
Programs	Mprog, P=/usr/sbin/smrsh, F=lsDFMoqeu9, S=10/30, R=20/40, D=$z:/, T=X-Unix, A=sh -c $u
Relay	Mrelay, P=[IPC], F=mDFMuXa8, S=11/31, R=61, E=\r\n, L=2040, T=DNS/RFC822/SMTP, A=IPC $h

Delivery Agent Equates

Equate	Feild Name	Meaning
C=	Argv	Delivery agent's command-line arguments
C=	Charest	Default MINE character
D=	Directory	Paths to directories for execution
E=	EOL	End-of-line string
F=	Flags	Flags decribing a delivery agent's behavior
L=	Linelimit	Maximum line length
M=	Maximum	Maximum message size
N=	Niceness	How to nice(2) the agent
P=	Path	Path to the delivery program
R=	Recipient	Recipient rewriting rule set
S=	Sender	Sender rewriting rule set
T=	Type	Types for DSN diagnostics
U=	UID	Run agent as uid:gid

F= Flags

Flag	Version	Meaning
0	V8.8+	Turn off MX lookupsfor delivery agent
3	V8.7+	Extend quoted-printable to EBCDIC
5	V8.7+	Use rule set 5 after local aliasing
7	V8.6+	Strip the high bit when delivering
8	V8.7+	Suppress EightBitMode=m MINE encoding
9		Convert 7- to 8-bit if appropiate
:	V8.7+	Check for :include: files
\|	V8.7+	Check for \|porgram adress
/	V8.7+	Check for /file addresses
@	V8.7+	User can be User Database key
a	V8.6+	Run extended SMTP protocol
A	V8.7+	User can be LHS of an alias
b	V8.6+	Add a blank line after message
c	V8.6+	Exclude comment from $g in headers
C		Add @domain to recipient
d	V8.7+	Never enclose route address in <>
D		Need `Date:` in header
e		Mark expensive delivery agents
E		Change extra From into >From
f		Delivery agent adds -f to argv
F		Need `From:` in header
g	V8.6+	Suppress From:<>
h		Preserve uppercase in hostname
i	V8.7+	Do UDB sender rewrite on envelope
I		Send SMTP VERB to other site
j	V8.7+	Perform UDB rewrite of header recipient addresses
k	V8.7+	Don't check for loops in HELO command
l		Delivery agent is local (final) delivery
L		Specify SMTP line limits(Obsolete
m		Multiple recipents possible
M		Need `Message-ID:` in header
n		Don't ise UNIX-style From in header
o	V8.7+	Always run delivery agent as recipient
p	.	Process return path correctly (deprecated)
P		Need `Retrun-Path:` in header
q	V8.8+	250 verus 252 return for SMTP VRFY
r		Delivery agent adds -r to argv
R	V8	Use a reserved TCP port
s		Strip quotation marks
S		Assume specified uid and gud
u	V8.7	Reserve uppercase for username
U		Use Unix-style `From` line
w	V8.7+	Check for /etc/passwd entry
x		Need `Full-Name:` in header
X		Delivery agent needs RFC821 hidden dot

m4 Macros

m4 macros take all the hassle out of constructing sendmail.cf files.

A 50 line m4 file can create a customised 1200 line sendmail.cf file.

Most distributions of sendmail come with some predefined ma files. For example Redhat 6.2 ships with sendmail.rpm version 8.9.3 and contains these m4 examples-

chez.cs.mc

clientproto.mc

cs-hpux10.mc

cs-hpux9.mc

cs-osf1.mc

cs-solaris2.mc

cs-sunos4.1.mc

cs-ultrix4.mc

cyrusproto.mc

generic-bsd4.4.mc

generic-hpux10.mc

generic-hpux9.mc

generic-nextstep3.3.mc

generic-osf1.mc

generic-solaris2.mc

generic-sunos4.1.mc

generic-ultrix4.mc

huginn.cs.mc

knecht.mc

mail.cs.mc

mail.eecs.mc

mailspool.cs.mc

python.cs.mc

redhat.mc

s2k-osf1.mc

s2k-ultrix4.mc

tcpproto.mc

ucbarpa.mc

ucbvax.mc

uucpproto.mc

vangogh.cs.mc

For which the Redhat.mc is the most useful for starting a dail-up setup.

Sendmail for the SOHO

This is the configuration I use at home for two Linux boxes and a Win98 machine. All mail is sent to mail.morpheux.org from the MUA and masqueraded on the way in and out.

divert(-1)
dnl This is the macro config file used to generate the /etc/sendmail.cf
dnl file. If you modify thei file you will have to regenerate the
dnl /etc/sendmail.cf by running this macro config through the m4
dnl preprocessor:
dnl
dnl m4 /etc/sendmail.mc > /etc/sendmail.cf
dnl
dnl You will need to have the sendmail-cf package installed for this to
dnl work.
include(`../m4/cf.m4')
define(`confDEF_USER_ID',``8:12'')
OSTYPE(`linux')
undefine(`UUCP_RELAY')
undefine(`BITNET_RELAY')
define(`confAUTO_REBUILD')
define(`confTO_CONNECT', `1m')
define(`confTRY_NULL_MX_LIST',true)
define(`confDONT_PROBE_INTERFACES',true)
define(`PROCMAIL_MAILER_PATH',`/usr/bin/procmail')
define(`SMART_HOST', `mail.ednet.co.uk')
define(`confME_TOO', `true')
define(`confTO_QUEUEWARN', `8h')
define(`confDIAL_DELAY', `30s')
FEATURE(`smrsh',`/usr/sbin/smrsh')
FEATURE(mailertable)
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable')
FEATURE(redirect)
FEATURE(always_add_domain)
FEATURE(use_cw_file)
FEATURE(local_procmail)
MAILER(procmail)
MAILER(smtp)
FEATURE(`access_db')
FEATURE(`blacklist_recipients')
FEATURE(`rbl')
FEATURE(`accept_unresolvable_domains')
dnl We strongly recommend to comment this one out if you want to protect
dnl yourself from spam. However, the laptop and users on computers that do
dnl not hav 24x7 DNS do need this.
FEATURE(`relay_based_on_MX')
FEATURE(`masquerade_envelope')
FEATURE(`masquerade_entire_domain')
FEATURE(`limited_masquerade')
FEATURE(`allmasquerade')
MASQUERADE_DOMAIN(morpheus.ednet.co.uk)
MASQUERADE_AS(morpheux.org)
Cwlocalhost morpheux.org mail.morpheux.org morpheus.morpheus.ednet.co.uk mail.morpheus.ednet.co.uk eros.morpheus.ednet.co.uk morpheus.ednet.co.uk morpheus.morpheux.org eros.morpheux.org bob.morpheux.org
CMlocalhost morpheus.morpheux.org eros.morpheux.org bob.morpheux.org mail.morpheux.org

8.10.1 Features

New Database class for Arithmetic expresions

New map class arith to allow for computations in rules. The operation (+, -, *, /, l (for less than), and =) is given as key. The two operands are specified as arguments; the lookup returns the result of the computation. For example, "$(arith l $@ 4 $@ 2 $)" will return "FALSE" and "$(arith + $@ 4 $@ 2 $)" will return "6".

New Anti-spam rules

Mail can be rejectd on the basis of the headers of the email.

New Authentication protocols, which alows sendmail to recognise users allowed to contact the server.

Support SMTP AUTH (see RFC 2554). New macros for this purpose are ${auth_authen}, ${auth_type}, and ${auth_author} which hold the client's authentication credentials, the mechanism used for authentication, and the authorization identity (i.e., the AUTH= parameter if supplied). Based on code contributed by Tim Martin of CMU.

Support multiple queue directories.

Support multiple queue directories. To use multiple queues, supply a QueueDirectory option value ending with an asterisk. For example, /var/spool/mqueue/q* will use all of the directories or symbolic links to directories beginning with 'q' in /var/spool/mqueue as queue directories. Keep in mind, the queue directory structure should not be changed while sendmail is running. Queue runs create a separate process for running each queue unless the verbose flag is given on a non-daemon queue run. New items are randomly assigned to a queue. Contributed by Exactis.com, Inc.

Support different directories for qf, df, and xf queue files; if subdirectories or symbolic links to directories of those names exist in the queue directories, they are used for the corresponding queue files. Keep in mind, the queue directory structure should not be changed while sendmail is running. Proposed by Mathias Koerber of Singapore Telecommunications Ltd.

IPv6 support.

IPv6 support based on patches from John Kennedy of Cal State University, Chico, Motonori Nakamura of Kyoto University, and John Beck of Sun Microsystems.