Subversion Repositories planix.SVN

.TH SCANMAIL 8

.SH NAME

scanmail, testscan \-  spam filters

.SH SYNOPSIS

.B upas/scanmail

[

.I options

]

[

.I qer-args

]

.I root

.B mail

.I sender system rcpt-list

.PP

.B upas/testscan

[

.B -avd

]

[

.B -p

.I patfile

]

[

.I filename

]

.SH DESCRIPTION

.B Scanmail

accepts a mail message supplied on standard input,

applies a file of patterns to a portion of it,

and dispatches

the message based

on the results.

It exactly replaces the

generic queuing command

.IR qer (8)

that is executed from the

.IR rc (1)

script

.B /mail/lib/qmail

in the mail processing pipeline.

Associated with each pattern is an

.I action

in order of decreasing priority:

.in +5

.TP 10

.B dump

the message is deleted and a log entry is written to

.B /sys/log/smtpd

.TP 10

.B hold

the message is placed in a queue for human inspection

.TP

.B log

a line containing the matching portion of the message is written to a log

.in -5

.PP

If no pattern matches or only patterns with an action of

.B log

match, the message is accepted and

.I scanmail

queues the message for delivery.

.I Scanmail

meshes with the blocking facilities

of

.IR smtpd (6)

to provide several layers of

filtering on gateway systems.  In all cases the sender

is notified that the message has been successfully

delivered,

leaving the sender unaware that the message has been potentially delayed or deleted.

.PP

.I Scanmail

accepts the arguments of

.IR qer (8)

as well as the following:

.TF filename

.TP

.B  -c

Save a copy of each message in a

randomly-named file in

directory

.BR /mail/copy .

.TP

.B -d

Write debugging information to standard error.

.TP

.B -h

Queue

.I held

messages by sending domain name.

The

.B -q

option must specify a root directory; messages

are queued in subdirectories of this directory.

If the

.B -h

option is not specified,

messages are accumulated in a subdirectory of

.B /mail/queue.hold

named for the contents of

.BR /dev/user ,

usually

.BR none .

.TF filename

.TP

.B -n

Messages are never held for inspection, but are delivered.  Also known as

.IR "vacation mode" .

.TP

.BI -p " filename"

Read the patterns from

.I filename

rather than

.BR /mail/lib/patterns .

.TP

.BI -q " holdroot"

Queue deliverable messages in subdirectories of

.IR holdroot .

This option is the same as the

.B -q

option of

.IR qer (8)

and must be present if the

.B -h

option is given.

.TP

.B  -s

Save deleted

messages.   Messages are stored, one per randomly-named file,

in subdirectories of

.B /mail/queue.dump

named with the date.

.TP

.B -t

Test mode.  The pattern matcher is applied but the message is

discarded and the result is not logged.

.TP

.B -v

Print the highest priority match.

This is useful

with the

.B -t

option for testing the pattern matcher without actually

sending a message.

.PD

.PP

.I Testscan

is the command line version of

.IR scanmail .

If

.I filename

is missing, it applies the pattern set to

the message on standard input.  Unlike

.IR scanmail ,

which finds the highest priority match,

.I testscan

prints all matches in the portion of the message under test.

It is useful for testing a pattern set or

implementing a personal filter

using the

.B pipeto

file in a user's mail directory.

.I Testscan

accepts the following options:

.TP

.B -a

Print matches in the complete input message

.TP

.B -d

Enable debug mode

.TP

.B -v

Print the message after conversion to canonical form

.RI ( q.v. ).

.TP

.BI -p " filename"

Read the patterns from

.I filename

rather than

.BR /mail/lib/patterns .

.SS Canonicalization

Before pattern matching, both programs convert a portion of

the message header and the beginning of the

message to a canonical form.  The amount of the header

and message body processed are set by

compile-time parameters in the source files.

The canonicalization process converts letters to lower-case and

replaces consecutive spaces, tabs and newline characters

with a single space.  HTML commands are

deleted except for the parameters following

.B A

.BR HREF ,

.B IMG

.BR SRC ,

and

.B IMG

.B BORDER

directives.  Additionally, the following MIME escape sequences

are replaced by their ASCII

equivalents:

.PP

.EX

           Escape Seq   ASCII

           ----------   -----

                =2e       .

                =2f       /

                =20    <space>

                =3d       =

.EE

and the sequence

.I =<newline>

is elided.

.I Scanmail

assembles the sender, destination domain and recipient fields of

the command line into a string that is

subjected to the same canonical processing.

Following canonicalization, the command line and

the two long strings containing

the header and the message body are passed to the

matching engine for analysis.

.SS Pattern Syntax

The matching engine compiles the pattern set

and matches it to each canonicalized input string.

Patterns are specified one per line

as follows:

.PP

.EX

	{*}\fIaction\fP: \fIpattern-spec\fP {~~\fIoverride\fP...~~\fIoverride\fP}

.EE

.PP

On all lines, a

.B #

introduces a comment; there is no way to escape this character.

.PP

Lines beginning with

.B *

contain a

.I pattern-spec

that is a string; otherwise, the the

.I pattern-spec

is a regular expression in the style of

.IR regexp (6).

Regular expression matching is many

times less efficient than string matching, so it is

wiser to enumerate several similar strings

than to combine them into a regular expression.

The

.I action

is a keyword terminated by a

.B : 

and separated from the pattern by optional white-space.

It must be one of the following:

.TP 10

.B dump

if the pattern matches, the message is deleted.  If the

.B -s

command line option is set, the message is saved.

.TP 10

.B hold

if the pattern matches, the message is queued in a subdirectory

of

.B /mail/queue.hold

for manual inspection.  After inspection, the queue can be swept

manually using

.B runq

(see

.IR qer (8))

to deliver messages that were inadvertently matched.

.TP 10

.B header

this is the same as the

.B hold

action, except the pattern is only applied to the message header.

This optimization is useful for patterns that match header fields

that are unlikely to be present in the body of the message.

.TP 10

.B line

the sender and a section of the message around the match are written to

the file

.BR /sys/log/lines .

The message is always delivered.

.TP 10

.B loff

patterns of this type are applied only to the canonicalized command line.

When a match occurs, all patterns with

.B line

actions are disabled.  This is useful for limiting

the size of the log file by excluding repetitive messages, such

as those from mailing lists.

.PP

Patterns are accumulated into pattern sets sharing the same action.

The matching engine applies the

.B dump

pattern set first, then the

.B header

and

.B hold

pattern sets, and finally the

.B line

pattern set.  Each pattern set is applied three times:

to the canonicalized command line, to the message header, and

finally to the message body.  The ordering of patterns

in the pattern file is insignificant.

.PP

The

.I pattern-spec

is a string of characters terminated by a

.BR newline ,

.B #

or override indicator,

.BR ~~ .

Trailing white-space is deleted but

patterns containing leading or trailing white-space can

be enclosed in double-quote

characters.  A pattern containing a double-quote

must be enclosed in double-quote

characters and preceded by a backslash.

For example, the pattern

.PP

.EX

	"this is not \\"spam\\""

.EE

.PP

matches the string \fLthis is not "spam"\fP.

The

.I pattern-spec

is followed by zero or more

.I override

strings.  When the specific pattern matches,

each override is applied and

if one matches, it cancels the effect of the pattern.

Overrides must be strings; regular expressions are not supported.

Each override is introduced by the string

.BR ~~

and continues until a subsequent

.BR ~~ ,

.B #

or

.BR newline ,

white-space included.

A

.B ~~

immediately followed by a

.B newline

indicates a line continuation and further overrides continue

on the following line.

Leading white-space

on the continuation line is ignored.  For example,

.PP

.EX

        *hold:   sex.com~~essex.com~~sussex.com~~sysex.com~~

                 lasex.com~~cse.psu.edu!owner-9fans

.EE

.PP

matches all input containing the string

.B sex.com

except for messages that also contain the

strings in the override list.  Often it

is desirable to override a pattern based on

the name of the sender or

recipient.  For this reason, each override

pattern is applied to the header and the command line as well

as the section of the

canonicalized input containing the matching data.

Thus a pattern matching the command line or the header

searches both the command line and the header

for overrides while a match in the body searches

the body, header and command line for overrides.

.PP

The structure of the pattern file and the matching

algorithm define the strategy for detecting

and filtering unwanted messages.  Ideally, a

.B hold

pattern selects a message for inspection and if it

is determined to be undesirable, a specific

.B dump

pattern is added to delete further instances

of the message.  Additionally, it is often

useful to block the sender by updating the

.B smtpd

control file.

.PP

In this regime, patterns with a

.I dump

action, generally match phrases

that are likely to be unique.  Patterns that

hold a message for inspection

match phrases commonly found in undesirable material and

occasionally in legitimate messages.  Patterns

that log matches are less specific yet.  In all

cases the ability to override a pattern by

matching another string, allows repetitive messages

that trigger the pattern, such as mailing lists,

to pass the filter after the first one is processed

manually.  The

.B -s

option allows deleted messages to be salvaged

by either manual or semi-automatic review, supporting

the specification of more aggressive patterns.

Finally, the utility of the pattern matcher is not

confined to filtering spam; it is a generally useful

administrative tool for deleting inadvertently harmful

messages, for example, mail loops, stuck senders or viruses.

It is also useful for collecting or counting messages

matching certain criteria.

.SH FILES

.TF /mail/queue.dump/*

.TP

.B /mail/lib/patterns

default pattern file

.TP

.B /sys/log/smtpd

log of deleted messages

.TP

.B /mail/log/lines

file where

.I log

matches are logged

.TP

.B /mail/queue/*

directories where legitimate messages are queued for delivery

.TP

.B /mail/queue.hold

directory where held messages are queued for inspection

.TP

.B /mail/queue.dump/*

directory where

.I dumped

messages are stored when the

.B -s

command line option is specified.

.TP

.B /mail/copy/*

directory where copies of all incoming messages

are stored.

.SH SOURCE

.TP

.B /sys/src/cmd/upas/scanmail

.SH "SEE ALSO"

.IR mail (1),

.IR qer (8),

.IR smtpd (6)

.SH BUGS

.I Testscan

does not report a match when the body of a message

contains exactly one line.