Sieve implementation for Dovecot v1.2

For the most up-to-date information you are referred to the Dovecot wiki:

http://wiki.dovecot.org/LDA/Sieve/Dovecot

Introduction
------------

Sieve is a machine language specifically tailored for internet message 
filtering. This package compiles into a Sieve plugin for the Dovecot local 
delivery agent called Deliver. The plugin adds Sieve filtering support to the 
delivery process.  

Previously, the same functionality was provided by the cmusieve plugin for 
Dovecot. This old plugin is based on the CMU Sieve implementation included with
the Cyrus project. This new package provides a complete rewrite of the Sieve 
engine integrating it tightly with Dovecot. The actual execution of the Sieve 
actions is based on the original cmusieve plugin, but only on the code added to 
interface the CMU Sieve implementation with Dovocot. 

The main reason for rewriting the Sieve engine is to provide more reliable 
script execution and to provide better error messages to users and system 
administrators. Also, since the Sieve language evolves quickly, with new 
language extensions published every year, the aim is to provide support for 
quickly extending the engine with new features. 

Features
--------

* Well-structured 3-stage compiler:
 
  Uses dovecot framework and avoids using lex/yacc. Compiler doesn't bail on 
  first error, but tries to find more. Produced errors are aimed to be useful 
  and generally user-comprehensible. Things like 'Generic error' are a nuisance 
  of the past. 

* Highly extendable with new Sieve capabilities: 

  This keeps the possibility of plugins in mind. It should eventually provide 
  the necessary infrastructure for at least all currently known (proposed) 
  extensions. The goal is to keep the extension interface provided by sieve 
  engine as generic as possible, i.e. without explicit support for specific 
  extensions. New similar extensions can then use the same interface methods 
  without changes to the sieve engine code. If an extension is not loaded using 
  the require command, the compiler truly does not know of its existance. 

* Supports all extensions provided by the original CMUSieve plugin:
 
  In addition, it has support for the new and very useful variables extension
  (see next section). 

  NOTE: The original CMUSieve plugin is based on old specifications of the 
  imap4flags and enotify extension. Among other subtle differences, these 
  extensions were known as 'imapflags' and 'notify' for the CMU Sieve plugin.
  Support for the old imapflags extension is provided for backwards compatibility.

* Supports executing multiple scrips sequentially.
  
  Using this feature it is possible to execute administrator-controlled Sieve
  scripts before and after the user's Sieve script is executed. As long as the
  verdict is at least (implicit) keep, the execution will continue with the next
  script. Multiple scripts can be executed before or after the user's script by 
  specifying directories containing sieve files.

* Supported by ManageSieve service:

  This Sieve implementation is supported by the ManageSieve implementation for 
  Dovecot v1.2. Therefore, ManageSieve support can be added to Dovecot for the
  new Sieve plugin just as for the cmusieve plugin.

* Test suite included:
	
  This package includes a test suite to automatically asses whether the compiled 
  sieve engine works correctly. The test suite is an extension to the Sieve 
  language and is therefore easily extended with new tests. Currently, the 
  test suite is mostly limited to testing script processing. The performed actions 
  are not tested fully yet. 

Implementation Status
---------------------

The the core of the language (as specified in RFC 5228) is fully supported. In 
addition to that, this Sieve implementation features various extensions. The 
following list outlines the implementation status of each supported extension:

  The the core of the language (as specified in RFC 5228) is fully supported, 
  including the language extensions defined in the base specification:

    encoded-character (RFC 5228; page 10)
    fileinto (RFC 5228; page 23)
    envelope (RFC 5228; page 27)

  The following Sieve language extensions are also supported:

    copy (RFC 3894): fully supported
    body (RFC 5173): almost fully supported, but the text body-transform 
      implementation is simple and some issues make it still not completely
      RFC compliant.
    environment (RFC 5183): basic support is provided (v0.1.5+)
    variables (RFC 5229): almost fully supported, but currently no support 
      for namespaces is available (include depends on this)
    vacation (RFC 5230): fully supported
    relational (RFC 5231): fully supported
    imap4flags (RFC 5232): fully supported
    subaddress (RFC 5233): fully supported, but with limited configurability
    date (RFC 5260; page 3): fully supported (v0.1.12+)
    reject (RFC 5429; page 6): fully supported
    enotify (RFC 5435): fully supported (v0.1.3+), but only the mailto 
      notification mechanism is available
    mailbox (RFC 5490; page 2): fully supported (v0.1.10+), but 
      ACL permissions are not verified for mailboxexists
    include (draft): almost fully supported, but the global namespace is 
      missing and the global command is can only be placed at the top of the 
      script
    regex (draft): almost fully supported, but UTF-8 is not supported. 

  The following deprecated extensions are supported for backwards
  compatibility:

    imapflags (obsolete draft): fully backwards compatible (v0.1.3+)
    notify (obsolete draft): denotify command is a dummy  

    The availability of these deprecated extensions is disabled by default.

  The following extensions are under development:

    ereject (RFC 5429; page 4): implemented, but currently equal to reject 

  Many more extensions to the language exist. Not all of these extensions are
  useful for Dovecot in particular, but many of them are. Currently, the
  author has taken notice of the following extensions:

    spamtest and virustest (RFC 5235): planned
    index (RFC 5260; page 7): planned
    editheader (RFC 5293): planned
    foreverypart, mime, replace, enclose, and extracttext (RFC 5703): planned 

    These extensions will be added as soon as the necessary infrastructure is
    available.

Compiling and Configuring
-------------------------

Refer to INSTALL file.

Using
-----

The main purpose of this package is to replace the existing cmusieve plugin 
that is currently available for Dovecot's deliver. With respect to its main 
functionalityit is currently not very different from the cmusieve plugin 
implementation.

However, unlike cmusieve, this sieve module logs runtime errors to
<scriptfile>.log if it can and not <scriptfile>.err. Also, the cmusieve plugin
compiled the script into a file with an appended 'c', e.g. 'test.sievec'.
This new implementation recognizes scripts to have the .sieve  extension. 
The binary is (by default) written to a file with extension .svbin. This is
explained further in section `Script Compiling' below.

To test the sieve engine outside deliver, it is useful to try the commands that 
exist in the src/sieve-tools/ directory of this package. After installation, 
these are available at your $prefix/bin directory. The following commands are 
installed:

sievec     - Compiles sieve scripts into a binary representation for later 
             execution. Refer to `Script Compiling' section below. 

sieve-test - This is a universal Sieve test tool for testing the effect of a
             Sieve script on a particular message. It allows compiling, running 
             and testing Sieve scripts. It can either be used to display the
             actions that would be performed on the provided test message or it
             can be used to test the actual delivery of the message and show the
             messages that would normally be sent through SMTP.

sieved     - Dumps the content of a Sieve binary file for (development) 
             debugging purposes.

When installed, man pages are also available for these commands. In this package
the man pages are present in doc/man and can be viewed before install using
e.g.: 

man -l doc/man/sieve-test.1

Various example scripts are bundled in the directory 'examples'. These scripts
were downloaded from various locations. View the top comment in the scripts for 
url and author information.

Script Compiling
----------------

When the Sieve plugin executes a script for the first time (or after it has been
changed), it's compiled into into a binary form. Dovecot Sieve implementation
uses the .svbin extension to store compiled Sieve scripts (e.g. .dovecot.svbin).
To store the binary, the plugin needs write access in the directory in which the
script is located.

A problem occurs when a global script is encountered by the plugin. For security
reasons, global script directories are not supposed to be writable by the user.
Therefore, the plugin cannot store the binary when the script is first compiled.
Note that this doesn't mean that the old compiled version of the script is used
when the binary cannot be written: it compiles and uses the current script
version. The only real problem is that the plugin will not be able to update
the binary on disk, meaning that the global script needs to be recompiled each
time it needs to be executed, i.e. for every incoming message, which is
inefficient.

To mitigate this problem, the administrator must manually pre-compile global
scripts using the sievec command line tool. For example:

sievec /var/lib/dovecot/sieve/global/

This is necessary for scripts listed in the sieve_global_path, sieve_before and
sieve_after settings. For global scripts that are only included in other scripts
using the include extension, this step is not necessary, since included scripts
are incorporated into the binary produced for the main script.

When manually compiling scripts with sievec, if those scripts use the include
sieve extension and your sieve_dir is not the sieve subfolder of the directory
of the main file, you can specify it by defining the SIEVE_DIR environment
variable (e.g SIEVE_DIR=~/.sieve sievec .dovecot.sieve )

Compile and Runtime Logging
---------------------------

Log messages produced at runtime by the Sieve plugin are written to two
locations:

  * A log file is written in the same directory as the user's main private 
    script (as specified by the sieve setting). This log file bears the name of
    that script file appended with ".log", e.g. .dovecot.sieve.log. If there are
    errors or warnings in the script, the messages are appended to that log file
    until it eventually grows too large. When that happens, the old log file is
    rotated to a ".log.0" file and an empty log file is started. Informational
    messages are not written to this log file and the log file is not created
    until messages are actually logged, i.e. when an error or warning is
    produced.

  * Messages that could be of interest to the system administrator are also
    written to the Dovecot logging facility (usually syslog). This includes
    informational messages that indicate what actions are executed on incoming
    messages. Compile errors encountered in the user's private script are not
    logged here.

Known issues
------------

Most open issues are outlined in the TODO file. The more generic ones are (re-)
listed here:

- Compile errors are sometimes a bit obscure and long. This needs work. 
  Suggestions for improvement are welcome. 
- The documentation needs work.

Authors
-------

Refer to AUTHORS file.

Contact Info
------------

Stephan Bosch <stephan at rename-it dot nl>
IRC: Freenode, #dovecot, S[r]us

Please use the Dovecot mailing list <dovecot at dovecot.org> for questions about 
this package. You can post to the list without subscribing, the mail then waits 
in a moderator queue for a while. See http://dovecot.org/mailinglists.html
