HowTo: Inspect and synchronize SMTP

pwallace · ‎11-30-2020

If you wish to use Stingray to inspect or modify the headers or body of SMTP messages, you need to write some TrafficScript to do so.

Your TrafficScript code will run on a Server First protocol, and you will need to detect the start and end of each SMTP message (request and response) so that your code is executed correctly and you don't stall the connection by waiting for client (or server) data that will never arrive. This article explains how to do this, and presents an example TrafficScript rule that can be used as the basis of more advanced SMTP rules.

Background - salient details of SMTP

SMTP is a server first TCP protocol, usually run on port 25. Lines are terminated by carriage return and line feed (CRLF). A typical SMTP conversation proceeds as follows:

Client Server

1. Opens connection to server

2. Server sends banner

3. sends EHLO ...

4. Server acknowledges/rejects

# Header processing

5. sends MAIL FROM: [email protected]

6. Server acknowledges/rejects

7. sends RCPT TO: [email protected]

8. Server acknowledges/rejects

9. sends DATA

10. Server acknowledges

10. Client sends body terminated by '.' on new line

11. Server acknowledges

(Client optionally sends another email, by returning to step 5., otherwise sends "QUIT" and server closes connection)

It's important to note the following:

The protocol is server first (use the server_first banner setting in Stingray), so a proxy must send the server's banner before waiting for a request from the client.
Each request or response is terminated by CRLF. However, once the server has acknowledged the client's "DATA" request, the client continues to send CRLF terminated lines, without waiting for any response from the server, until it sends the final '.'
Real world clients may not wait for a response from the server before sending the next header line. It's quite usual to see a client send all the headers in the packet it sends at step 5. The proxy must handle this situation as well as the conventional alternating request/response sequence of events.

Synchronizing SMTP with TrafficScript

To handle SMTP, we need a TrafficScript rule that creates a state-machine with two states:

State 1. Reading Headers: Read the request a line at a time, sending each line on to the server. If the request is "DATA", then transition to state 2.

State 2. Reading Body: Read the request a line at a time, but don't send it to the server straight away. Instead buffer the request until the entire body has been received. Once the whole body has been received, send the body and transition to state 1, ready to receive another email or another command, such as QUIT.

Since our TrafficScript rule is going to be executed every time a new line of data is received from the client, not just once per email sent, we need to keep track of the state between invocations of the rule. TrafficScript provides connection.data.set() and connection.data.get() to set flags that will let us keep track of which state we are in. As their names suggest, these commands allow you to set and read variables that persist as long as the TCP connection they are associated with.

Multiple lines might arrive in the same packet from the client, even though we only wish to pass them to the server one at a time. We therefore need to indicate to Stingray how to split the data into individual lines. We can do this using request.endsWith(), which takes a single argument, a regular expression that defines the end of the current request. Any data that comes after this will be kept for the next time the request rule is run.

Putting this together, the following is a request rule that will synchronise SMTP. There are three places in it from which you could hook in code to inspect/modify the headers and body on the way through.

# SMTP Synchronizer - Stingray Request Rule  
  
  
if (connection.data.get("BODY_COMPLETE")) {  
    # We've finished one email, so reset the state flags, in case  
    # another email is on its way.  
    connection.data.set ("DATA_RECEIVED", 0);  
    connection.data.set ("BODY_COMPLETE", 0);  
}  
  
if (! connection.data.get("DATA_RECEIVED")) {  
    # State 1: Reading Headers  
    # We haven't seen "DATA" yet, so process headers, line by line  
  
    # Read the next line of the request into $request  
    $request = request.endsWith("\r\n");  
    if (string.regexmatch($request, "^DATA\r\n")) {  
        # DATA seen - set the DATA_RECEIVED flag to switch into  
        # the Reading Body state  
        connection.data.set("DATA_RECEIVED", 1);  
    }  
  
    #--------------------------------------------------  
    # INSPECTION/MODIFICATION HOOK  
    # USE $request TO MODIFY/INSPECT HEADERS HERE  
    #--------------------------------------------------  
  
} else {  
    # State 2: Reading Body  
    # Keep reading the body until we see the end of it  
    $line = "";  
    $next = 0;  
  
    # The regular expression in the next line matches the  
    # "end of body" indicator (a full stop on a line on its own)  
    while ( ! string.regexmatch($line, "^\\.\r\n")) {  
        $line = request.getline("\r\n", $next);  
        $next = $1;  
  
        #----------------------------------------------------  
        # INSPECTION/MODIFICATION HOOK  
        # USE $line TO INSPECT/MODIFY BODY LINE BY LINE HERE  
        #----------------------------------------------------  
  
        $request = $request . $line;  
    }  
  
    #--------------------------------------------------  
    # INSPECTION/MODIFICATION HOOK  
    # USE $request to INSPECT/MODIFY WHOLE BODY HERE  
    #--------------------------------------------------  
  
    # We've read the whole body, so set the BODY_COMPLETE flag  
    connection.data.set("BODY_COMPLETE", 1);  
}  
  
# Send the request on to the back-end server.  
# At this point $request will contain either a single header line, or  
# the entire body, depending on which state we are in.  
  
request.set($request);

You should note the difference in behavior of request.endsWith() and request.getline(). Whilst they both return some data from the request, request.endsWith() tells Stingray that this is the end of this request, and no more data should be read from the client until after the request has been sent to the server and a response received. request.getline(), on the other hand, merely reads some data from the client without ending the request, and so can be called multiple times within the same invocation of a request rule. If no data is available when request.getline() is called, processing will pause until the client sends some. We can thus use it to read and buffer the body. If we tried to use request.endsWith() instead, then the connection would stall. Stingray would read the first line of the body, send it to the server, then pause waiting for a response from the server, which would never come.

The condensed version

If you don't want to perform line by line inspection or modification of the body, you can condense the above rule into the more elegant one below. This uses request.endswith() to read both the headers and the body, with only a single connection.data variable that both tracks the state and supplies the appropriate regular expression to request.endswith().

# SMTP Synchronizer Condensed- ZXTM Request Rule                                          
  
# State is tracked in the connection.data variable "STATE", which is             
# set to one of the following two constants.  The constants themselves           
# are the regular expressions supplied as arguments to                           
# request.endsWith() to define the end of a request in each state.  
  
$STATE_HEADERS = "\r\n";  
$STATE_BODY    = "\r\n\\.\r\n";  
  
# Initialise STATE if this is the first request on a new connection              
if (! connection.data.get("STATE")) {  
    connection.data.set("STATE", $STATE_HEADERS);  
}  
  
$request = request.endsWith(connection.data.get("STATE"));  
  
  
#### STATE SWITCHING                                                              
# We read the whole body in one go, so if we've got to here, and the             
# state is STATE_BODY, then switch back to reading headers.                      
if (connection.data.get("STATE") == $STATE_BODY) {  
    connection.data.set("STATE", $STATE_HEADERS);  
} else if (string.regexmatch($request, "^DATA\r\n")) {  
    # Whereas if we were reading headers, and see DATA, we should                
    # prepare ourselves for the body                                             
    connection.data.set("STATE", $STATE_BODY);  
}  
#### END STATE SWITCHING                                                         
  
#----------------------------------------------------------------------                              
# INSPECTION/MODIFICATION HOOK                                                   
# USE $request to INSPECT/MODIFY WHOLE BODY OR INDIVIDUAL HEADERS HERE                                 
#----------------------------------------------------------------------                              
  
request.set($request);

For more details and examples of this synchronization technique, you can refer to the following resources:

HowTo: Inspect and synchronize SMTP

Background - salient details of SMTP

Synchronizing SMTP with TrafficScript

The condensed version

Tech Tips