Checking IP addresses against a DNS blacklist with Traffic Manager

pwallace · ‎06-14-2019

This short article explains how you can match the IP addresses of remote clients with a DNS blacklist. In this example, we'll use the Spamhaus XBL blacklist service (http://www.spamhaus.org/xbl/).

This article updated following discussion and feedback from Ulrich Babiak - thanks!

Basic principles

The basic principle of a DNS-based blacklist such as Spamhaus' is as follows:

Perform a reverse DNS lookup of the IP address in question, using xbl.spamhaus.org rather than the traditional in-addr.arpa domain
Entries that are not in the blacklist don't return a response (NXDOMAIN); entries that are in the blacklist return a particular IP/domain response indicating their status

Important note: some public DNS servers don't respond to spamhaus.org lookups (see http://www.spamhaus.org/faq/section/DNSBL%20Usage#261). Ensure that Traffic Manager is configured to use a working DNS server.

Simple implementation

A simple implementation is as follows:

1

2

3

4

5

6

7

8

9

10

11

$ip = request.getRemoteIP();  
   
# Reverse the IP, and append ".zen.spamhaus.org".  
$bytes = string.dottedToBytes( $ip );  
$bytes = string.reverse( $bytes );  
$query = string.bytesToDotted( $bytes ).".xbl.spamhaus.org";  
   
if( $res = net.dns.resolveHost( $query ) ) {  
   log.warn( "Connection from IP ".$ip." should be blocked - status: ".$res );  
   # Refer to Zen return codes at http://www.spamhaus.org/zen/  
}  

This implementation will issue a DNS request on every request, but Traffic Manager caches DNS responses internally so there's little risk that you will overload the target DNS server with duplicate requests:

Traffic Manager DNS settings in the Global configuration

You may wish to increase the dns!negative_expiry setting because DNS lookups against non-blacklisted IP addresses will 'fail'.

A more sophisticated implementation may interpret the response codes and decide to block requests from proxies (the Spamhaus XBL list), while ignoring requests from known spam sources.

What if my DNS server is slow, or fails? What if I want to use a different resolver for the blacklist lookups?

One undesired consequence of this configuration is that it makes the DNS server a single point of failure and a performance bottleneck. Each unrecognised (or expired) IP address needs to be matched against the DNS server, and the connection is blocked while this happens.

In normal usage, a single delay of 100ms or so against the very first request is acceptable, but a DNS failure (Stingray times out after 12 seconds by default) or slowdown is more serious.

In addition, Traffic Manager uses a single system-wide resolver for all DNS operations. If you are hosting a local cache of the blacklist, you'd want to separate DNS traffic accordingly.

Use Traffic Manager to manage the DNS traffic?

A potential solution would be to configure Traffic Manager to use itself (127.0.0.1) as a DNS resolver, and create a virtual server/pool listening on UDP:53. All locally-generated DNS requests would be delivered to that virtual server, which would then forward them to the real DNS server. The virtual server could inspect the DNS traffic and route blacklist lookups to the local cache, and other requests to a real DNS server.

You could then use a health monitor (such as the included dns.pl) to check the operation of the real DNS server and mark it as down if it has failed or times out after a short period. In that event, the virtual server can determine that the pool is down (pool.activenodes() == 0) and respond directly to the DNS request using a response generated by HowTo: Respond directly to DNS requests using libDNS.rts.

Re-implement the resolver

An alternative is to re-implement the TrafficScript resolver using Matthew Geldert's libDNS.rts: Interrogating and managing DNS traffic in Traffic Manager TrafficScript library to construct the queries and analyse the responses. Then you can use the TrafficScript function tcp.send() to submit your DNS lookups to the local cache (unfortunately, we've not got a udp.send function yet!):

sub resolveHost( $host, $resolver ) {  
   import libDNS.rts as dns;  
   
   $packet = dns.newDnsObject();   
   $packet = dns.setQuestion( $packet, $host, "A", "IN" );  
   $data = dns.convertObjectToRawData( $packet, "tcp" );   
   
   $sock = tcp.connect( $resolver, 53, 1000 );  
   tcp.write( $sock, $data, 1000 );  
   $rdata = tcp.read( $sock, 1024, 1000 );  
   tcp.close( $sock );  
   
   $resp = dns.convertRawDatatoObject( $rdata, "tcp" );  
   
   if( $resp["answercount"] >= 1 ) return $resp["answer"][0]["host"];  
}  

Note that we're applying 1000ms timeouts to each network operation.

Let's try this, and compare the responses from OpenDNS and from Google's DNS servers. Our 'bad guy' is 201.116.241.246, so we're going to resolve 246.241.116.201.xbl.spamhaus.org:

$badguy = "246.241.116.201.xbl.spamhaus.org ";  
   
$text .= "Trying OpenDNS...\n";  
$host = resolveHost( $badguy, "208.67.222.222" );  
if( $host ) {  
   $text .= $badguy . " resolved to " . $host . "\n";  
} else {  
   $text .= $badguy . " did not resolve\n";  
}  
   
$text .= "Trying Google...\n";  
$host = resolveHost( $badguy, "8.8.8.8" );  
if( $host ) {  
   $text .= $badguy . " resolved to " . $host . "\n";  
} else {  
   $text .= $badguy . " did not resolve\n";  
}  
   
http.sendResponse( 200, "text/plain", $text, "" );  

(this is just a snippet - remember to paste the resolveHost() implementation, and anything else you need, in here)

This illustrates that OpenDNS resolves the spamhaus.org domain fine, and Google does not issue a response.

Caching the responses

This approach has one disadvantage; because it does not use Traffic Manager's resolver, it does not cache the responses, so you'll hit the resolver on every request unless you cache the responses yourself.

Here's a function that calls the resolveHost function above, and caches the result locally for 3600 seconds. It returns 'B' for a bad guy, and 'G' for a good guy:

sub getStatus( $ip, $resolver ) {  
   $key = "xbl-spamhaus-org-".$resolver."-".$ip; # Any key prefix will do  
     
   $cache = data.get( $key );  
   if( $cache ) {  
      $status = string.left( $cache, 1 );  
      $expiry = string.skip( $cache, 1 );  
        
      if( $expiry < sys.time() ) {  
         data.remove( $key );  
         $status = "";  
      }  
   }  
     
   if( !$status ) {  
   
      # We don't have a (valid) entry in our cache, so look the IP up  
     
      # Reverse the IP, and append ".xbl.spamhaus.org".  
      $bytes = string.dottedToBytes( $ip );  
      $bytes = string.reverse( $bytes );  
      $query = string.bytesToDotted( $bytes ).".xbl.spamhaus.org";  
     
      $host = resolveHost( $query, $resolver );  
     
      if( $host ) {  
         $status = "B";  
      } else {  
         $status = "G";  
      }  
      data.set( $key, $status.(sys.time()+3600) );  
   }  
   return $status;  
}  

ubabiak · ‎04-24-2013

Thanks for this handy article - it makes a helpful starting point. Anybody using this recipe in a copy-and-paste manner should replace zen.spamhaus.org with xbl.spamhaus.org to avoid blocking of users with residential IPs.

Like you say this solution might become a performance bottleneck if the DNS lookups are too slow ...

Actually, DNS Usage for BL lookups is quite different from normal lookups so for me it feels a bit risky to make all these changes (i.e. running a DNS vhost , making the dns cache 10 times as high, minimize the dns lookup timeout etc.) because it would mean to "bend" the global DNS settings only for this single purpose.

In our network, we are already running a dedicated fast local Blacklist resolver where we combine the blacklist data of several sources , but I don't see how I could use it from Stingray since that requires the configuration of a separate resolver IP just for this purpose.

Ideally, the dns resolver functions in Trafficscript would have parameters for both dnsserver and timeout. While the first might be a little bit far-fetched and difficult to implement , the timeout parameter would be a very useful feature - not only for BL checks but for other use cases of dns lookups, too. Actually any network-related function should have an optional timeout parameter ...

Regards,

Ulrich

(PS: there is a little typo in line 9 using + instead of . for string concat)

owen · ‎04-24-2013

Hi Ulrich,

All services on the Stingray host (including net.dns.resolvehost) uses the system resolver;

If you'd like to use a different resolver for BL lookups, one option is to make 'localhost' the system resolver, and create a virtual server that examines and routes DNS traffic. For example, you could use Matthew Geldert's libDNS.rts: Interrogating and managing DNS traffic in Stingray library to examine the queries and route lookups for *.spamhaus.org to a different DNS server.

A better option would be to re-implement net.dns.resolvehost so that it can target a specific nameserver. Try this:

sub resolveHost( $host, $resolver ) {

import libDNS.rts as dns;

$packet = dns.newDnsObject();

$packet = dns.setQuestion( $packet, $host, "A", "IN" );

$data = dns.convertObjectToRawData( $packet, "tcp" );

$sock = tcp.connect( $resolver, 53, 1000 );

tcp.write( $sock, $data, 1000 );

$rdata = tcp.read( $sock, 1024, 1000 );

tcp.close( $sock );

$resp = dns.convertRawDatatoObject( $rdata, "tcp" );

if( $resp["answercount"] >= 1 ) return $resp["answer"][0]["host"];

}

$badguy = "120.104.22.84.zen.spamhaus.org";

$text .= "Trying OpenDNS...\n";

$host = resolveHost( $badguy, "208.67.222.222" );

if( $host ) {

$text .= $badguy . " resolved to " . $host . "\n";

} else {

$text .= $badguy . " did not resolve\n";

}

$text .= "Trying Google...\n";

$host = resolveHost( $badguy, "8.8.8.8" );

if( $host ) {

$text .= $badguy . " resolved to " . $host . "\n";

} else {

$text .= $badguy . " did not resolve\n";

}

http.sendResponse( 200, "text/plain", $text, "" );

This illustrates that OpenDNS resolves the spamhaus.org domain fine, and Google does not issue a response.

This approach has one disadvantage; because it does not use Stingray's resolver, it does not cache the responses, so you'll hit the resolver on every request unless you cache the responses yourself.

Here's a function that calls the resolveHost function above, and caches the result locally for 3600 seconds. It returns 'B' for a bad guy, and 'G' for a good guy:

sub getStatus( $ip, $resolver ) {

$key = "zen-spamhaus-org-".$resolver."-".$ip;

$cache = data.get( $key );

if( $cache ) {

$status = string.left( $cache, 1 );

$expiry = string.skip( $cache, 1 );

if( $expiry < sys.time() ) {

data.remove( $key );

$status = "";

}

if( !$status ) {

# We don't have a (valid) entry in our cache, so look the IP up

# Reverse the IP, and append ".zen.spamhaus.org".

$bytes = string.dottedToBytes( $ip );

$bytes = string.reverse( $bytes );

$query = string.bytesToDotted( $bytes ).".zen.spamhaus.org";

$host = resolveHost( $query, $resolver );

if( $host ) {

$status = "B";

} else {

$status = "G";

}

data.set( $key, $status.(sys.time()+3600) );

}

return $status;

}

I'll update the article above, and thanks for pointing out the error in line 9; I'll correct that too.

regards

Owen

ubabiak · ‎04-24-2013

A great workaround - you pretty much nailed every aspect : custom DNS server, short timeout, caching - wow! While I still don't feel comfortable with adding so much additional overhead for every request, I might be forced to try it if them bots continue to get on my nerves as much as they currently do ... Thanks for this instructive and informative gem of insider wisdom

Checking IP addresses against a DNS blacklist with Traffic Manager

Basic principles

Simple implementation

What if my DNS server is slow, or fails? What if I want to use a different resolver for the blacklist lookups?

Use Traffic Manager to manage the DNS traffic?

Re-implement the resolver

Caching the responses

Solutions