Just a short status update regarding the automatic renewal: even after the latest upgrade (18.2 on a VMware appliance) the renewals are not completed automatically . Actually we are using the letsencryptforvtm.sh script from this thread, which was of course not changed by the 18.2 upgrade, but I was thinking that maybe the environment might have changed in 18.2.   According to the zxtm error log, the renewal command ran fine, but the certificates are not placed into their respective directory. I looked into letsencryptforvtm.sh, all the steps/commands seem to have been executed. The last step there is to execute the generated *.script files with zcli. Since these *.script files are generated correctly, I manually ran   /opt/zeus/zxtm/bin/zcli MYDOMAIN/zcli_MYDOMAIN.script Result: no error messages, but also no certifcates in the cert directory. After that, I ran the renewal command exactly as logged in the error log:   [17/Sep/2018:10:17:24 +0200] INFO actions/letsencrypt_renewal handler_ignore Running program: /opt/zeus/zxtm/conf/actionprogs/letsencryptforvtm.sh "--eventtype=LetsEncrypt SSL Certificate Expiry" "WARN vservers/xxxxxx_ssl vssslcerttoexpire expires=1537764393 Public SSL certificate 'c_xxxxxx_rsa' will expire on Mon, 24 Sep 2018 04:46:33 GMT"  Manual execution: /opt/zeus/zxtm/conf/actionprogs/letsencryptforvtm.sh "--eventtype=LetsEncrypt SSL Certificate Expiry" "WARN vservers/xxxxxx_ssl vssslcerttoexpire expires=1537764393 Public SSL certificate 'c_xxxxxxx_rsa' will expire on Mon, 24 Sep 2018 04:46:33 GMT"  Result: after manually executing the command, the certificates were renewed sucessfully.   Regards Ulrich ... View more

Hey, what about integrating Letsencrypt process as a native feature? That must have been on the roadmap as far as I remember, wasn't it? Given that from now on Chrome is flagging all http sites as insecure, it becomes more and more necessary to automatically renew the certificates in a safe and less clumsy way. Regards Ulrich ... View more

Hello Matthias,   I have addressed the details of this problem (along with TSR files and logs) in the support case number 00307146 - you might have access to it (do you?) Since the ownership of that case changed recently,  I now hope to see progress there regarding the aborted connections.   This community post was meant to collect additional input about the optimal backend configuration, which is something that can be as well discussed in public and would make a good topic for a KB article. The default, even if the settings appear to work, might not be the optimal/fastest setup ...   Our nodes are plain debian (8.10) , with the following packages: ii apache2 2.4.10-10+deb8u12 amd64 Apache HTTP Server ii apache2-mpm-event 2.4.10-10+deb8u12 amd64 transitional event MPM package for apache2 (...) ii libgnutls-openssl27:amd64 3.3.8-6+deb8u7 amd64 GNU TLS library - OpenSSL wrapper ii libssl-dev:amd64 1.0.1t-1+deb8u8 amd64 Secure Sockets Layer toolkit - development files ii libssl1.0.0:amd64 1.0.1t-1+deb8u8 amd64 Secure Sockets Layer toolkit - shared libraries ii openssl 1.0.1t-1+deb8u8 amd64 Secure Sockets Layer toolkit - cryptographic utility ii ssl-cert 1.0.35 all simple debconf wrapper for OpenSSL (...)   The VATM config is mostly default -  non-default settings are kept in the conf/settings.cfg , right? So these are the settings: # grep ssl conf/settings.cfg ssl!allow_rehandshake safe ssl!cache!expiry 1800 ssl!cipher_suites ssl!support_ssl3 No # grep ssl conf/vservers/VHOST-XXXX log!ssl_failures       Yes ssl_decrypt     Yes ssl_headers     No ssl_ocsp!issuer!_DEFAULT_!aia   Yes ssl_ocsp!issuer!_DEFAULT_!nonce off ssl_ocsp!issuer!_DEFAULT_!required      optional ssl_ocsp!issuer!_DEFAULT_!responder_cert ssl_ocsp!issuer!_DEFAULT_!signer ssl_ocsp!issuer!_DEFAULT_!url ssl_ocsp_stapling       No On the nodes, we also kept the defaults:   # from ssl.conf : SSLCipherSuite HIGH:!aNULL # Default: Off #SSLHonorCipherOrder on # The protocols to enable. # Available values: all, SSLv3, TLSv1, TLSv1.1, TLSv1.2 # SSL v2 is no longer supported SSLProtocol all -SSLv3 There might be potential for optimizations  regarding the cipher suites, do you have any recommendations ? Regards Ulrich     ... View more

@pwallace: a wrap-up would be great! Let me add just one more observation: I compared the environment vars between the ( failing ) action-triggered call and the (successful) manual . The only relevant difference seems to be the presence of LE_WORKING_DIR=/root/.acme.sh BTW we are on a vmware virtual appliance , running 17.2r1   ### Environment of automatically triggered script call PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin:/bin PWD=/opt/zeus ZEUSHOME=/opt/zeus LANG=en_US.UTF-8 SHLVL=1 _=/usr/bin/env ### ---------------------------- ### Environment of manual call (shortened) LC_ALL=en_US USER=admin PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games PWD=/opt/zeus/zxtm/conf/actionprogs ZEUSHOME=/opt/zeus LANG=en_US.UTF-8 SHLVL=1 HOME=/root LOGNAME=admin LE_WORKING_DIR=/root/.acme.sh regards Ulrich ... View more

Hi,   I did the renewal on the command line first, and only afterwards I looked into the permissions. So I can't tell you whether the changes were sufficient.  The renewal process touches/creates some files below the .acme.sh directory, so I made this directory and the specific vhost subdirectory  world writable.  However there is a one week period between the first alarm event and the expiry, so that should give you enough time to renew the cert.   These are the files that were modified:   #find .acme.sh -mtime -2 .acme.sh/account.conf .acme.sh/www.MYDOMAIN.de .acme.sh/www.MYDOMAIN.de/www.MYDOMAIN.de.conf .acme.sh/www.MYDOMAIN.de/ca.cer .acme.sh/www.MYDOMAIN.de/www.MYDOMAIN.de.cer .acme.sh/www.MYDOMAIN.de/zcli_www.MYDOMAIN.de.script .acme.sh/www.MYDOMAIN.de/www.MYDOMAIN.de.csr.conf .acme.sh/www.MYDOMAIN.de/fullchain.cer .acme.sh/www.MYDOMAIN.de/www.MYDOMAIN.de.csr .acme.sh/http.header Regards Ulrich ... View more

I am using a vATM virtual appliance. 3 months after successfully installing a letsencrypt cert, I encountered the first expiry action. According to zxtm/log/errors, the script was called with the following parameters when the expiry alert was triggered: [08/Nov/2017:12:48:43 +0100] INFO actions/letsencrypt_renewal handler_ignore Running program: /opt/zeus/zxtm/conf/actionprogs/letsencryptforvtm.sh "--eventtype=LetsEncrypt SSL Certificate Expiry" "WARN vservers/MY_VHOST vssslcerttoexpire expires=1510742940 Public SSL certificate 'c_www.MYDOMAIN.XX_rsa' will expire on Wed, 15 Nov 2017 10:49:00 GMT" [08/Nov/2017:12:48:45 +0100] INFO actions/letsencrypt_renewal handler_ignore Program finished However, there was no new cert, the expiry warning persisted. Since there is no other logging output, I could not figure out what went wrong. Calling the script manually using the "admin" account (= root on the vATM) on the command line with the exact same parameters worked fine, so I assume it might have been a permission problem on the .acme.sh subdirectory.   I would suggest to revise the acme.sh postinstallation instructions regarding the permissions, +x is probably not sufficient.   Regards, Ulrich ... View more

@Baptiste Assmann wrote: This installation procedure can be applied the same way if you're running the Brocade vTM appliance or the Brocade vTM software on your own Linux distribution.   First of all thank you for developing this lets-encrypt-workaround.    An annotation for anyone who follows these step-by-step-instructions:   I set up and tested  the process with the TEST variable enabled. Even after disabling TEST for production, I kept getting test certificates ("CN=Fake LE Intermediate X1" ) for the domains that i used during the test. Probably acme.sh was caching some values? Since I didn't have the time for extended debugging, I simply deleted the .acme.sh directory and re-installed acme.sh. From then on, correct production certs were generated. Great!   Now I wonder  how to  protect the letsencrypt setup from being erased on every system upgrade if using the (virtual/VMware) Appliance ? Currently, I am hiding my custom stuff inside a directory on the /logs/ partition .... But maybe  the next update already contains the native LetsE-Support  so no protection is needed? Regards, Ulrich ... View more

At the time when we started this discussion,  searching for heartbleed or the CVE-ID did not turn up anything, but now we have what we are looking for. Thanks for providing the link ( and to Riverbed for providing the information ... ) ... View more

dhodges schrieb: While the above all seems true, it would be good to see an official response from Riverbed rather than just a quote from an e-mail Exactly! That's what I was after. Plus, maybe a little more technical information as to how the SSL implementation is better than OpenSSL. In our company, we are going to replace the ssl certs on every system that was potentially open to the exploit, so it makes a huge difference (regarding  both budget and work) to know that the web sites behind our Zeus/STingray have not been vulnerable. It seems as if we all need to contact the support individually to receive the desired official statement - at least that's what I will do. ... View more

Actually this reply is not so helpful. The test site is overloaded and does not respond (currently), not to mention that we don't even know whether it is trustworthy and correct. Just because it is on Github and has a nice "official" heartbleed logo  doesn't mean anything. Given the severity of the issue, I would expect some sort of official statement regarding the products. In my case, I am running Stingray on my own hardware running on Debian Squeeze. Openssl is at 0.9.8*, so not vulnerable. But since I don't know the internals of Stingray, I would appreciate a more concise explanation whether or not services are vulnerable, and whether it depends on the Version of OpenSSL in the underlying OS. Regards, Ulrich ... View more

A great workaround - you pretty much nailed every aspect : custom DNS server, short timeout, caching  - wow!  While I still don't feel comfortable with adding so much additional overhead for every request,   I might be forced to try it if them bots continue to get on my nerves as much as they currently do ...   Thanks for this instructive and informative gem of insider wisdom ... View more

Thanks for this handy article - it makes a helpful starting point. Anybody using this recipe in a copy-and-paste manner should replace zen.spamhaus.org with xbl.spamhaus.org to avoid blocking of  users with residential IPs. Like you say this solution might become a performance bottleneck if the DNS lookups are too slow ... Actually, DNS Usage for BL lookups is quite different from normal lookups so for me it feels a bit risky to make all these changes (i.e. running a DNS vhost , making the dns cache 10 times as high, minimize the dns lookup timeout etc.) because it would mean to "bend" the global DNS settings only for this single  purpose.  In our network, we are already running a dedicated fast local Blacklist resolver where we combine the blacklist  data of several sources , but I don't see how I could use it from Stingray since that requires the configuration of a separate resolver IP just for this purpose.  Ideally, the dns resolver functions in Trafficscript would have parameters for both dnsserver and timeout. While the first might be a little bit far-fetched and difficult to implement , the timeout parameter would be a very useful feature  - not only for BL checks but for other use cases of dns lookups, too. Actually any network-related function should have an optional  timeout parameter ... Regards, Ulrich (PS: there is a little typo in line 9 using + instead of . for string concat) ... View more

Thanks Owen for your fast and comprehensive reply. I wasn't aware of the local DNS cache, so this might be worth a try. On the technical side, I will comment on your solution in the context of the article you wrote. Thanks and best regards, Ulrich ... View more

Just a short follow-up  for those who may find this thread  later by looking for the "timing out new connection"  message: Like I said I had opened a support case to find out what was going on. Eventually, I received the following explanation which seems logical to me: -- Quoted from Support message - The error messages you are seeing normally doesn't' indicate any problem, I will explain why. Take this one for example: 30/Mar/2012:15:37:06 +0200]    INFO    vservers/minisites      connerror       "Timing out new connection" 78.34.110.99:49219 78.35.5.81:80 "-" "-" - "-" R 0 0 0 0 11 12 0 0 - - - This shows that the client establishes a TCP connection (after 3-way-handshake), and then don't send any request on that established connection, the connection then is closed after the timeout is hit. The 3 consecutive "-" normally shows the data sent from/to client and back-end but here they are empty. One reason for such connections is the way Google Chrome works, which establishes two connections every time a user tries to request a URL. This is a browser feature just to speed up the process, i.e. if the first tcp connection is lost the second is tried to send a request to the server, and if one of the connections succeeds the second is ignored and hence on the server-side the connection times out. You can search this on the internet, just an example see <a target="_blank" href="http://stackoverflow.com/questions/4460661/what-to-do-with-chrome-sending-extra-requests.">http://stackoverflow.com/questions/4460661/what-to-do-with-chrome-sending-extra-requests.</a> This was just one example that I'm aware of that is causing these errors, but it could be some<br>other client using same technique or could be a buggy client. -- end of  quote - This would explain the constant ratio between hits and error messages, and it also clarifies the reason what the log message means.  My personal conclusion is to ignore theses messages in the future, like we all ignore those "connection  reset by peer" and others. And BTW the support guys also recommended to enable the 'log!client_connection_failures' only temporarily for debugging. Regards, Ulrich ... View more

Michael,   I have the same problem on my site (i.e. lots and lots of  "Timing out new connection" errors) ;  actually I am not sure whether it is a real problem or some logging artifact  that can be ignored.   This btw is a good question for a community site, so why don't we compare our numbers?   In my case I have  a ratio between 1 : 30-50  i.e.for every 30-50 requests on a particular vhost on my Trafficmanager, I get 1 "timing out" error.    What are your numbers? Everybody else reading this - can you compare and tell us your findings? Enable logging temporarily in Virtual Servers > SERVERNAME > Connection Management >Connection Error Settings> log!client_connection_failures     My sites are primarily B2C sites (news, entertainment) so I have lots of residential users with dialup/DSL ; further research has shown that most of the logged IPs had hostnames like dsl-something, cable-something etc., sounding like they were used for residential access.   Over the years, I have learned to safely ignore all those zillions of "connection reset by peer" errors found in the log of every busy webserver; that's something that just  happens since you can't expect that each and every connection can be handled correctly.  ""Timing out new connection"",  is obviously a Zeus-specific message, so I am lacking this experience.   I have opened  a support case for this issue and so far, they recommended to check out the listen queue size and other tcp parameters as described in  <a target="_blank" href="http://blogs.riverbed.com/stingray/2005/09/tuning-zeus-traffic-manager-for-maximum-performance.html">http://blogs.riverbed.com/stingray/2005/09/tuning-zeus-traffic-manager-for-maximum-performance.html</a>   That , however, has had no effect on the number of error messages. >>So in your case, you have a client read which sent no data in any direction because it timed out.   Sounds like Chris is assuming a generic network related communication problem.  If I knew that other sites have the same percentage of log entries, I would think that they probably can be ignored.   Regards, Ulrich   ... View more