Dealing with a too big POST'd Content-Length

[sameh]

Hello,

The most common request to time out on my cluster is linked to a POST'd form.

The client sends POST request headers, including a Content-Length > 0.

Then, body is either missing, or smaller than advertised CL.

For the record I have no idea why exactly, I suspect this is a browser bug (the request is sent using AJAX in awkward conditions) where the window is destroyed while the request is sent.

Because ZXT^WStingrayTM forwards the POST to a node before receiving the body, the answer will trigger a timeout (Apache backend does not even know the request has been fully sent, it is still waiting for the full body).

I currently have no idea how to get rid of the "Timing out connection (increasing the 'timeout' setting on the virtual server's 'Connection Managem... message.

To at least keep less connections busy I tried to use request.setMaxReplyTime(5); because I know it is supposed to be answered very quickly, but the new timeout does not change anything (timeout still occurs after default timeout setting).

I could detect the lack of POST body at the TrafficScript level I guess, but I am not sure this is the most graceful way dealing with this.

Is there some config option somewhere that could help me?

Cheers,

Sameh.

[sameh]

To summarize the issue:

1) I would like POST request handling to deal with non-existent bodies (a different timeout setting?).

2) request.setMaxReplyTime() is not working in this case.

[aannavarapu]

Hi Sameh

There is a setting in the TM instance that you can try and tweak for the pool in question.

The setting is under Services--> Pools --> <the pool name in question> --> Connection Management. There is a setting there with a default of 30s wait for a reply from a node.

Can you check that if you haven't already done so?

Regards,

Arun

[sameh]

Hello Arun,

It's set at 35s currently. But I want to override this setting only for specific requests.

request.setMaxReplyTime() would be great if it worked .

[aannavarapu]

Hi Sameh

Makes sense to do that for specific requests. Can you please share the traffic script that you are using for this purpose? I can try and suggest something based on that !

Regards

Arun

[jsteele · Accepted solution]

Addressing your summary statements (in reverse order):

2) request.setMaxReplyTime() is not working in this case.

You are correct in determining that this will not work in this case.  This timeout is specifically for the response from the server; in your case the request to the server is still on-going from Stingray's point of view (which is waiting for the client to finish POST-ing all the data it has indicated it is intending to do).

1) I would like POST request handling to deal with non-existent bodies (a different timeout setting?).

There isn't a specific timeout setting for empty body POST requests, but there are two other timeout settings relevant beyond the max_reply_time for the pool.  These are associated with the virtual server settings (as we are considering the behaviour of the request from a client), under the Timeout Settings of the Connection Management section:

• timeout - An inactivity timeout for data sent/received on a connection from the client to the virtual server.
• max_transaction_duration (since v9.6) - A global timer on the lifetime of a transaction (such as a HTTP request and associated response).

Both can be set on a per request basis through TrafficScript (request.setVirtualServerTimeout and request.setMaxTransactionDuration).  For your case, the "timeout" setting is probably the most appropriate (if the client isn't really sending any data).  If the client is "drip-feeding" data for the POST then max_transaction_duration might be a more suitable option.  If the amount of data a client can POST can vary, you should consider taking the amount of data into account when calculating the timeouts dynamically in a rule.

The log message you describe indicates you are probably hitting the "timeout" timeout - all these timeouts will generate a log message when connection failure logging is enabled on the virtual server.

[mikeg_2]

If the client sends a POST and a Content-Length, but then never sends any data, you can intercept that in a request rule like this:

if( http.getMethod() == "POST" ) {

   $ct_len = http.getHeader( "Content-Length" );

   if( $ct_len > 0 ) {

      http.getBody( $ct_len < 1024 ? $ct_len: 1024 );

   }

}

Note that the amount of data to buffer in the rule is limited to 1kB to avoid running out of memory.  If the client never sends any data, this will timeout during rule processing, and will therefore never use up precious connections to the web servers.  If the client announces more than 1k, but sends just 1k, this rule wouldn't catch that, so it's probably best to combine this with the use of request.setMaxTransactionDuration() as suggested by James.

[sameh]

Thank you for your wise answers.

Indeed request.setMaxReplyTime() shouldn't work in this scenario,  I have been tricked by the TrafficScript reference:

"request.setMaxReplyTime( seconds )

Overrides the pool's max reply time for this request, also overriding the virtual server timeout if necessary."

Seems like it didn't deem override the VS timeout necessary . Not sure if the reference is wrong, if this is a bug, or if my understanding is to blame.

I am going to use request.setVirtualServerTimeout and request.setMaxTransactionDuration in the smart way you pointed out.

[sameh]

Hi Michael, thank you, I will definitely implement this and combine it with James' proposal.

I would love to be able to remove the verbose logging of those VS timeouts for those requests only but I don't see any way to do it.