Item5960: Content-length breaks HTTP headers, a.o. pound and chrome fail results

There seems to be a problem with the code that 'hacks' a Content-length header into the HTTP response. It sort of appends it, like this (from TWiki.pm, line 602 in 4.2.2 tarball):

602 my $len = length($text);

603 $hdr =~ s/\n$/Content-Length: $len\n\n/s if $len;

However, this does not really work, since headers are \r\n delimited, so the result is that you get something like:

Content-Type: text/html; charset=iso-8859-1\r\n\rContent-Length: 1234

Ie.: badness!! HTTP fail!! On multiple levels!!

My quick fix has been:

602 my $len = length($text);

603 $hdr =~ s/\r\n$/Content-Length: $len\n\n/s if $len;

That's not the end of it though. Section 4.4 of RFC 2616 (that's the HTTP/1.1 rfc) states:

Messages MUST NOT include both a Content-Length header field and a
non-identity transfer-coding. If the message does include a non-
identity transfer-coding, the Content-Length MUST be ignored.

Now, what i'm seeing is that twiki responses have a 'chunked' transfer-coding, thus the Content-length field MUST NOT be there (in fact, having both headers present can lead to all sorts of security implications, opening up avenues to poison caches etc..).

Why I care about all this? Because twiki currently breaks when put behind pound (a load balancer / SSL endpoint tool).

Lesson learned: don't mess with HTTP headers if you're not sure what you're doing.

What is beyond me, by the way, is that one of my twiki instances does not generate a Content-length header, whereas the other does. There are numerous differences between the various installs (the former is on an apache with mod_perl installed, though not used with twiki, the latter is on an apache with chrooted fastcgi, etc..). The differences are numerous, and it's late so i'm not very inclined to get to the bottom of this. Perhaps one of the TWiki guru's can shed some light on this peculiar bit of code, and on why the Content-length header is present at all.

(ps: as you might know, i'm quite out of the twiki scene at the moment, and i intend to keep it that way. if anyone cares to communicate with me about this bug, please contact me on irc: gmc on the freenode network, or send me an email, as i'm not likely to check the bug database anytime soon).

-- TWiki:Main/KoenMartens - 25 Aug 2008

Note, btw, that 'MUST NOT' means that it is an absolute prohibition, ie: twiki is non-RFC compliant.

-- TWiki:Main.KoenMartens - 25 Aug 2008

Need to address this at release meeting. Or better. Here in the bug item. I think Koen needs feedback from the experts.

-- TWiki:Main.KennethLavrsen - 18 Sep 2008

I discussed with the experts on irc already, we concluded that it needs further investigating: while it is twiki that sets the Content-Length, it should only do so if the Content-Encoding is identity. However, this is not decided by TWiki and (as a preliminary scan revealed) also not by the CGI module. I suspect it might be as high up as the webserver (which would make sense). What needs to be investigated is when and why the Content-Encoding is set and whether twiki can detect this or maybe force the issue somehow.

-- TWiki:Main.KoenMartens - 19 Sep 2008

I've applied the quick fix to the 4.2 branch as someone else just reported the problem to me.

-- SvenDowideit - 14 Oct 2008

AFAIK the rationale for the Content-length header was "performance" - letting the browser know how much content is coming. From my own tests, I can't see that it makes a blind bit of difference - though that may be different with another server/browser combination (I tested on Apache 2 + FF 3). My vote is to drop the content-length header completely.

I see that Sven checked a fix into the release branch but not into trunk. Why?

-- TWiki:Main.CrawfordCurrie - 24 Oct 2008

iirc, its because this code is no longer present in trunk in this form - but things were moved around, so i may well have just failed to find them.

-- TWiki:Main.SvenDowideit - 24 Oct 2008

In trunk this kind of thing is responsibility of each engine. I also plan to move things like generateHTTPHeaders to runtime engines. Core code shouldn't worry about this. IIRC CGI engine (and others) generate the header using TWiki::Engine->CRLF.

-- TWiki:Main.GilmarSantosJr - 24 Oct 2008

FWIW, this bugs also breaks the software for google chrome...

-- KoenMartens - 17 Nov 2008 - 20:45

gmc: we are undecided whether to go with the proposed patch or whethet ro drop content-length completely
[3:49pm] gmc: i still have research the implications (rfc wise) of the latter proposal

---

Changed to "Being Worked On" because i believe Koen does actually have this on his queue, and it needs pinging anyway

-- CrawfordCurrie - 12 Dec 2008

Koen there are days not weeks till release. Are you fixing this one?

-- KennethLavrsen - 16 Dec 2008

Koen is not doing anything on this it seems. So I am putting it back in New. Please we need help on this one. Otherwise I will downgrade it to low because if a bug can be open from August till now - it cannot be urgent.

• KennethLavrsen - 24 Dec 2008

Per Koen and Stephane's advice lowering to Normal

• KennethLavrsen - 24 Dec 2008

Sopan has worked on this and closed the TWiki version of this bug.

Who is analysing his work? Who is applying a fix if needed?

-- KennethLavrsen - 21 Jan 2009

Sopan credits "Sven" for a fix - I assume it's SvenDowideit. He has coded in a content-length, but when I look at our code I see no equivalent.

We only added the content-length to overcome problems with certain older browsers (IE). Since we don't appear to generate a content-length any more, I'm assuming we have found our own way to resolve this.

Setting waiting for feedback from SvenDowideit, since he is implicated, and Gilmar, because the FSA code is likely to have an impact here.

-- CrawfordCurrie - 04 Mar 2009

yeah, sure - I did the first 2 commits to this bug - a line feed issue - though really, this should be clear by reading this Task log.

-- SvenDowideit - 04 Mar 2009

FSA fixed the original issue naturally (the problem with \r\n and wrong values), but it needs testing. I'm also working on Item1089 and this issue will be verified.

-- GilmarSantosJr - 19 Apr 2009

Closing this task. Ancient history, and we don't seem to have any issues with content length. Reopen if there is still an issue.

-- Main.GeorgeClark - 23 Jun 2015 - 23:34