Release Meeting: 27 Nov 2017

 (08:00:29 AM) gac410: hi all. Are we having a release meeting today?
(08:12:56 AM) MichaelDaum: Hi George. Sorry for being late.
(08:13:20 AM) gac410: no problem... I'm not all here yet either. Still waiting for coffee to digest.
(08:13:42 AM) vrurg: Hi everyone
(08:13:53 AM) gac410: hi vrurg
(08:14:44 AM) gac410: Michael, I've been working on an alternate nginx config for a new server I'm building. I think it solves some issues that the older examples have: https://pastebin.com/1W92GyGW
(08:15:09 AM) MichaelDaum: which issues are these?
(08:15:19 AM) MichaelDaum: Hey vrurg
(08:16:12 AM) gac410: Can't redirect bin/view/My/Topic => /MyTopic, and also rewrite /My/Topic to /bin/view/MyTopic ...
(08:16:20 AM) gac410: Also the [A-Z] match
(08:17:39 AM) gac410: Also i found https://github.com/oohnoitz/nginx-blacklist/blob/master/blacklist.conf as an alternative to our list of blocked agent strings in ApacheConfigGenerator. Better than a big IF statement.
(08:20:08 AM) MichaelDaum: I am using this one https://github.com/Stevie-Ray/referrer-spam-blocker
(08:20:42 AM) MichaelDaum: comes with files for nginx as well as apache
(08:22:51 AM) gac410: hm That one misses SemrushBot too :( But it's a much bigger list than the one I found.
(08:25:41 AM) gac410: Anyway, release meeting. PatchReleaseBlockers. http://foswiki.org/Tasks/Item14494 - PlainFileStore doesn't honor the "ProjectContributor" meta we build in distribution topics.
(08:26:07 AM) gac410: Embedded META is ignored if no history exists.
(08:27:01 AM) gac410: Also fairly new blocker: http://foswiki.org/Tasks/Item14516 - %FOO breaks EDITTABLE
(08:30:19 AM) MichaelDaum: %META:TOPICINFO is ignored?
(08:31:05 AM) gac410: y At least the Author is ignored. I noticed that on a fresh new install with PFS, all System topics show "UnknownUser" as the author.
(08:31:22 AM) MichaelDaum: :(
(08:31:23 AM) gac410: Switch to RcsStore and they become ProjectContributor
(08:32:22 AM) gac410: I thought I had broken something and eventually figured out it's been that way for a while.
(08:32:40 AM) MichaelDaum: it isn't even BaseUserMapping_111 ? ... the cuid for PJs
(08:33:27 AM) gac410: we've always set META to "ProjectContributor" ... That's been enforced since SVN days.
(08:33:34 AM) MichaelDaum: y
(08:33:47 AM) gac410: And that works fine on RCS store
(08:36:22 AM) gac410: BTW... we seem to be gaining a new developer. WillLe has been submitting some pull requests. Though his change - adding a cfg{RootDir} setting uinfying all our path settings needs some review.
(08:37:09 AM) MichaelDaum: good thing
(08:38:11 AM) gac410: I merged his changes into Item14530 branch. Though I've got another pull request to merge.
(08:39:07 AM) gac410: Any other Tasks to discuss?
(08:40:49 AM) gac410: vrurg fixed the EditRowPlugin "jumping rows" bug - https://foswiki.org/Tasks/Item14537 but needs to be committed.
(08:42:12 AM) vrurg: gac410: The fix is not totally clean. While rows remain steady by pointer is flickering when put over the drag button. Have neither idea nor any more time to find out what's causing it.
(08:42:26 AM) vrurg: s/by/but/
(08:43:19 AM) gac410: On the RootDir setting - I'm wondering why we never did that before. It seems so obvious. Is there some case where we actually bootstrap/detect directories in other paths? (like a cgi-bin directory?)
(08:44:47 AM) gac410: vrurg: unfortunately css and js fixes probably need Michael or cdot to "bless" ... I'm not all that good with it.
(08:46:30 AM) gac410: Any other tasks?
(08:46:53 AM) vrurg: Especially considering that EditRow is cdot's child.
(08:47:18 AM) MichaelDaum: will we ever release a 2.1.5?
(08:47:33 AM) MichaelDaum: there are some tasks ready to go for it
(08:47:44 AM) gac410: Sure. I've been waiting for a fix bad enough to trigger a release.
(08:47:46 AM) MichaelDaum: as far as I remember
(08:47:57 AM) MichaelDaum: Foswiki::Aux:: is bad enuf
(08:48:19 AM) gac410: The list is pretty short still I think Windows :P
(08:48:43 AM) MichaelDaum: for now there is no way for windies to run foswiki.
(08:49:05 AM) gac410: 13 fixes and 3 enhancements are waiting for release.
(08:49:19 AM) MichaelDaum: when they go back to a release before Aux has been introduced they get security problems fixed in 2.1.[34]
(08:49:30 AM) gac410: er... 11 fixes. Two release blockers.
(08:49:45 AM) MichaelDaum: all previous foswiki releases have problems with latest perls
(08:50:06 AM) MichaelDaum: however you do need a newer perl as per CVE-2017-12883 and CVE-2017-12837
(08:50:14 AM) gac410: The two blockers are opened by you Michael - The Foobar and Password Protected Links.
(08:50:17 AM) vrurg: gac410: Didn't you renabme Aux yet?
(08:50:27 AM) gac410: Yes it's reneamed, but we need to release ;)
(08:50:34 AM) vrurg: Ah, ok. ;)
(08:51:36 AM) gac410: Do we want to fix http://foswiki.org/Tasks/Item14516, http://foswiki.org/Tasks/Item14445 ? or do we lower them from urgent?
(08:52:24 AM) gac410: I'd also like to merge http://foswiki.org/Tasks/Item14532 - process name override for FCGI - a minor enhancement.
(08:54:28 AM) MichaelDaum: the FCGI enhancement should also be applied to WebDAVContrib's and VirtualHostingContrib FCGIs ... not core, but imho will benefit from the same patch.
(08:55:41 AM) MichaelDaum: wrt Item14445: I really would like to see somebody else commenting on the findings.
(08:57:00 AM) gac410: Anyway, I'm willing to work on a 2.1.5. On 14445, maybe that should be a minor rather than a patch. I just don't know the auth side of things well enough. It does seem like a good change, but ...
(08:57:05 AM) vrurg: As to 401: can't it be checked against user agent and returned 200 for Excel specifically?
(08:57:21 AM) gac410: ugh. do we really want to go there?
(08:57:39 AM) ***MichaelDaum suffering from a leaf blower down the road ....
(08:59:05 AM) vrurg: gac410: I don't see why it should be against standards only because of a Microsoft's bug? I still remember the times of the old IE which sucked a lot of blood from developers... :)
(08:59:06 AM) MichaelDaum: vrurg, did you read the last discussion logged on the task?
(08:59:59 AM) vrurg: Not the IRC part yet. Reading.
(09:00:18 AM) gac410: vrurg, I don't think it's against standards. The Template Login page deserves a 200. This actually fixes another browser bug. Some platforms detect a 401 and prompt for a user/pass, which then fails, *then* they put up the template login.
(09:00:28 AM) gac410: Dolphin on android ...
(09:01:36 AM) gac410: 401 says you must include an authenticated header. It's telling the *browser* to authenticate. which it cannot do with template auth.
(09:03:27 AM) gac410: From RFC7235: A 401 (Unauthorized) response message is used by an origin server to challenge the authorization of a user agent, including a WWW-Authenticate header field containing at least one challenge applicable to the requested resource.
(09:03:55 AM) gac410: Our template auth does not include a WWW-Authenticate header, and is not an "agent" challenge, but a direct user challenge.
(09:04:12 AM) MichaelDaum: right ...we don't want that using TemplateLogin
(09:04:58 AM) gac410: So we just have to make sure that we only generate a 200 for templateLogin. ApacheLogin still needs to do the 401, (As do rest and jsonrpc)
(09:06:05 AM) gac410: Though I'm not convinced that 401 is valid for rest/json either.
(09:06:20 AM) vrurg: I see the point now. Ok, sorry, was wrong.
(09:06:52 AM) gac410: No. please don't apologize. This is a change in a very critical area. We don't want to get it wrong.
(09:07:47 AM) MichaelDaum: LdapContrib's KerberosLogin will need to change accordingly ... causing backwards compatibility problems :(
(09:08:22 AM) gac410: Does KerberosLogin use a WWW-Authenticate header?
(09:09:39 AM) MichaelDaum: yes
(09:09:55 AM) MichaelDaum: ... negotiate
(09:09:59 AM) gac410: If there is a backwards compat issue, then maybe we need a configuration switch to restore the old behavior.
(09:11:05 AM) MichaelDaum: hm, this is putting users in a bad situation as the situation is broken in two way differently no matter what the switch is set to
(09:11:59 AM) MichaelDaum: I'd rather tell people to upgrade both
(09:12:20 AM) gac410: MichaelDaum: Can you check in a fix to Release02x01? Y if an updated contrib is available, that's the way to go.
(09:12:36 AM) MichaelDaum: okay, will do
(09:13:00 AM) gac410: If you can, a good description to the ReleaseNotes as well. What to look for if things stop working.
(09:14:19 AM) gac410: Is this mainly a on-liner for TemplateLogin.pm or is there more to it?
(09:16:00 AM) gac410: Hm... looking at the original changeset, we do include a custom WWW-Authenticate header "for use by Javascript".
(09:16:03 AM) MichaelDaum: yes: it deletes the part setting the 401 header
(09:16:21 AM) MichaelDaum: as well as the WWW-Authenticate part
(09:17:10 AM) gac410: can those parts be moved into REST and JSONRPC ie any action interacting with javascript might be able to process the header.
(09:19:48 AM) gac410: Interesting. I saw the block in LoginManager.pm when I was immplemeting the new Token authentication. the X-Foswiki authentication headers. But didn't connect that this was all related.
(09:21:22 AM) gac410: Speaking of Token auth. If anyone could checkout Item14506 branch and try my new ResetPassword code, that would be helpful. This is ready to merge into master AFAIK
(09:21:32 AM) MichaelDaum: as I said last time already: UI::Rest already triggers a 401 on its own ...
(09:21:49 AM) gac410: Yup... I thought you said that :D
(09:22:26 AM) MichaelDaum: meanwhile I've tried to debug Item14516
(09:22:59 AM) gac410: hm Rest.pm only throws a 401 exception. Should it generate the WWW-Authenticate header like TemplateLogin does?
(09:31:17 AM) gac410: New task popped up. https://foswiki.org/Tasks/Item14544 Probably urgent.
(09:37:45 AM) vrurg: Looks bad. Even though the situation is very rare.
(09:38:43 AM) gac410: y. I *partially* addresses this in bootstrap. We use FORWARDED_HOST (without spliiting it) for setting the defaultUrlHost. So that's part one of the bug.
(09:39:17 AM) gac410: But we should also detect it dynamically in SCRIPTURL* macros
(09:43:13 AM) vrurg: I can't find any standard where multiple hosts format would be defined in x-forwarded-host. Is it actually allowed?
(09:43:51 AM) gac410: It's definitely allowed in x-forwarded-for ... a list of IP addresses.
(09:44:21 AM) gac410: We have a config setting $Foswiki::cfg{PROXY}{UseForwardedForHeader} ... we probably need an equivalent UseForwardedHostHeader.
(09:44:36 AM) gac410: Similar to the ForceDefaultUrlHost
(09:46:21 AM) vrurg: Before starting to fix a probably non-existing problem, shouldn't we ask the reporter to post the headers his server receives?
(09:46:26 AM) gac410: See http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#x-headers ....
(09:46:44 AM) ***vrurg is on the very same page now. Reading...
(09:46:45 AM) gac410: Says be careful using any of the Forwarded-* headers, as they can contain a list of proxies
(09:47:39 AM) vrurg: "The original host requested by the client in the Host HTTP request header." – "host" is in singular.
(09:48:09 AM) vrurg: Oops, I'm still sleepy today. ;)
(09:51:07 AM) MichaelDaum: _any_ table parsed by Foswiki::Tables::Parser will suffer from Item14516 ... i.e. DataForm tables
(09:51:51 AM) gac410: yikes
(09:53:08 AM) MichaelDaum: maybe we can warm cdot to have a look at the parser
(09:54:34 AM) gac410: That would be nice. and if he could review vrurgs js/css fixes for ERP ... and maybe the PFS ignoring ProjectContributor that would be nice too :D
(09:54:38 AM) MichaelDaum: hey guys, I have to run and get some food.
(09:54:45 AM) gac410: okay MichaelDaum Thanks.
(09:54:54 AM) MichaelDaum: back in an hour or so
(09:55:03 AM) MichaelDaum is now known as MichaelDaum_
(09:55:05 AM) gac410: sounds good Eat well
(09:55:21 AM) vrurg: MichaelDaum_: thanks a cu
(09:56:09 AM) vrurg: gac410: It looks to me that not only X-Forwarded-Host by X-Forwarded-Port must be considered for building correct URL.
(09:56:35 AM) gac410: y. I agree.
(09:59:41 AM) vrurg: From what I see in the standard, correctly working reverser proxies / balancers would generate all X-Forwarded- in a list format. Perhaps a routine needed which would parse them and produce a list of hashes which could be kept on the Request object and be used anywhere in the code.
(10:02:09 AM) gac410: The engines all pull the originating IP from the X-Forwarded-For header ... if enabled by configuration. Bootstrap uses X-Forwarded-Host, but without splitting it. Foswiki.pm *should* use them when establishing defaults for getScriptUrl
(10:03:51 AM) gac410: (The engine changes for X-Forwarded-For are not in Release02x01 branch.
(10:05:34 AM) gac410: So... Is this a feature (defer to 2.2) or a bugfix ... Actually it's a feature, but incomplete on Master Item14380:  New feature - Support for reversy proxies
(10:09:03 AM) vrurg: My point here is that if something is used in more than one location (bootstrap, SCRIPTURL in this case) – make a routine, cache the result. Some plugin might also find this info useful for a purpose I cannot imagine right now. Having a complete Request object sounds like a good option to me.
(10:09:46 AM) gac410: Yes I agree.
(10:10:14 AM) vrurg: This way it makes it look more like a new feature.
(10:11:12 AM) vrurg: I will write a comment on Item14380.
(10:11:41 AM) gac410: I think the Request object already processes the headers correctly. Not Foswiki code though, but CGI:: inherited object.
(10:13:53 AM) vrurg: No, it doesn't. The first notion of X-Forwarded-Host is line 313 where the header value is used as-is to form a URL.
(10:14:48 AM) vrurg: In a chained-proxy env it will produce something like http://host1.domain,host2.domain,etc.domain
(10:15:06 AM) vrurg: Or even with spaces after the commas, as the standard suggests.
(10:15:30 AM) gac410: Line 313 of what?
(10:16:25 AM) vrurg: Of Request.pm
(10:18:34 AM) vrurg: Foswiki::Engine::Apache handles X-Forwarded-For correctly by splitting it with /, /.
(10:18:53 AM) gac410: CGI.pm has a method virtual_host which splits and uses X-Forwarded-Host header
(10:19:35 AM) vrurg: We're planning to get rid of CGI.pm, aren't we? Not in v2, of course, but still...
(10:20:13 AM) gac410: But it doesn't have a virtual_addr equivalent of remote_addr to split the Forwarded-For
(10:20:34 AM) gac410: y I agree. Just trying to refresh where I've found some of this stuff.
(10:21:04 AM) gac410: virtual_host is really a misnomer ... I don't think of it as a proxy host,
(10:21:32 AM) vrurg: Looks to me like it worth a new proposal.
(10:22:14 AM) gac410: and CGI's virtual_host is wrong anyway. It uses the Forwarded-Host header without splittting Though it does discard a :port# if present
(10:24:02 AM) gac410: I don't think we need a feature proposal. It's really just bugfix, though somewhat extensive
(10:24:54 AM) vrurg: Ok, do you want me to leave a comment on the task to summarize a little bit of what we discussed today?
(10:25:42 AM) gac410: sure.
(10:26:13 AM) gac410: My thoughs are that the engine should really do the "setting" of the data in the request. The request object needs accessors, but should not be parsing the headers.
(10:26:39 AM) gac410: ie right now Request has a remote_addr() method, which returns the data set by the engine.
(10:27:17 AM) vrurg: I'll note this.
(10:27:33 AM) vrurg: Time to get some breakfast finally. ;) Thanks and cu later!
(10:27:40 AM) gac410: Actually it's a setter/getter. But Engine is what sets it.
(10:29:18 AM) gac410: ugh And Request->url() uses Forwarded-host, but gets it wrong. Doesn't split it, and doesn't address the Forwarded-Port
(10:31:39 AM) gac410: Okay. This is clearly bugfix territory, not a feature. Request.pm supported (incorrectly) Forwarded-host since Item175 .. the TWiki import.