Release Meeting: 4 May 2015

(08:57:44 AM) kornbluth.freenode.net: (notice) [freenode-info] channel flooding and no channel staff around to help? Please check with freenode support: http://freenode.net/faq.shtml#gettinghelp
(09:00:30 AM) MichaelDaum entered the room.
(09:00:30 AM) MichaelDaum left the room (quit: Changing host).
(09:00:30 AM) MichaelDaum entered the room.
(09:03:03 AM) gac410: Hi everyone ...   
(09:03:24 AM) andrel entered the room.
(09:04:21 AM) MichaelDaum: Hey there
(09:05:27 AM) andreli: Hello
(09:05:31 AM) ***gac410 was hoping that CDot would make it.    He seems to know how to fix our blocker
(09:05:47 AM) gac410: hi andreli
(09:06:29 AM) gac410: Anyone have anything for the agenda other than our blocker  http://foswiki.org/Tasks/Item13378     agenda is at http://foswiki.org/Development/ReleaseMeeting01x02_20150504
(09:07:29 AM) MichaelDaum: let me check ... doing nasty system upgrades atm
(09:08:19 AM) gac410: CDot seems to think that any charset *except* utf8  is badly broken.      Issues are that JSON  and possibly CGI functions return utf8 strings.
(09:09:19 AM) gac410: The issue was found by Jomo.    Some Czech cause very broken rendering,    seems that the root cause, though, is the SubscribePlugin 
(09:09:31 AM) gac410: er  Czech characters 
(09:10:34 AM) MichaelDaum: why is SubscribePlugin the culprit
(09:11:16 AM) gac410: It embeds some JSON which is utf8 into the rendered output  which triggers certain double-byte characters to match wikiword 
(09:12:39 AM) MichaelDaum: this should be the same JSON that is being used in JsonRpcContrib
(09:12:55 AM) MichaelDaum: he also says "So long as only 7-bit characters are used, this is not a problem, but as soon as wide-byte characters are involved, it all falls apart."
(09:13:02 AM) gac410: yes.  But jsonrpc all uses ajax,  which seems to be already handled correctly
(09:13:48 AM) gac410: I had a one-line fix to SubscribePlugin which fixes the problem,  but CDot says it's not right.    Added a call:
(09:13:50 AM) gac410: +    $json = Encode::encode( $Foswiki::cfg{Site}{CharSet}, $json, Encode::FB_CROAK );
(09:13:58 AM) gac410: to the returned JSON data
(09:15:01 AM) MichaelDaum: SubscribePlugin seen that
(09:15:33 AM) MichaelDaum: it now makes use of Foswiki::UTF82SiteCharSet()
(09:15:40 AM) gac410: Which doesn't help
(09:15:51 AM) MichaelDaum: ... a pile of ancient utf8 wanna-be code
(09:16:08 AM) gac410: The problem is on sites already running UTF8    so the call to UTF82SiteCharSet   is a NOP
(09:17:04 AM) MichaelDaum: the $json obj is also passed through HTML::encode_entities ...AFTER being converted to site-charset 
(09:17:25 AM) MichaelDaum: I bet HTML::encode_entities re-introduces some more utf8 chars
(09:17:30 AM) JulianLevens entered the room.
(09:17:55 AM) MichaelDaum: see lines 130ff
(09:18:08 AM) gac410: All I can say is my patch  in http://foswiki.org/Tasks/Item13378     fixes it, and was confirmed by jomo
(09:18:32 AM) MichaelDaum: I don't see why HTML::encode_entities is required
(09:18:58 AM) ***gac410 doesn't understand it well enough    It's all black magic to me  :P
(09:19:22 AM) MichaelDaum: this %JSON_DATA% blob is used in attr position as can be see in subscribe.tmpl
(09:19:52 AM) gac410: yeah.    Thats the pollution to the page
(09:19:54 AM) MichaelDaum: this all is programmed in a bit nasty way
(09:20:20 AM) MichaelDaum: SubscribePlugin doesn't make use of html5 data attributes properly
(09:21:13 AM) MichaelDaum: it shouldn't stuff a json obj into an html attribute
(09:21:19 AM) MichaelDaum: this is a misconception
(09:21:28 AM) gac410: My concern is that CDot claims that it it's badly broken for ALL non-UTF8  site charactersets.      due to CGI
(09:22:00 AM) MichaelDaum: instead all data-* attributes are read by var json = $(elem).data(); ... this is when a json obj is generated ... not on the server side
(09:22:28 AM) andreli: You discussion sounds really scaring. I would ask: Are there tests and do they pass? If so, keep going.
(09:22:49 AM) gac410: Most of our tests don't really test multiple character sets 
(09:23:14 AM) MichaelDaum: the markup should be <a ... data-subscribe-topic="..." data-subscriber="..." datga-remove="..." data-validation-key="..." .... >%MAKETEXT{...}%</a> in subscribe.tmpl
(09:23:26 AM) MichaelDaum: andreli, naw
(09:23:41 AM) gac410: The issue MichaelDaum is that SubscribePlugin is used in both traditional <a href=...>   links and also in ajax based POST calls.
(09:23:51 AM) MichaelDaum: right
(09:24:12 AM) MichaelDaum: yet still the %SUSCRIBE makro is doing it wrong
(09:24:39 AM) MichaelDaum: and that's where the JSON code is being called ... yet again filtered thru HTML::encode_entities ... which is a beast of its own
(09:25:05 AM) gac410: If you have a fix,  the http://foswiki.org/pub/Tasks/Item13331/formtest.tgz     contains a file that demonstrates the breakage.     I can recreate it here as well.
(09:26:04 AM) MichaelDaum: thats only a use case for displaying forms. how is that related to SubscribePlugin?
(09:26:19 AM) gac410: With SubscribePlugin active,   the form data is corrupted
(09:26:33 AM) MichaelDaum: okay let me check
(09:26:37 AM) gac410: foswiki renders wikiwords where none exist
(09:27:14 AM) gac410: Specifically,   the single character ъ   gets linked as a missing wikiword when in a formfield
(09:28:09 AM) gac410: It gets corrupted in formfields,  and also in the "description" field of attachments.
(09:28:48 AM) MichaelDaum: I still can't reproduce the error
(09:29:13 AM) MichaelDaum: it all renders fine over here
(09:29:29 AM) gac410: your Site CharSet is utf8?
(09:29:38 AM) MichaelDaum: y
(09:29:41 AM) gac410: Pattern skin,  SubscribePlugin active?
(09:29:43 AM) MichaelDaum: y
(09:30:07 AM) gac410: wtf  CDot, myself and jomo all have the problem.
(09:30:07 AM) MichaelDaum: CGI-4.15
(09:30:31 AM) MichaelDaum: if you switch off SubscribePlugin, does that fix the error?
(09:30:35 AM) gac410: Yes
(09:31:00 AM) gac410: So you have to be logged in as a user that would get the subscribe buttons rendered
(09:31:09 AM) MichaelDaum: I have it on my screen
(09:31:11 AM) gac410: It doesn't fail for wikiguest
(09:31:27 AM) MichaelDaum: admin user
(09:32:45 AM) MichaelDaum: Љъб&#345;з&#350;&#316;&#318;&#355;&#314; is not creating any sorts of linking
(09:32:54 AM) gac410: Fails for me.      
(09:33:21 AM) gac410: I view AaaTesting1   page     form looks fine.   Then login as JoeUser,   and it is corrupted
(09:33:33 AM) gac410: , Љъ&#65533;&#65533;&#345;з&#350;&#316;&#318;&#355;&#314;,
(09:33:57 AM) gac410: Log out and it returns to Љъб&#345;з&#350;&#316;&#318;&#355;&#314;, &#286;
(09:36:12 AM) gac410: $Foswiki::cfg{Site}{CharSet} = 'utf-8';    $Foswiki::cfg{Site}{Locale} = 'en_US.utf8';    $Foswiki::cfg{Site}{LocaleRegexes} = 1;   and $Foswiki::cfg{UseLocale} = 0;
(09:36:43 AM) gac410: With UseLocale disabled,   {Site}{Locale} is ignored,  but {Site}{LocaleRegexes}   is still active
(09:37:41 AM) gac410: Disabling {Site}{LocaleRegexes}  also fixes the issue,  as it changes wikiword matching to explicit  [A-Za-z] style regexes
(09:38:56 AM) JulianLevens: Doesn't that strictly speaking mask the issue
(09:39:03 AM) gac410: Exactly
(09:40:03 AM) JulianLevens: Is there any difference between between AdminUser and JoeUser?
(09:40:21 AM) gac410: There should not be... Let me switch ..
(09:41:11 AM) gac410: yeah  fails as "admin" as well.   The key is being logged in as a user that triggeres rendering of the Subscribe/Unsubscribe links
(09:41:28 AM) MichaelDaum: I disabled nearly all plugins ... only leaving SubscribePlugin enabled
(09:41:59 AM) JulianLevens: thanks, gac410, dumb question but wanted to be sure
(09:42:06 AM) MichaelDaum: only other difference to our setup might be store impl: I am still using Rcs
(09:42:58 AM) gac410: okay.   It still fails for me with ?debugenableplugins=SubscribePlugin,JQueryPlugin      and yes I'm on Rcs
(09:43:42 AM) gac410: RcsLite to be specific.    
(09:43:56 AM) MichaelDaum: RcsWrap over here
(09:44:02 AM) gac410: one moment...
(09:44:28 AM) MichaelDaum: FCGI, nginx
(09:44:51 AM) gac410: Ah.     Plain old CGI
(09:45:00 AM) MichaelDaum: page cache disabled?
(09:45:12 AM) gac410: Apache.     And still fails with RcsWrap.    Right, no page cache
(09:45:43 AM) gac410: Basically a completely default install - bootstrapped with ,v files present in the store.  
(09:45:57 AM) MichaelDaum: could you please enable FCGI?
(09:47:22 AM) gac410: ugh.   I need to go boot a vm and update foswiki to latest 
(09:48:51 AM) gac410: lemme see if my local apache fcgi will work ... been a long time.
(09:49:03 AM) MichaelDaum: anyway. I am going to fix SubscribePlugin to extract the brain-damage 
(09:49:46 AM) gac410: okay.   If you do that I can test.
(09:50:51 AM) gac410: My fcgi install on my laptop is not working.    
(09:53:28 AM) JulianLevens: FWIW my vanilla install with nginx and FCGI does *not* fail
(09:53:57 AM) gac410: interesting.   so same conditions as Michael.
(09:54:08 AM) JulianLevens: Alas, I cannot bring up an apache install quickly
(09:54:44 AM) gac410: And I'm in the same boat for an apache fcgi.  
(09:55:23 AM) JulianLevens: What I do not understand is that if SubscribePlugin via RPC is corrupting the data then how can Apache vs Nginx make any difference?
(09:55:31 AM) gac410: btw   I'm on CGI 3.65   
(09:55:41 AM) gac410: I can switch that easily enough thoush
(09:57:38 AM) gac410: Okay.  Fails for me with CGI 4.14
(09:57:43 AM) andreli: Have to go, an other meeting is calling. Keep up the good work. Thanks a lot. Bye.
(09:57:54 AM) gac410: bye andreli
(09:57:57 AM) andreli left the room (quit: Remote host closed the connection).
(09:59:22 AM) gac410: And fails on CGI 4.15 as well 
(10:03:27 AM) ***gac410 is booting his debian unstable vm with nginx   ... see if he can recreate the issue there.
(10:09:51 AM) gac410: okay.   Updated my Debian / Nginx / fcgi system.    Fails.
(10:10:15 AM) gac410: CGI 4.09
(10:12:36 AM) gac410: And that's the newest CGI available in Debian unstable.
(10:14:03 AM) gac410: MichaelDaum:   Do you know where CDot has been?   Is May 1st a holiday time over in Europe?
(10:14:23 AM) MichaelDaum: it is a holiday in Germany ... not sure where CDot is
(10:15:16 AM) ***gac410 has one other change to check into master.   But wan't CDot's blessing.     Item13391 branch on github.   Has major changes to the CLI extension installer and to tools/configure.
(10:16:01 AM) gac410: Basically now that Package.pm  attempts to update the configuration, the old   <SomeExtension_installer>  is broken
(10:16:32 AM) gac410: Fix is to change tools/extender.pl   to convert the old CLI options into the equivalent tools/configure options  and let the wizard do the install
(10:17:02 AM) gac410: All tested and working, but CDot is the expert on all things configure.
(10:32:29 AM) Lynnwood  entered the room.
(10:35:30 AM) JulianLevens: gac410: What's the status?
(10:36:12 AM) gac410: Waiting on MichaelDaum ... he's working on an alternate html5 fix for SubscribePlugin
(10:36:59 AM) gac410: speaking of which ...   
(10:37:02 AM) ***gac410 goes to test
(10:37:39 AM) JulianLevens: Hmm, still have an uneasy feeling that we have not got to the heart of the problem
(10:37:52 AM) gac410: Bingo ... fixed     But need to test more.
(10:38:16 AM) MichaelDaum: I've checked in a new SubscribePlugin
(10:38:44 AM) gac410: MichaelDaum:    Yes, it does seem to fix the issue for me.    Will test further though.      
(10:38:50 AM) MichaelDaum: okay
(10:39:02 AM) MichaelDaum: for the logs: https://github.com/foswiki/distro/compare/27fec49452d6...685b1daf6dcd
(10:39:09 AM) gac410: I'm still really uneasy about Crawford's comments.   Regarding foswiki being fundamentally broken for any characterset   Except utf8
(10:40:27 AM) MichaelDaum: I am not convinced that this is the case. We should have noticed before.
(10:41:41 AM) gac410: I agree with you.   I don't understand what we changed that would have made things so broken.   trunk.foswiki.org is running fine on iso-8859-1 
(10:42:51 AM) MichaelDaum: that's why I'd say: beta2
(10:42:54 AM) gac410: Well so far  the forms with Czech characters seem to be working fine.    I think that this is fixed.
(10:43:39 AM) gac410: I'll email CDot.    I'll have jomo test your fixes  if I can find him.   And if I don't hear from CDot,  I'll merge my extension installer fixes any and build beta 2.
(10:44:16 AM) gac410: I'd rather wait one more day and get a beta2 that will hopefully become a release candidate.
(10:44:30 AM) JulianLevens: Sounds like a plan :)
(10:45:16 AM) JulianLevens: Bye guys need to go
(10:45:18 AM) gac410: Your fix tests fine on CGI 3.65 and 4.09.    I'll switch to 4.15 and give it one more try.
(10:45:24 AM) gac410: Seeya JulianLevens    Thanks!
(10:45:28 AM) JulianLevens: Thanks too
(10:45:34 AM) JulianLevens left the room.
(10:49:46 AM) MichaelDaum: can we talk on feature requests?
(10:50:05 AM) gac410: For 1.2.0?
(10:50:11 AM) MichaelDaum: Lynnwood, could you describe which kind of actions you'd like to see on http://foswiki.org/Development/CustomizeDisplayedFieldTitle
(10:50:18 AM) MichaelDaum: gac410, of course not
(10:50:22 AM) gac410: :D
(10:50:37 AM) Lynnwood: hey there.
(10:50:40 AM) MichaelDaum: hey :)
(10:51:21 AM) Lynnwood: let me take a look. I realize that this may be too big a task to take on at late date.
(10:51:32 AM) Lynnwood: (in terms of actually changing the functionality)
(10:51:45 AM) MichaelDaum: any feature request is a post 1.2 thingy anyway
(10:52:00 AM) Lynnwood: the alternative short-term action would be to fix the docs so they aren't wrong/misleading.
(10:52:32 AM) MichaelDaum: how do I say it polite: lipstick on a pig?
(10:52:56 AM) Lynnwood: well... i think the change in the docs that made them wrong are pretty recent.
(10:54:03 AM) MichaelDaum: okay could you add your action points to the feature request please, and then we know better what the scope of the FR is.
(10:54:21 AM) Lynnwood: i had never really noticed the issue until seeing that support request so I can't say the task itself is that pressing. None-the-less, when you consider it, it's a reasonalbe request.
(10:54:23 AM) Lynnwood: sure
(10:56:03 AM) MichaelDaum: I agree. Having a both - a display title and a formfield ID - makes totally sense.
(10:56:32 AM) MichaelDaum: however, sad as it is, this has never been solved correctly in the specs since the beginning of it.
(10:58:00 AM) MichaelDaum: my pov is, that while we could docu the status quo of the specs, they don't really fulfill the request.
(10:58:23 AM) MichaelDaum: so you'd get correct docu of broken specs at best
(10:59:08 AM) Lynnwood: correct, but tmk, no one has ever actually requested it be fixed before.
(10:59:38 AM) MichaelDaum: everybody was working around it in wiki apps
(10:59:47 AM) MichaelDaum: <form> 
(11:00:00 AM) MichaelDaum: *My Label*: %FORMFIELD{"foo"}%
(11:00:00 AM) Lynnwood: yes, and FlexFormPlugin makes it not very difficult
(11:00:01 AM) MichaelDaum: </form>
(11:01:07 AM) Lynnwood: The "fixes" i imagined seemed like they would be too much to implement for 1.2 so that's why i suggested simply fixing the docs for now.
(11:01:33 AM) MichaelDaum: i.c. thats quite low risk for 1.2.0.
(11:01:37 AM) Lynnwood: ...but i'm not qualified to determine how hard a "real" fix would be to code.
(11:02:04 AM) MichaelDaum: oh thats an understatement
(11:02:26 AM) Lynnwood: What approach do you think would be best? Allowing "ActualFieldName=My displayed field name" for field name value?
(11:03:01 AM) Lynnwood: It's at least consistent with +values syntax.
(11:03:34 AM) Lynnwood: It seems to me that the [[FieldName][Descriptive human-friendly Field Name]] would only confuse the syntax.
(11:05:03 AM) MichaelDaum: further note that you'd require a topic FieldName to be present holding all possible values. 
(11:05:25 AM) MichaelDaum: not all formfield types are reading from these value defining topics
(11:05:33 AM) Lynnwood: right
(11:06:33 AM) MichaelDaum: there are two things we need to change here
(11:06:41 AM) MichaelDaum: (1) data form definitions
(11:06:50 AM) MichaelDaum: (2) the %META:FIELD format
(11:08:13 AM) MichaelDaum: there is a title definition in | My New Field | text | 80 | ...| already
(11:08:36 AM) Lynnwood: yes
(11:08:53 AM) MichaelDaum: the data is accessible via %FORMFIELD{"MyNewField"}% ... but as far as I remember also via %FORMFIELD{"My New Field"}% ...
(11:09:42 AM) Lynnwood: i'm not sure about that, but i can confirm that %QUERY{"My New Field"}% does work.
(11:10:06 AM) MichaelDaum: y. there is some perl api to derive the formfield name from a display title
(11:10:35 AM) MichaelDaum: odd
(11:11:05 AM) MichaelDaum: this all is not fit for i18n
(11:11:27 AM) Lynnwood: btw, it looks to me that the only current doc fix that's needed is simply to remove the line "If you want the Field name to include embedded spaces, use the format [<nop>[FieldName][Descriptive human-friendly Field Name]]."
(11:12:14 AM) Lynnwood: that is both misleading in terms of how things work, but also just plain wrong. It doesn't do what it says even for topic-stored values.
(11:12:37 AM) MichaelDaum: okay
(11:12:49 AM) MichaelDaum: well thats a plain DocuFix
(11:13:02 AM) Lynnwood: y
(11:14:17 AM) MichaelDaum: which you cloud simply checkin as part of http://foswiki.org/Tasks/Item9693
(11:14:52 AM) Lynnwood: yep
(11:33:49 AM) gac410: MichaelDaum:    Is there a simple command I can use from the shell update script to clear the cache on trunk.foswiki.org after each code change is pulled in from github?
(11:34:31 AM) MichaelDaum: ./view refresh=all
(11:35:30 AM) gac410: Ah...    run that from the shell.   okay.     I was looking for a way to rm the cache files or something.
(11:40:08 AM) MichaelDaum: thats possible as well: rm working/cache/*
(11:40:37 AM) MichaelDaum: as long as you don't use a real db to store the cache meta data
(11:40:53 AM) MichaelDaum: sqlite is there as well the html blobs
(11:40:58 AM) gac410: Okay.   I think I'll go that route.   I already have code that cleans up a bunch of stuff    hm.   I assume trunk.foswiki.org doesn't use a real db.   
(11:41:40 AM) gac410: Hm.   hang on ....    I issue a  "git clean -fdx"    That should remove the cache stuff from working as well.
(11:42:42 AM) gac410: Trunk is using Foswiki::PageCache::DBI::SQLite. 
(11:44:35 AM) gac410: Anyway  I've sent email to CDot about your fix and my extender.pl / tools/configure  changes.        If I don't hear in the next 24 hours,  I'll build Beta2
(11:45:10 AM) gac410: So    I think that closes our release meeting.   Thanks everyone.    
(11:59:23 AM) MichaelDaum: yw
(12:01:56 PM) gac410: Oh.. btw, MichaelDaum    You might take a peek at http://foswiki.org/Tasks/Item13388     Another utf8 list of bugs from jomo.   But not urgent, so will not block 1.2.0
(12:07:16 PM) MichaelDaum: %LTWL1% is not defined
(12:13:24 PM) gac410: MichaelDaum:  ???   %LTWEL1%   ?
(12:13:51 PM) MichaelDaum: thats in the attached test topic
(12:14:34 PM) MichaelDaum: there is a javascript error as well, presumably because the jquery code has not been run under NoConflict ... which every dev should do
(12:14:56 PM) MichaelDaum: other than that those tests show quite some errors
(12:14:57 PM) gac410: hm   Maybe the wrong topic?   
(12:15:12 PM) gac410: Topic name is AaaTest1    with an AaaTest1Form
(12:15:29 PM) MichaelDaum: thats a different test 
(12:15:38 PM) MichaelDaum: UtfErrors.txt
(12:15:48 PM) MichaelDaum: %QUERY and %CALC errors
(12:16:20 PM) gac410: Oh... okay   yeah I have not really dug into that one.    
(12:17:04 PM) MichaelDaum: lc and uc require properly decoded strings. otherwise they just f*up the bytes
(12:17:48 PM) gac410: okay.   So broken test then?
(12:18:10 PM) MichaelDaum: the tests are a tiny bit broken. the code is quite broken.
(12:18:29 PM) MichaelDaum: in core
(12:18:35 PM) gac410: Oh...  1.2.0 badly broken,  or broken for 1.3/2.0  
(12:18:51 PM) gac410: Ie  is this another urgent  :(
(12:19:49 PM) gac410: In some ways I'm fearing  "feature creep"   2.0 is supposed to be our big unicode release ... if we ever get there.
(12:20:04 PM) MichaelDaum: we might be able to hack SpreadShit to work fine. however these fixes wont be TheRealThing
(12:20:49 PM) ***gac410 is willing to document as 'known issues'  if they are too complex to fix for 1.2.0
(12:21:05 PM) gac410: What we've got is still 1000 fold better than 1.1.9
(12:21:30 PM) gac410: And I assume that 1.1.9 is just as badly broken.   We've not made it worse
(12:22:38 PM) MichaelDaum: lc, uc, and substr all suffer from the same disease
(12:23:26 PM) gac410: SubscribePlugin unit tests fail.  8 of 11.  Maybe simple.    Expected status 200,  got status 400.
(12:23:38 PM) MichaelDaum: the hack is to do: (1) decode(site-charset), (2) lc/uc/substr (3) encode(site-charset) ... which sux
(12:23:46 PM) gac410: and malformed JSON string, neither array, object, number, string or atom, at charact
(12:25:32 PM) MichaelDaum: the tests deepen the shit the plugin created 
(12:29:29 PM) gac410: well ... lunchtime for me   and I need to get off the couch and move lest I grow roots.   :)     It's 24C out.   Warmest day of the year so far.   yardwork awaits.  
(12:29:45 PM) gac410: I'll work on the SubscribePlugin tests tonight.     
(12:30:33 PM) gac410: MichaelDaum:   I assume that any spreadsheet function that uses regexes might be broken then.   not just those three.
(12:31:25 PM) gac410: or at least regexes that might match character classes
(12:35:16 PM) gac410: ugh.  And of course SSP uses stuff like [A-Z] in places.   Indeed you expand the acronym correctly.
(12:40:14 PM) gac410: SSP  $SUBSTITUTE can accept a generic regex,  so could involve character classes.    Same with $SEARCH
(12:46:59 PM) MichaelDaum left the room (quit: ).
(12:47:00 PM) gac410: And the Date functions in SSP   use ([A-Z][a-z][a-z])    to match month names, day names, etc.   yeesh.