Release Meeting: 30 Nov 2015

(08:01:35 AM) gac410: hm.   Rather thin turnout.  ...
(08:01:53 AM) gac410: Shall we postpone an hour to 1400z  and see if anyone else turns up.  
(08:03:44 AM) gac410: I don't want to get into a practice of release meetings with only 1-2 attendees making decisions. ... which happens at the "other project".   
(08:13:09 AM) JulianLevens: Happy to postpone in principle but I'll need to leave around 1445Z
(08:13:39 AM) gac410: Okay.   Well   how did you make out with the backlinks work?
(08:14:26 AM) gac410: Maybe we can't make formal decisions,  but we can still chat.   
(08:15:29 AM) gac410: :(   I take it this is one way to force a decision to delay 2.1    If nobody shows to plan, I guess the decision is forced.
(08:22:49 AM) JulianLevens: I added the syntax I did basic testing with hardcoded results
(08:23:06 AM) gac410: Cool.   Good start
(08:23:42 AM) JulianLevens: Adding the further actual regex to scan backlinks via regex as done now, is breaking and I do not know why yet
(08:24:06 AM) JulianLevens: This is probably simple enough (when I find it)
(08:24:46 AM) JulianLevens: In some ways this may prove to be smallest amount of time
(08:25:12 AM) gac410: yeah  those backlink regexes are really ugly.  There are so many different ways to write topic links.
(08:25:44 AM) gac410: In some ways it would be eaiser to cache all the backlinks using a scan of the rendered results in a completePageHandler   
(08:25:55 AM) JulianLevens: Ideally I need to amend some templates
(08:27:18 AM) gac410: hm   You know,  maybe a database of the links would be a solution to consider.    It would need a one-time scan to "view" every possible topic,  and then maintenance on each topic save.
(08:27:28 AM) JulianLevens: That may prove to be fiddly or I may add the macros %GLOBAL_SEARCH% and %LOCAL_SEARCH% properly (docs etc) which get turned into %SEARCH{"backlinks" type=query}%
(08:27:53 AM) gac410: The big benefit is that it can also find html links,  "generated" links and broken links.
(08:28:26 AM) JulianLevens: If you check the feature proposal you'll find those ideas are covered
(08:28:40 AM) gac410: oh...  okay   :D   
(08:29:44 AM) JulianLevens: However, this is just adding the backlinks syntax to the query language. Anyone can then add an Extension to achieve what you want, most likely as part of a store
(08:30:02 AM) gac410: Okay.   
(08:30:42 AM) gac410: I'll go re-read the proposal more carefully.    In the meantime,  how are you with utf-8 / unicode questions.   Item13876    I'm still missing why / when this will happen.
(08:30:56 AM) JulianLevens: Of course I need to implement a default implementation of query-backlinks which will just be the existing regex
(08:31:59 AM) gac410: Ah   very second sentence.   "capture backlinks during save" ...   doh.    Nothing like re-inventing your invention  ;)
(08:36:16 AM) JulianLevens: That %BB being replaced by %BB1 sounds like a work-around rather than a solution, but that'll take time for me to dig into into properly and understand
(08:37:54 AM) JulianLevens: I think the MACRO parsing may well be too crude or we need a way to escape around this (very cleanly that is)
(08:38:25 AM) gac410: What I don't really get is why / when should url encoded text appear in TML prior to rendering.
(08:39:12 AM) gac410: We would have to deprecate any macro that begins with  [A-Fa-f]{2,2}
(08:39:40 AM) gac410: no... actually  [A-F][A-Fa-f0-9]
(08:40:00 AM) JulianLevens: Could they be writing there own URL
(08:40:48 AM) JulianLevens: s/there/their/
(08:40:48 AM) gac410: y.   Hm   do we have  a markup that disables %macros%  but leaves html active.    <literal> ... </literal>  ?
(08:42:08 AM) JulianLevens: Would %C2%BB%C2%BB -> %C2<nop>%BB<nop>%C2<nop>%BB fix this
(08:43:09 AM) JulianLevens: The macro parse won't see these as macros any more at least
(08:43:59 AM) gac410: The thing with macros, they are "user defined"   So maybe BB is a system one,  But nothing to stop a user from defining * Set C1=   * Set C2=  ...
(08:44:17 AM) JulianLevens: y, exactly
(08:44:56 AM) gac410: When *we* create a url encoded URL we always use lower case.   so %c2%bb%c2...
(08:45:28 AM) JulianLevens: OTOH, if URL are hand-written then as long as there an escape mech we may just need to document that
(08:46:18 AM) JulianLevens: Or just doc to use lower case %bb as you just said - it's an escape of sorts as well
(08:46:53 AM) gac410: Except that any user could also * Set c2=   * Set bb =    So not a perfect solution
(08:49:12 AM) gac410: We don't have a markup that turns off both TML and MACRO expansion.   <literal>  does HTML + MACROS,  but disables =other= | TML |
(08:52:58 AM) gac410: Maybe need to add something like  <nomacro>  ...  </nomacro>
(08:54:02 AM) JulianLevens: Are you sure that there isn't an existing way of escaping this?
(08:54:43 AM) JulianLevens: I fee that the task does not really give enough context
(08:54:43 AM) JulianLevens: s/fee/feel/
(08:55:10 AM) gac410: I just went and checked EditingShorthand.     The way to stop macro expansion  is with !%BB%       <nop>%BB% does not stop the macro, it stops an expanded wikiword from being linked
(08:55:53 AM) gac410: So Set BB = WikiWord       !%BB%  gives you %BB%,     <nop>%BB%   gives you an un-linked WikiWord
(08:56:49 AM) gac410: Y  I agree,  I really need more context to propose a solution,  but indeed macro expansion of urlencoded data is  problem.
(08:58:04 AM) JulianLevens: Ok so how would you escape %BB%C2%BB%C2  which are meant to represent 4 characters but are seen by TML as %BB%  C2  %BB% C2
(09:00:29 AM) JulianLevens: If you add spaces you get %BB %C2 %BB %C2 and the macro parse will not see them as macros, you just need something instead of space which is eliminated in the final rendered text
(09:01:04 AM) gac410: If the source is a user cut/paste of a URL from another source,  that will be an awful mess
(09:01:47 AM) gac410: [[%ATTACHURL%/F%c4%bel%c3%a9xP%c3%a1p%c4%9b%c5%95P%c4%ba%c3%bag%c3%ad%c5%88.pdf]]
(09:02:08 AM) JulianLevens: y, it's amazing how something so innocuous can be so difficult
(09:02:18 AM) gac410: Is a link I've been testing with for Item13874
(09:02:19 AM) jomo entered the room.
(09:02:42 AM) jast entered the room.
(09:02:51 AM) jomo: hi all :)
(09:03:08 AM) gac410: Hi jomo.    Nobody can figure out timezones.   1300Z   was an hour ago.   :(
(09:03:40 AM) gac410: Hi jast ..  you too   
(09:03:56 AM) gac410: So far it's been just JulianLevens and me.    Lavr's presence is a bot I suspect.
(09:04:05 AM) jomo: hm..
(09:05:18 AM) gac410: Nobody's a pilot or in the military here I guess.   keeping zulu time is rather ordinary for pilots ... and I assume military too  
(09:05:50 AM) gac410: Anyway,  jomo,  we've been discussing http://foswiki.org/Tasks/Item13876   while waiting for others ...   
(09:06:21 AM) gac410: Foswiki macro syntax   is colliding with ULR encoding   for anyone who pastes in a URL that's been encoded. 
(09:06:26 AM) jast: sorry, I was busy elsewhere :(
(09:06:30 AM) JulianLevens: Actually I need to go already, I'd forgotten about another event I need to attend
(09:06:41 AM) gac410: At least I am guessing the context ...     Okay JulianLevens    See you .  Thanks for coming by.
(09:06:54 AM) gac410: And thanks for working on backlinks.
(09:07:06 AM) gac410: My guess is we'll push out 2.1  for at least 30-60 days
(09:07:18 AM) JulianLevens: I leave you and your new friends with this tricky problem
(09:07:19 AM) JulianLevens left the room.
(09:07:23 AM) jomo: never used BB - so, need learn what it is...
(09:07:37 AM) gac410: It's just a macro that expands into a bullet
(09:08:22 AM) gac410: The generic issue is that any macro  of   [A-Fa-f][A-Fa-f0-9]   can collide with url encoding.
(09:10:04 AM) jomo: ah yes... now got it %AA macro is the same as %AA urlencoded code...
(09:10:16 AM) gac410: It could even happen with any attachment link inserted by the Attach "Insert link" feature.   Since it writes the encoded link into TML
(09:11:05 AM) gac410: *generated* links,  like we do with Attach,   I'm proposing that we eliminate the encoding in http://foswiki.org/Tasks/Item13874
(09:11:20 AM) gac410: But user pasted links are still an issue.
(09:11:29 AM) jomo: hm... how that doesn't happens before utf8?
(09:11:49 AM) gac410: It probably did.   Probably just wasn't noticed.
(09:11:53 AM) jomo: e.g. %BB - before utf8 - is still colliding..
(09:12:06 AM) gac410: Yes you are correct.
(09:12:21 AM) jast: I agree non-ASCII chars shouldn't be encoded
(09:12:21 AM) jomo: :)
(09:12:52 AM) gac410: So that still leaves us with the cut/paste of a  url from another source.   
(09:13:38 AM) jast: can't really think of a way to fix that without some serious downsides
(09:14:50 AM) gac410: Just verified.   If a link contains unicode,  You "click" it,  and then paste it from the browser url bar,   it is entity encoded.     So it's a generic cut/paste issue.
(09:15:06 AM) jast: for instance, we could ignore these kinds of macros inside anything that looks like it might be a link... but that's easier said than done, and who knows if there isn't a special case in which we actually do want these things expanded
(09:15:32 AM) gac410: http://%PUBPATH%/blah
(09:15:41 AM) jast: depends on the browser, too, I think
(09:16:15 AM) jast: yeah, I meant /%[a-f][\da-f]%/ macros specifically
(09:16:21 AM) jast: /i
(09:16:22 AM) gac410: Okay yeah maybe.  Firefox at least,  the "URL bar"  displays unicode but puts encoded utf-8 onto the clipboard
(09:16:53 AM) jomo left the room (quit: Ping timeout: 257 seconds).
(09:17:05 AM) gac410: jast.  That would really really be difficult I suspect.   What a mess of regex   to disable 2-character "hex" macros when inside of a link
(09:17:38 AM) jast: yes, absolutely
(09:17:51 AM) Lynnwood entered the room.
(09:17:56 AM) jast: I don't have any better ideas right now
(09:20:58 AM) gac410: Anyway guys.   I'm not sure what to do.  Meeting time was 1 hour ago.  We've lost Julian,   and  still don't have  MichaelDaum or CDot,   both of which have outstanding proposals to decide about for 2.1
(09:21:36 AM) gac410: I was hoping to do a serious feature review ... agenda in http://foswiki.org/Development/ReleaseMeeting02x01_20151130
(09:23:02 AM) gac410: MichaelDaum has 4 proposals for 2.1,  none of which have public implementations that I'm aware of anyway
(09:23:32 AM) gac410: JulianLevens has one .. .to fix the backlinks mess,   which is partially implemented but needs more time.
(09:24:18 AM) gac410: So the question to put before everyone here:   http://foswiki.org/Development/ReleaseMeeting02x01_20151130#A_2._Development_Discussion     
(09:25:29 AM) gac410: Should we push 2.1 out to March 2016.   I think with the holidays  30 day delay is insufficient.       Or should we push ahead with a *small* 2.1,  with the 7 or 8 merged or read-to-merge features
(09:26:37 AM) gac410: We also have "Accepted" proposals - 2 from Lavr,   which have no submitted work.    Lavr  are you around
(09:27:23 AM) jast: IMO there's nothing wrong with a smaller release
(09:28:16 AM) jast: if we keep on delaying releases because people don't drive their proposals forward, it'll be 1.2 all over again
(09:28:21 AM) gac410: That was my opinion too.     Michael really wanted his stuff to get in.    But the train is at the platform ...
(09:30:09 AM) jast: there are things I'd like to make happen in 2.1, too, but there's simply no way I'd get them done in time even if I did have complete proposals
(09:30:20 AM) gac410: If he was *here*,   I'd be willing to hear arguments to delay to March.      I guess my concern is how big is the broken backlinks issue.  Currently pure ascii only [A-Z][a-z] 
(09:30:27 AM) jast: right now I can't even remember most of them ;)
(09:31:19 AM) gac410: I see the backlinks problem as really more of a blocker bug for unicode sites,    That's what JulianLeven's proposal is to fix.   I suspect if we "feature freeze" today,  he has too much to get working to meet a 1/1/2016 release.
(09:33:18 AM) gac410: 5 of the 8 completed proposals were merged by me.  So I don't object to delay,  none were particularly critical.
(09:33:39 AM) gac410: But vrurg has worked hard on the mime email support,   
(09:34:54 AM) gac410: One was Julian's   and he has a lot of work pending on the backlinks.   And one was Michaels'  and I needed his guidance on what to do with it.
(09:35:20 AM) gac410: Lynnwood:  Did you make any progress with http://foswiki.org/Development/CentralizeAndHideAdminInstructions
(09:35:48 AM) jast: I can't comment on what they've been doing and how important or how close to completion it is, all I've got is a general opinion... sorry... :)
(09:35:59 AM) Lynnwood: no i haven't but this week i should have time to evaluate if it's obsolete or not. 
(09:36:14 AM) gac410: okay thanks     Do you have an opinion?    Delay 2.1?   
(09:36:47 AM) Lynnwood: no, i'm reviewing the various proposals but not knowledgable enough to have opinion
(09:36:54 AM) gac410: Okay.   
(09:41:56 AM) gac410: Well  crap...    My temptation is to merge the remaing stuff that's "ready to merge"  and if others can't bother showing up or representing their opinions through other forums .. even a comment in the published agenda.
(09:50:37 AM) jomo entered the room.
(09:50:54 AM) jomo: got a disconnect from the chat.freenode.net... ;(
(09:51:25 AM) gac410: reported here as a "ping timeout"
(09:51:47 AM) jomo: now changed the server...
(09:52:25 AM) gac410: Well I guess without the proposers,   we can't get anywhere on proposals.     So another item is tasks that rise above the simple bugfix level
(09:52:29 AM) jomo: so, back to %BB - and what is someone create a macro such %BAG ... same issue for the %BA
(09:53:17 AM) gac410: Exactly.   I have no idea how to fix.      I don't think we can safely block all macros inside of http links.
(09:53:21 AM) jast: yeah, about the same thing can happen with that
(09:53:32 AM) jast: it may just be something we'll have to live with
(09:53:45 AM) jast: maybe provide an option to disable the conflicting core macros
(09:54:12 AM) gac410: jast,   I'm wondering about adding a <nomacro> </nomacro>   block  that would disable macros while allowing normal rendering.
(09:54:14 AM) jast: though the short codes like BB are implemented as preferences, I think, so that's not straightforward either
(09:54:52 AM) jast: that might be good, though I'm not sure it's easy to get to work with the current rendering order
(09:55:19 AM) jast: it would allow people to disable macros and TML in arbitrary combinations
(09:56:32 AM) gac410: yeah.   We have <literal>   which disables TML only,  leaves macros and html    verbatim>  which disables TML, macros and HTML   
(09:57:09 AM) jast: verbatim> is easy to implement, so is <literal> and <nomacro> individually... but both of them, that's a different story
(09:57:28 AM) gac410: But we don't have the one in-between  
(09:58:34 AM) gac410: I think a <nomacro>  zone  makes sense,   and could be done with a takeOutBlock   just before macro expanson,  and then a putBack
(09:58:49 AM) gac410: But that one really needs a feature proposal. 
(09:59:07 AM) gac410: jomo,   another question...   Normalization.   
(09:59:36 AM) gac410: If CDot had not been so sparse,  I was going to suggest that it be picked up in 2.1
(10:00:01 AM) gac410: http://foswiki.org/Development/ReleaseMeeting02x01_20151130#Tasks_47_minor_enhancements
(10:01:38 AM) jomo: so, the solution will be - everywhere where we doing decode_utf8( will do instead NFC(decode_utf8( ? or?
(10:01:47 AM) jast: using something like NFD/NFKD doesn't magically fix all sorting
(10:01:54 AM) jast: it just makes it approximately correct
(10:02:15 AM) gac410: Is there a better sorting solution? 
(10:02:25 AM) jast: correct sorting is always locale-specific
(10:02:51 AM) jomo: for  the correct sorting we will need Unicode::Collate::Locate - but IMHO - the current "approximative"  is OK... (at least for now)...
(10:02:57 AM) gac410: How is 2.0 with Locale's enabled?    Now that taint checking is disabled.  
(10:03:07 AM) jast: in most cases sorting with characters like äéç should be done as though the normal character without mark was used
(10:03:45 AM) jast: I haven't tested locale, I'd need to set up some extra time for that
(10:06:12 AM) gac410: jomo,   I think cdot crafted the decode_utf8  so that it always calls a Foswiki::   function instead of directly calling Encode:: ....  This way we could add normalization in a single place.
(10:06:30 AM) gac410: Except for all the exceptions where someone calls Encode:: directly :(
(10:06:35 AM) jomo: :) :)
(10:07:33 AM) gac410: I think we'd need to talk to CDot about it,  and also do some timing with various perl versions.    How significant is the overhead of always normalizing 
(10:08:49 AM) jomo: img it is small - it could add few microseconds - but in compare with the whole request cycle it isn't much (and we haven't much other solututions anyway - for OS X ) :(
(10:09:05 AM) jomo: damm - mean s/img/IMHO/
(10:09:35 AM) gac410: right.   We just need to know.        The UNICODE regex issue is a huge problem for anyone using Perl older than 5.20  for ex.
(10:10:02 AM) gac410: What will normalizing everything do to other unrelated regex matching.   All the takeOutBlocks,   etc.
(10:11:03 AM) gac410: the code that "protects" html entities, in render.   made bit EditRowPlugin pages take 20 minutes in some cases to render
(10:11:46 AM) gac410: Perl 5.20 took the render from minutes to seconds.
(10:12:54 AM) jomo: ad Encode: - here is a MUCH faster solution Unicode::UTF8 - but it isn't failsafe - so usable when the source us ensured utf8 - in such case the Unicode::UTF8 is MUCH faster as Encode.... for the Unicode::Normalize - so - need create some benchmarks?
(10:14:13 AM) gac410: Well the bug we ran into had nothing to do with encoding.   The issue is that on perl older than 5.20, regexes against UNICODE fall back to an un-optimized scan.   
(10:15:00 AM) gac410: https://rt.perl.org/Public/Bug/Display.html?id=66852
(10:15:35 AM) gac410: But jomo,  you were the one who found that one  ;)  
(10:16:21 AM) gac410: Anyway. ... to everyone who showed up today... Thank you very much.    As far as 2.1 goes,   I'm going to delay the decision rather than delay the releae.
(10:16:53 AM) jomo left the room (quit: Quit: jomo).
(10:17:11 AM) gac410: Next meeting December 14, 
(10:17:19 AM) jomo entered the room.
(10:18:09 AM) gac410: jomo... Next meeting  14 Dec,   1300Z     with an attempt to cover the exact same agenda that we had for today.  :P
(10:18:44 AM) jomo: okay - i changed the calendar entry  - today has some internet problems anyway)... ;)
(10:18:45 AM) jomo: :(
(10:19:14 AM) gac410: In the meantime  I'm going to merge the remaining "Needs merge" proposals   .. that I can anyway   
(10:20:45 AM) jomo: ok - what is the best way create the normalisation benchmark? - the simple demo-perl script will NOT show the impact to the whole FW... so?
(10:21:14 AM) gac410: Lavr?   If you review your logs.    Please park your proposals if you don't intend to develop them.  There are two in the backlog. 
(10:21:56 AM) gac410: jomo,  I bet a view on a large table with EditRowPlugin enabled.  That generates lots of regex matches due to the html entities. 
(10:24:44 AM) jomo left the room (quit: Quit: jomo).
(10:25:22 AM) gac410: So barring any major objections.  Feature freeze now planned for 15 December,  following our 14 Dec. release meeting.   2.1 target now 15 Jan 2016
(10:26:14 AM) gac410: And any proposals that are left on the platform can wait until the July 2016 release.   Probably 2.2      I think it's reasonable for 6-month minor feature releases.
(10:26:57 AM) gac410: Objections?   Agreements?     going once ...  going twice ...
(10:27:13 AM) gac410: And "Move to adjourn"