Release Meeting: 31 Oct 2016

(09:01:16 AM) gac410: Hi everyone...   ready for release meeting?
(09:03:37 AM) gac410: On today's agenda.   New tasks,  and new Proposals.  
(09:04:56 AM) vrurg: Hi everybody!
(09:05:22 AM) gac410: howdy vrurg.
(09:07:04 AM) gac410: Big response so far.  ;)  
(09:07:09 AM) vrurg: It seems like nobody here...
(09:07:13 AM) vrurg: :)
(09:07:47 AM) vrurg: I saw your comments on my last proposal. Unfortunately, had time to read, not to write.
(09:08:01 AM) vrurg: Same with JSON matters.
(09:08:14 AM) MichaelDaum: hi there
(09:08:22 AM) gac410: For agenda,  we have a few tasks to review.  
(09:08:36 AM) gac410: Hi MichaelDaum   Back from travels   or just near a hotspot  ;)
(09:08:46 AM) MichaelDaum: back at work
(09:09:20 AM) gac410: https://foswiki.org/Tasks/Item14202 -  New urgent task.  Made changes to cache to try to reduce the exponential growth of the dependencies table.
(09:09:21 AM) vrurg: Hi Michael
(09:10:43 AM) MichaelDaum: gac410, I am not sure it grows "exponential".
(09:10:50 AM) gac410: I plan to deploy fix to 14202 on foswiki.org at some point.
(09:10:52 AM) MichaelDaum: Hi Vadim
(09:11:12 AM) MichaelDaum: didn't have made it thru to look at the patch for 142o2
(09:11:39 AM) MichaelDaum: the root cause for the slow down isnt addressed as far as I can see though...
(09:11:58 AM) MichaelDaum: there is too much perl code which shouldbe sql code
(09:12:13 AM) gac410: The deps table gets so large that it is impossible to even count the # of rows.  
(09:12:23 AM) MichaelDaum: thus offloading deletion of dependencies to the sql store ... instead of doing it in perl 
(09:15:04 AM) gac410: The growth may not be exponential,  but it sure is big.   Just observing the contents of deps table.   many "dependencies" are added because of simple links - so cache can be invalidated if a topic is added/remove  changing a link from "new topic" to rendered.
(09:15:53 AM) MichaelDaum: y
(09:16:01 AM) gac410: By suppressing these dependencies for guests,  it doesn't degrade the accuracy of the cache that much, and certainly slows down the growth.
(09:16:17 AM) gac410: Made it configurable though,  with default being current behavior.
(09:16:48 AM) MichaelDaum: I would not call it addTopicRef
(09:17:00 AM) MichaelDaum: this is quite confusing
(09:17:11 AM) MichaelDaum: addDependencyForWikiWord()
(09:17:28 AM) gac410: Also it really points out the damage caused by the WebLeftBarWebsList.  which I further disabled on foswiki.or
(09:17:31 AM) MichaelDaum: makes more sense
(09:18:41 AM) MichaelDaum: I did not quite get your evaluation results
(09:18:46 AM) gac410: well simple enough to adjust.   I thought a casual reference could be treated as a less important thing than a dependency for topic contents.
(09:19:13 AM) gac410: what do you mean my "evaluation results"
(09:19:20 AM) MichaelDaum: sorry
(09:19:25 AM) MichaelDaum: for not being precise
(09:19:44 AM) MichaelDaum: did you say it didn't reduce dependencies as much as you thought?
(09:20:29 AM) gac410: Oh.   okay.  Anyway, simple enough to change name.  ... I didn't take full statistics  it was really anecdotal / spot checking of topics.    Table grows so fast that it's hard to manage
(09:22:29 AM) MichaelDaum: alright. so this is quite experimental and results are only available on the patient itself?
(09:23:00 AM) MichaelDaum: strange a select count(*) does not return an accurate number of rows on the deps table?
(09:23:10 AM) MichaelDaum: wtf-ish
(09:23:12 AM) gac410: It never returns. 
(09:23:27 AM) MichaelDaum: should be O(1)
(09:23:35 AM) gac410: well 10 minutes later it still sits there chugging.    Table up in millions of rows.
(09:23:57 AM) MichaelDaum: sql dbs are made for large tables arent they
(09:24:23 AM) MichaelDaum: which one are you using?
(09:24:29 AM) gac410: yes,   but still,   on foswiki.org,  after a few days of caching,  it is no longer possible to get the count(*) from the dependencies table.    MySQL
(09:25:14 AM) ***MichaelDaum looking at the patch
(09:25:15 AM) gac410: Either it's due to simple size,  or it's due to lock contention while fw is still live.
(09:25:26 AM) MichaelDaum: i.c.
(09:25:27 AM) gac410: hard to say.
(09:25:56 AM) gac410: next time I go looking and have a non-responding query,  I'll consider stopping apache for a bit and see if I can get the count(*)
(09:26:04 AM) MichaelDaum: the number of rows in a table is pretty essential to the db itself...and surely should not depend on the size of the table itself.
(09:26:33 AM) MichaelDaum: in terms of complexity to "compute" the number of rows
(09:27:00 AM) MichaelDaum: the db engine added those rows _itself_ in the first place. so keeping track of the count should be a side product
(09:27:06 AM) gac410: I don't know.   I'm not really a MySQL expert.  Just know what I'm observing.   There is an alternate way to ask the engine for the table size,  and that responds,  but I can never remember how,  and can't remember how to find it.
(09:27:24 AM) MichaelDaum: ah ok
(09:27:47 AM) MichaelDaum: anyhow. hard to analyze the data any further going beyond basic counts.
(09:28:42 AM) gac410: What I was doing was looking at individual pages,  early, after a full cache refresh,   and examining what deps were added for each page,  then looking at why the deps were added.
(09:28:44 AM) MichaelDaum: however it is interesting in which order we are working here 1k, 10k, 100k, ...
(09:29:37 AM) MichaelDaum: btw why did you add addTopicRef to Search.pm
(09:29:44 AM) gac410: we are using MariaDB,  not MySQL on f.o,
(09:30:14 AM) MichaelDaum: as far as I understand the idea, this is about rendering WikiWords ... not hit sets of a search
(09:30:51 AM) gac410: Emperical, debugging.     The names were being added multiple times.    Once when "hit",  and then separately when the subset was rendered
(09:31:09 AM) gac410: So in some cases a search hit 100 topics,   but rendered 50.    
(09:31:21 AM) MichaelDaum: what I suspect is that I fucked up the impl ... again
(09:31:26 AM) gac410: So the add Dependency was hit 150 times.   
(09:32:39 AM) MichaelDaum: there _are_ a lot of dependencies. before reducing them on an adhoc base, we should fix their deletion. Only then do we know whether we actually need further optimization by ignoring dependencies to a certain degree.
(09:33:21 AM) gac410: Hm   Someone must have just refreshed the cache.  4729 pages cached,   476803 entries in deps
(09:33:39 AM) gac410: 100:1 growth of deps per page
(09:33:51 AM) MichaelDaum: and that implies rethinking the PageCache.pm so that deleting all affected dependencies are deleted in one transaction performed by the sql store.
(09:34:28 AM) MichaelDaum: yes. 100 topics, that sounds about right.
(09:34:39 AM) ***gac410 is fighting a fire on foswiki.org.   Leaving the engineers to design fireproof buildings to people above my pay grade.  :D
(09:35:23 AM) MichaelDaum: preferences, and links mostly
(09:36:09 AM) gac410: y.   And IMHO allowing the "guests" cache to permit somewhat stale data is preferred to filling a db to the point of non-responsiveness.  
(09:36:11 AM) MichaelDaum: but also anything that read a topic, such as InterWiki, CommentPlugin, etc
(09:36:52 AM) MichaelDaum: yes I agree. good idea. but does it work?
(09:37:05 AM) gac410: Right.  I was getting a pretty good picture of the "whys"  examining one-off topics and dumping all the deps.     
(09:37:54 AM) gac410: Y,  on my test system,  I was definitely seeing a reduction in growth.   Some topics by 1/2 fewer deps or more.    but acid test is to put the patch on f.o.
(09:38:13 AM) MichaelDaum: give it a try
(09:38:36 AM) gac410: I left it in an item branch to get your feedback.  But y,  I'll give it a try.  
(09:38:48 AM) gac410: Not right now though. 
(09:38:56 AM) MichaelDaum: this definitely needs more research...and f.o. is the right place to do so
(09:40:15 AM) gac410: Okay ... That was the only new urgent task.   
(09:40:49 AM) gac410: There were a few new feature request.    vrurg has requested and implemented in an item branch,  and auto-cpanm-feature 
(09:41:02 AM) MichaelDaum: awesome
(09:41:25 AM) MichaelDaum: is it compatible with cpanm being run from within a plenv or perlbrew env?
(09:42:14 AM) vrurg: MichaelDaum: I used the downloadable command line tool which should be able to run pretty much anywhere.
(09:42:15 AM) gac410: https://foswiki.org/Development/DependenciesFreedom  ... strange.  Not showing up in FeatureProposals page
(09:42:45 AM) gac410: which tool .. the old dependencies_installer.pl which IMO was a disaster?
(09:42:57 AM) MichaelDaum: heh
(09:42:58 AM) gac410: Won't work on my gentoo servers.   :P 
(09:43:18 AM) MichaelDaum: yea..and there really is no reason to reinvent cpanm
(09:43:26 AM) vrurg: gac410: Nah, it's about cpanm
(09:44:01 AM) MichaelDaum: I wish we could install Foswiki using cpanm Foswiki
(09:44:21 AM) vrurg: What is not really clear from the proposal is that it primarly targeted at limited hosting environments and at entry-level users who wish to test foswiki.
(09:45:08 AM) vrurg: MichaelDaum: You're not alone in this. But this is currently beyond my abilities. 
(09:45:46 AM) vrurg: Current test subsystem, I'm afraid, would make it a huge headache to integrate into Module::Builder or MakeMaker.
(09:46:11 AM) MichaelDaum: not sure we really would need to rewrite our test code
(09:47:11 AM) MichaelDaum: wrt entry level hosting envs: don't expect them to ship a decent perl or even enough of the modules
(09:48:26 AM) ***gac410 thinks that docker containers are still worth exploring,  as an *alternative* packaging in addition to our usual distribution methods.  
(09:48:29 AM) vrurg: I don't need a single module nor perl executable.
(09:48:35 AM) MichaelDaum: all of the perl community is in quite some trouble wrt distros and hosting envs ... they are more into php and stuff ... but perl?
(09:48:58 AM) gac410: perl gets no respect  
(09:49:46 AM) MichaelDaum: guys I have to leave and accompany my son to the doctor. will be back later during the day.
(09:49:54 AM) gac410: IMO biggest issue we have wrt packaging is the mixture of user files and distributed files
(09:50:10 AM) gac410: MichaelDaum:  okay ... good luck.   cu later
(09:50:26 AM) MichaelDaum: forget about perl modules coming with the distro.
(09:50:34 AM) vrurg: By entry level we better be considering people able to use a linux/bsd to run a test installation. This is more to corporate admins, I would say. Just to let them see foswiki's face before they throw it away.
(09:50:53 AM) MichaelDaum: a local perl really is the only option we have ... either via plenv, perlbrew or (*sigh*) docker.
(09:50:53 AM) vrurg: MichaelDaum: see you!
(09:51:16 AM) gac410: MichaelDaum:  actually on recent ubuntu,  local perl is completely adequate.  
(09:51:59 AM) gac410: Current LTS ubuntu is shipping 5.22.  
(09:52:05 AM) vrurg: It should work pretty much anywhere where it's possible to run a foswiki's script and have access to local file system.
(09:52:07 AM) gac410: Only one release back
(09:52:29 AM) MichaelDaum: setting up a local perl and compile all modules using cpanm is just fine to be scripted ...
(09:52:33 AM) gac410: vrurg,  do you bootstrap cpanm as well?
(09:52:49 AM) MichaelDaum is now known as MichaelDaum_
(09:53:13 AM) vrurg: gac410: No as this would require a module to work with http. I decided it't be better to have it included in tools/
(09:53:40 AM) gac410: Okay.   that works too.
(09:54:54 AM) gac410: As long as sites that *require* platform provided packages as specified by their IT Overlords,  can prevent cpanm installed code from "slipping in",   I have no objections.
(09:55:01 AM) vrurg: They made a smart thing of bundling every cpanm dependency into the script's body. So, if I don't have perl binary to execute then I simply extract the module part from the script, eval it and run it manually.
(09:55:36 AM) gac410: "they made a smart thing" ... .  I lost the context somewhere
(09:55:54 AM) vrurg: 'They' is related to cpanm creators. 
(09:56:02 AM) gac410: oh, right. 
(09:56:06 AM) vrurg: It's because I started typing earlier.
(09:56:14 AM) gac410: y,  sometimes hard to follow.  
(09:57:19 AM) vrurg: The only issue which worries me is that sometimes CPAN rejects http connections several times in a row blocking installation of a dependency. Then the whole thing fails and the script has to be re-run. 
(09:57:40 AM) gac410: Where we ran into issues in the past, was cpan installers that insisted on:  interactive prompting,  or external access during tests.     And some sites that block all external access.
(09:57:57 AM) vrurg: Don't know if it's an issue of vitualized environments I use for testing or general CPAN protection from DDOS.
(09:58:22 AM) gac410: One of our regular contributors here works in a site where he has to hand-carry external stuff to the wiki server -all external access blocked by the overlords.  
(09:58:59 AM) vrurg: In a jail like this there is no solution possible. Just ignore it.
(09:59:39 AM) vrurg: If they block external access – we're helpless in this case.
(09:59:41 AM) gac410: So maybe a url param that says  "don't use cpanm"   or an ENV variable,  or some way to prevent a lost cause.
(10:00:28 AM) vrurg: I'll get back to this question.
(10:01:01 AM) gac410: It's more common than you think.  Where I used to work,  we got "acquired"  and the parent co.  (an old telco)   actually blocked all external access ... period,    It was culture clash for an internet router company for sure.
(10:01:43 AM) vrurg: To complete the issues subject: the real issue is XS modules where no Perl-only variant. This is where limited hosting environments will fail and we can do nothing about it too unless the owners pre-install required modules for the user.
(10:02:29 AM) gac410: y.    The SSL code is one gotcha.   Also some plugins,  DirectedGraph,  ImagePlugin   GenPDF*   etc.
(10:02:53 AM) vrurg: But we cannot cover any possible problem. I would just don't use that kind of hosting and don't understand what keeps pepople from doing the same.
(10:02:54 AM) gac410: Probably SSL for email is the worst one.  some of the core crypt modules require openssl.
(10:03:13 AM) gac410: Sometimes again it comes from above.  
(10:04:49 AM) vrurg: As to "don't use" option. Generally, the starting script is using 'first run' option of the module. It means that the module will auto-check for deps if there is no perl5 subdir and there is not .checksum file in it. Then it assumes that it's the first time the script is run.
(10:05:16 AM) vrurg: Only then it tries to install the deps.
(10:06:55 AM) vrurg: It also tries to isntall deps of installed extensions but doesn't consider them as 'mandatory'. I.e. if a required dependency of an extension doesn't install – it's not a problem, unless the extension is pre-declared as 'required'.
(10:06:57 AM) gac410: So a statement to the "Enterprise customer"  "To prevent auto-installation of dependencies,  you should ... <do this thing>,  and then ensure all dependencies are installed as listed in ..."  
(10:07:26 AM) vrurg: Yes, that's the best approach.
(10:07:56 AM) vrurg: For now as the implementation is more of a demo there is no such option but adding it is only a matter of 10mins.
(10:08:02 AM) gac410: I could see someone being fired for installing cpan mods outside of a managed package env.    So the question is  "What is <do this thing>"
(10:08:08 AM) gac410: Ah... okay  :D
(10:09:18 AM) vrurg: Actually, it would be possible to use a local mirror of CPAN with security patched modules in it. 
(10:09:31 AM) gac410: btw ... to get your dev. proposal to bubble up to "needing a decision",  you should set a commitment date. 
(10:09:48 AM) gac410: Yes indeed.   Or even "DarkPAN" 
(10:10:03 AM) vrurg: What is DarkPAN?
(10:10:16 AM) gac410: That's an offline CPAN repo ... I think
(10:11:49 AM) vrurg: Well, if it's an easy-to-install thing – yes. Still, those capabilities are side-effects of the main idea: I want to lower the entry threshold for new users.
(10:12:27 AM) gac410: Right.  I'm *not* suggesting we use darkpan,  just commenting that it's a way to offline mirror cpan. 
(10:12:28 AM) vrurg: It's scary thing to think of how many of them tried to start foswiki and quit because it's a helluva lot of preparation before it gets ready.
(10:13:48 AM) vrurg: Of course, 'cpanm Foswiki' or 'pkg install foswiki' are the best options here but not before someone changes the whole build process.
(10:14:43 AM) gac410: For quick, kick the tires,  we have our VM image.  which works quite well.  
(10:15:15 AM) gac410: More than the build process.   We need to separate out configuration and user data from code and "shipped" content I suspect.
(10:16:04 AM) vrurg: It's not always possible to deploy it. Or, as in my case, I have a server where I can test pretty much anything.
(10:16:59 AM) vrurg: Ok, it's even more in it that than I thought. 
(10:18:20 AM) gac410: At least I think our design of intermixing stuff in the System web, with installed extensions,  and Users in the Main web,  etc.    is not conducive to good packaging.
(10:19:32 AM) vrurg: Definetely not. 
(10:20:40 AM) vrurg: Modification of topics shipped in a package is the upgrade hell I would nobody wish to come thru.
(10:20:51 AM) gac410: Maybe a "Union FS" concept   where webs consist of  empty directories that overlay System/distro,  and for Main & Sandbox,  create them during the install from templates.
(10:22:12 AM) vrurg: Sounds good but the usual question: who would develop it? :(
(10:22:16 AM) gac410: And Foswiki is completely structured around "modification of topics shipped in a package",  something that dev's have attempted to mitigate by using templates, conditional includes,  etc.
(10:22:28 AM) gac410: vrurg,  nobody ... that's the issue.  
(10:23:17 AM) gac410: I mad a proposal ages ago to consider shipping topics in a distro container of some sort and copying them in on first run.   but that got objected to 
(10:23:48 AM) vrurg: This is why I wanted to make it easier to do a first start of foswiki. It's a hope for 'more users – more developers' cycle.
(10:24:08 AM) gac410: https://foswiki.org/Development/ShipCommonlyTailoredTopicsInConfigure
(10:27:08 AM) vrurg: Quoting 'kung fu panda': "Don't tempt me!" ;)
(10:27:55 AM) vrurg: I'll look at it. But as it's a topic of bigger subject I think the focus must be on things we can do now.
(10:28:12 AM) gac410: Reviewing my proposal and the comments,  it's an area that really needs more consideration,   review with the community.
(10:28:55 AM) vrurg: I'll look at it later today. Time to do some work while my people in Ukraine are still in the office. :)
(10:29:29 AM) gac410: As Sven mentions,  we have multiple ways already implemented to attempt to solve this issue.   ViewTemplates,  %INCLUDE{ topic,  fallbacktopic }% ...  Default* topics,  
(10:29:30 AM) vrurg: BTW, it set the commit date on the proposal as of today.
(10:29:40 AM) gac410: Okay  thanks.
(10:29:53 AM) vrurg: cu later today! Thanks!
(10:30:00 AM) gac410: btw.   Did you see my checkin for json tests.
(10:30:11 AM) gac410: Still finding some issues.  :(   
(10:30:25 AM) vrurg: Yes, I've seen the commits and your IRC notes. But got no time for replying.
(10:30:48 AM) gac410: As I turn on tests,  run into cases that not handled.    No problem / no hurry.  Just stuff we need to deal with sometime :D
(10:31:00 AM) vrurg: I'll be back later on the main channel.
(10:31:31 AM) gac410: I should have added to the todo list.     Also, I could not get the Net:: email code to connect to gmail.    Works on master though.
(10:31:41 AM) gac410: Didn't spend time debugging.
(10:32:17 AM) gac410: Okay everyone.   Thanks for attending.  I'll add this to a release meeting topic.   Next meeting November 14th 
(01:54:40 PM) MichaelDaum_ left the room (quit: Quit: quit).