Release Meeting: 27 July 2015

(09:00:16 AM) jomo entered the room.
(09:00:35 AM) MichaelDaum: hi everybody
(09:00:45 AM) gac410: Hi everyone ...    Welcome to Foswiki 2.1 kick-off
(09:00:48 AM) jomo: hi
(09:01:07 AM) MichaelDaum: gac410, how about discussing 2.0.1 first
(09:01:23 AM) gac410: yes indeed.  First on the agenda. 
(09:02:01 AM) gac410: I've got the RC built  but have not installed it yet on foswiki.org.    There are 168 file changes between 2.0.0 and 2.0.1
(09:02:23 AM) MichaelDaum: in sum, what are the issues fixed?
(09:02:50 AM) gac410: There are 20 fixes and 3 minor enhancements.    
(09:03:05 AM) gac410: Most important are the bulk_copy fixes  
(09:03:05 AM) MichaelDaum: does it fix missing {Module} entries in LSC?
(09:03:16 AM) andreli entered the room.
(09:03:23 AM) gac410: No task on that one.   I've never encountered that unless you pseudo-install
(09:03:50 AM) MichaelDaum: Ive installed a fresh 2.0.0 and ran thru normal configure. 
(09:04:00 AM) MichaelDaum: I then installed more plugins using unzip on the local filesys
(09:04:10 AM) MichaelDaum: then ran configure to enable those plugins
(09:04:17 AM) MichaelDaum: which all went fine ... alas
(09:04:18 AM) gac410: Oh.. unzip....     do we still support htat.
(09:04:35 AM) MichaelDaum: the error.log listed lots of {Module} entries being guessed
(09:04:38 AM) gac410: I did change core to warn but operate ... uses a sane default if the {module} is missing
(09:04:48 AM) jast: side note, I went through web-based extension install again today... installed a plugin that I knew had no spec issues, and the install went through and told me to hit 'save' to activate the defaults. save wasn't visible, though.
(09:05:07 AM) Lavr entered the room.
(09:05:14 AM) MichaelDaum: ... which is fine. I just removed the warning msg for now as there is no penalty in not having them in LSC
(09:05:23 AM) gac410: If we don't get tasks opened,  things will never get fixed. 
(09:05:26 AM) Lavr: Lurking while being on a conference call ;-)
(09:05:37 AM) gac410: okay   Welcome back Kenneth !
(09:06:00 AM) jast: I did open a task :-)
(09:06:07 AM) MichaelDaum: also: lots of plugin defaults werent propagated from their Config.spec into LSC ... with lots of error reports all over
(09:06:16 AM) jast: it's the one about install UX in general, I believe it covers that
(09:06:33 AM) MichaelDaum: jast, this one needs splitting up
(09:06:34 AM) gac410: Michael  if you don't use the installer,  the config will be a mess.
(09:06:56 AM) MichaelDaum: gac410, when did we decide not to support unzip?
(09:07:10 AM) MichaelDaum: I would have cried out loud
(09:07:27 AM) andreli: so would I
(09:07:46 AM) andreli: still using unzip to install plugins
(09:07:47 AM) MichaelDaum: my clients get their update via git push to their repo
(09:07:51 AM) gac410: Probably an oversight,  but this is the first time anyone has brought it up.  It's been that way since the new configure was built back before first beta
(09:08:15 AM) MichaelDaum: gac410, thats why it is called 2.0-oh-oh
(09:08:16 AM) jast: so we need a way to apply defaults
(09:08:36 AM) JulianLevens entered the room.
(09:09:01 AM) jast: probably not big deal implementing that, but the code is a bit difficult to follow at times
(09:09:11 AM) gac410: Michael, That's also why we run alpha, beta and RC's    ...   For defaults Run installer.   That's the only way currently.   we need a blocker for 2.0.2 
(09:09:26 AM) MichaelDaum: okay
(09:09:33 AM) MichaelDaum: I am all for rapid patch releases!
(09:09:34 AM) gac410: The bulk_copy is too important to get out IMO  ... 
(09:09:39 AM) MichaelDaum: agreed
(09:09:51 AM) jast: yeah, we ran with the current bulk_copy today and it skipped only part of the data this time ;-)
(09:10:11 AM) gac410: Still skips data ???   yikes.    
(09:10:42 AM) jast: it's on our list for investigating
(09:10:47 AM) gac410: jast what were the conditions for that.    IMO that's a blocker.
(09:11:32 AM) jast: don't know exactly, another guy in the office did it
(09:11:49 AM) gac410: for those using unzip,   I think that running    tools/configure -save     will merge in the Config.spec files.    But needs checking
(09:11:55 AM) jast: at first glance, nothing extra special
(09:12:33 AM) jast: the copy in distro master is up-to-date, right? (from 6 days ago)
(09:12:45 AM) gac410: bulk_copy will definitely skip.    System web.    Any subdirectories of an attachment dir,    and any attachments hidden behind corrupted meta
(09:13:08 AM) gac410: master is up to date.   I build 2.0.1-RC1 from it last night.   But not loaded anywhere yet.
(09:13:33 AM) jast: I believe the only special thing about the attachments was that they had the H flag
(09:13:50 AM) jast: not necessarily all of the ones that got skipped, I only looked at this very briefly
(09:13:58 AM) gac410: They definitely should have been copied. ... the tool even logs that it is copying a hidden attachment.
(09:14:18 AM) jast: according to the guy who ran it, the script claimed to have copied files that it didn't actually copy
(09:14:29 AM) gac410: oh.. .that's not good.
(09:15:31 AM) gac410: darn.   IMO that's enough to block 2.0.1 ...  please open an urgent task.   
(09:18:11 AM) gac410: The current contents of 2.0.1 is http://trunk.foswiki.org/System/ReleaseNotes02x00#Foswiki_Release_2.0.1_Details
(09:18:23 AM) gac410: Okay ... anything else regarding 2.0.1    
(09:19:00 AM) gac410: And also someone please open a task about installing from zipfiles.   thanks.
(09:20:26 AM) gac410: Anything else  2.0.1 ...  going  .. going    
(09:20:31 AM) MichaelDaum: hm
(09:20:43 AM) MichaelDaum: not much feedback on http://foswiki.org/Tasks/Item13525
(09:21:01 AM) MichaelDaum: I am probably the only one to run nyt profile on foswiki ?
(09:21:53 AM) MichaelDaum: theres a script attached to the item to ease the process.
(09:21:59 AM) andreli: It's a while back we ran nyt.
(09:22:09 AM) MichaelDaum: please do run it and see what happens on your platforms
(09:22:10 AM) jast: we're running nyt on our configuration, but haven't managed to make it play nice with 2.x so far
(09:22:13 AM) gac410: I have not done it.  I did make one change for 2.0.1 that should have improved TipsContrib, by restricting the searches.
(09:22:28 AM) andreli: We concluded, the performance is mor CPU limited then IO
(09:22:49 AM) jomo: andreli: +1
(09:22:56 AM) gac410: Just general feel,  2.0 seems much "crisper" than 1.1. was
(09:23:07 AM) MichaelDaum: true.
(09:23:12 AM) jast: we managed to shave off >95% of overhead seen on a wiki with huge LDAP by patching LdapContrib ;)
(09:23:48 AM) MichaelDaum: the question on Item13525 is not about LDAP
(09:23:53 AM) MichaelDaum: it is about System.WebHome
(09:23:58 AM) MichaelDaum: what does it take to render it
(09:24:01 AM) jast: I know, it was meant as a side note
(09:24:26 AM) andreli: jast: Is your LdapContrib public?
(09:24:27 AM) gac410: The TipsContrib had a rather poor search.   It should be better now.
(09:24:39 AM) gac410: Used on System.Webhomme
(09:24:41 AM) MichaelDaum: it should be removed from System.WebHome
(09:24:46 AM) MichaelDaum: ^Tips
(09:25:13 AM) jast: andreli: it is, but we removed some features to make it faster
(09:25:27 AM) MichaelDaum: EARCH is the no.1 resource drain on System.WebHome ...
(09:25:32 AM) MichaelDaum:  % SEARCH is the no.1 resource drain on System.WebHome ...
(09:25:40 AM) jast: SEARCH almost always is the #1 resource drain :)
(09:26:04 AM) gac410: And the search was particularly bad.   Un-anchored *TipTopic* search across System and Main webs.
(09:26:17 AM) MichaelDaum: 2nd finding: readFile("foobar.txt") was called repeatedly for no good reason
(09:26:46 AM) MichaelDaum: 3rd finding: Foswiki::Store::Rcs::Handler::getRevision($version) ignores the $version argument
(09:27:34 AM) gac410: That sounds like 3 tasks.     #2 is a feature for 2.1 ... a new low level cache.    #3 ... is it a bug.   Needs a unit test.  what are side effects?
(09:27:56 AM) andreli: Heavy searches over seldom changing public content: Cron over to Topic with %SEARCH -> write output in a new Topic -> include the new topic
(09:28:04 AM) MichaelDaum: dunno. it seems to be an api+docu error. 
(09:29:12 AM) MichaelDaum: this needs Cdot
(09:29:38 AM) gac410: So searches.  The point of TipsContrib is a randomly changing tip.   caching it rather makes it less useful.   But System.WebHome ... how often is that used?
(09:30:06 AM) MichaelDaum: it is visited by everybody looking up Documentation
(09:30:09 AM) gac410: y.    we need separate tasks for these things.  
(09:30:18 AM) gac410: And tips can be useful. 
(09:30:28 AM) gac410: Though ours are a bit stale  :)
(09:30:37 AM) MichaelDaum: y
(09:31:09 AM) MichaelDaum: well back to the real issue in Item13525
(09:31:30 AM) MichaelDaum: I counted 118 readFile() operations readign System.WebHome
(09:31:33 AM) gac410: I'd say this is another feature request.  Redesign TipsContrib to not be resource intensive.   (Maybe cache complete list in a single topic and random pick from witthin the topic.
(09:31:58 AM) gac410: You've implemented a new feature as a solution.  A new low level cache.   2.1
(09:31:58 AM) MichaelDaum: PatterSkinNavigation.txt opened 18 times approx
(09:32:01 AM) andreli: An other approach for Tips: Cache the complete list and create the randomness in Javascript.
(09:32:44 AM) MichaelDaum: andreli, can you come up with a patch?
(09:32:47 AM) MichaelDaum: good idea
(09:32:50 AM) jast: I'm not sure caching all file reads is correct
(09:33:09 AM) gac410: So to wrap up 2.0.1.    We have a possible blocker.   bulk_copy lies.   jast,   can you focus on getting that documented.    That one will block 2.0.1
(09:33:26 AM) jast: will do
(09:34:18 AM) gac410: caching low-level reads.   Another new feature.   Needs a request, if jast is raising a concern.   Needs discussion.
(09:34:18 AM) MichaelDaum: especially people with their data dir on a remote filesystem will suffer from Item13525
(09:34:55 AM) jast: I haven't thought about it properly, so I'm not formally raising a concern
(09:34:56 AM) gac410: And RcsHandler ignoring version.   Needs a task, and if possible a unit test,   and/or review with CDot.
(09:35:33 AM) jast: just going with my general experience that caching adds just as many problems as it solves ;)
(09:35:41 AM) jomo: jast: +1
(09:35:45 AM) jomo: :)
(09:36:09 AM) gac410: thats why I'm pushing that it needs to be a feature for 2.1,  and not a quick& dirty tweak to 2.0.x
(09:37:11 AM) gac410: Okay...   Other than Jast's possible blocker to 2.0.1    Anything else before we move to 2.1  development process?
(09:38:02 AM) MichaelDaum: http://foswiki.org/Development/PreventRedundantFileReadsInStore
(09:39:00 AM) MichaelDaum: to all: please run NYT yourself before and after applying this patch
(09:39:06 AM) MichaelDaum: add your findings, please
(09:39:36 AM) gac410: Summary.  Needs tasks  for   zipfile installls (bug 2.0.2),  bulk_copy issues (blocker 2.0.1)    rcs api: possible (bug 2.0.2 unless proven urgent) 
(09:39:43 AM) MichaelDaum: I'll keep the item branch in sync with master in the meantime.
(09:39:51 AM) gac410: thanks Michael
(09:41:04 AM) gac410: So for 2.1 ... JulianLevens   Item13135 branch... seems to be stale from .gitignore changes ...   we can drop that one
(09:41:15 AM) JulianLevens: y
(09:42:17 AM) gac410: HTMLtemplates branch.    really needs firming up of spec in http://foswiki.org/Development/ReduceImpactOfCGIDotPMinFoswiki   and a new task.
(09:42:45 AM) MichaelDaum: y
(09:43:01 AM) MichaelDaum: let's try an impl and see where we get with that one
(09:43:49 AM) gac410: Since neither feature branches are ready to merge into 2.1   I'm suggesting we delay doing the branch for 2.0.x maintenance. and stay on master for a bit longer.
(09:44:39 AM) MichaelDaum: yea. no reason create a 2.1 branch next to master atm. 
(09:44:59 AM) MichaelDaum: best would be to only have two active branches at any time
(09:45:17 AM) gac410: andreli: MichaelDaum:   regarding zipfile install.  would it acceptable to run tools/configure -save    as a "merge" of the Spec files,  and if missing, add the {Module} ?
(09:45:18 AM) MichaelDaum: with item and feature branches merged to them
(09:45:33 AM) gac410: Agreed.
(09:46:01 AM) gac410: Jast.    when we do decide to branch out 2.0.x  for patching,  would you be able to add it into Weblate as another branch?
(09:46:32 AM) MichaelDaum: gac410, let's call it file-level installs. unzip or git pull ... whatever is used to materialize a plugin on the filesystem.
(09:46:41 AM) andreli: gac410: fine with me.
(09:46:54 AM) MichaelDaum: best would be to fix configure
(09:47:08 AM) gac410: yes.   fine Michael.    It applies to pseudo-install as well.
(09:47:11 AM) MichaelDaum: it propagated Config.specs to LSC before 2.0
(09:47:30 AM) MichaelDaum: we basically need that behavior back
(09:48:24 AM) gac410: If you "install" while bypassing configure,   I think that running tools/configure fixes it up.   IMO patching stuff in from the GUI is fraught with danger.
(09:48:42 AM) gac410: Every time I touch it,  "bad things happen"  
(09:48:46 AM) gac410: ;)
(09:48:56 AM) MichaelDaum: not really: clicking on "Enable" for a plugin should do required steps.
(09:49:40 AM) jast: gac410: sure
(09:49:49 AM) MichaelDaum: as things are right now, configure ui does not. instead the core warns about missing {Module} entries.
(09:50:05 AM) MichaelDaum: btw an even simpler fix would be to remove the warning message
(09:51:48 AM) gac410: hm.   I can't remember.   It used to just crash.   I changed it to a warning. 
(09:52:02 AM) Lynnwood entered the room.
(09:52:13 AM) MichaelDaum: _that_ would have been a blocker
(09:53:38 AM) MichaelDaum: okay. I ran configure -save.
(09:53:52 AM) jomo: getting the bulk_copy out is imho the priority - the configure issues are fixable by admin - the bulk_copy isn't
(09:53:54 AM) MichaelDaum: now all perl hashes are uglified again
(09:53:56 AM) gac410: Okay..    again everyone.  we need specific tasks for issues.   "Catch all's"  are nice, but get overlooked.
(09:54:09 AM) MichaelDaum: and it did not create missing {Module} entries
(09:54:22 AM) MichaelDaum: so no go
(09:54:33 AM) gac410: MichaelDaum:    Then that's exactly what would happen with gui install.    configure -save runs the same save wizard.
(09:54:56 AM) jast: gui install does stage the {Module} entry
(09:55:14 AM) jast: or at least it claims it does in the installer output
(09:55:18 AM) jast: but I couldn't get it to save that
(09:55:49 AM) gac410: Yes   gui install works fine.    Gui save after a unzip install    is what I'm mentioning.     Save in gui depends upon the DOM knowing that something changed.
(09:56:15 AM) gac410: If the dom doesn't know,  then you won't be able to trigger the Save wizard.     
(09:56:48 AM) gac410: So need more tasks.     Ugly hashes   separate bug from missing module.   
(09:57:49 AM) gac410: Right.  IMO the only blocker for 2.0.1  is bulk_copy.  And it's BADLY broken in 2.0.0,  so that needs some serious attention.    Sure wish CDot was here.
(09:58:25 AM) MichaelDaum: reported as http://foswiki.org/Tasks/Item13560
(09:59:44 AM) gac410: Thanks Michael.   but re:  This especially applies to extensions installed via unzip or git,  ...   really it *only* applies in that case, right?   Hopefully using the install extension wizard is completely correct.
(10:00:10 AM) MichaelDaum: gac410, not sure
(10:00:28 AM) MichaelDaum: what I am sure of is that clicking on "Enable" does not fully enable plugins.
(10:00:47 AM) gac410: I've not seen any instance of install using the wizard, not completely configure it.  
(10:00:49 AM) MichaelDaum: which would include: (1) create a {Module} entry (2) propagate Config.spec
(10:00:52 AM) MichaelDaum: defaults
(10:01:16 AM) gac410: A premise of new configure is   ABSOLUTE.    Changing one setting  should NEVER change a 2nd setting.    Clicking enable may NOT set a module.  
(10:01:19 AM) MichaelDaum: what happend to the issue that congigure uglifies perl hashes in LSC?
(10:02:10 AM) MichaelDaum: this renders {FromTypes} and the like unmaintainable
(10:02:13 AM) gac410: Needs at task.  I have not see that.   I had opened a task about that ages ago and verified that it was fixed. So you are hitting a different path.
(10:02:27 AM) gac410: I agree.   It was a blocker and CDot had fixed it.
(10:02:54 AM) MichaelDaum: Foswiki.spec -> {FormTypes} is nice perl. LSC -> {FormTypes} is uglified
(10:03:12 AM) gac410: the normal path  using the extension_installer,  or the InstallExtension wizard    doesn't corrupt these things I don't believe.
(10:03:26 AM) MichaelDaum: this is {FormTypes} ... a core setting
(10:03:56 AM) gac410: Then his fix got broken.    It was working at one point.
(10:05:02 AM) gac410: It was a difficult fix,  and he changed Spec processing from an eval to a line/by/line interpreter / copy   to fix the issue.
(10:06:16 AM) MichaelDaum: reoported as http://foswiki.org/Tasks/Item13561
(10:06:51 AM) gac410: All I can say is please open a task.   But it won't block 2.0.1 . ...  (Thanks).. if it makes it, great.  But though very annoying,  it's  not on the scale of bulk_copy corrupting your store ... silently no less.
(10:06:59 AM) MichaelDaum: feel free to reasign it to another patch release in case you'd prefver to get 2.0.1 out without fixing these
(10:07:45 AM) gac410: y.    we need bulk_copy fixed.   That's critical.   It breaks store by missing data,  and if jast is right, lying about it no less.
(10:07:47 AM) MichaelDaum: 50 shades of ugrgent.
(10:08:42 AM) gac410: JulianLevens:   Would you have some time to further validate bulk_copy?    
(10:10:59 AM) gac410: okay ... 1 hour into meeting.   Next on agenda was 2.1 feature review.   The bottom line is our proposals system is a mess of stale stuff.   Needs a bit of a cleanup.
(10:11:23 AM) ***MichaelDaum just added a new proposal for 2.1
(10:11:43 AM) gac410: The items in 14 day need to be either voted, rejected or accepted. 
(10:12:27 AM) gac410: JulianLevens:  Your proposal http://foswiki.org/Development/AddBacklinksToQuery    I escalated that a bit and slotted it for 2.1.    
(10:12:29 AM) MichaelDaum: ... and committed developers known to have left the project need to be removed from the proposal
(10:14:02 AM) gac410: The issue is that on i18n systems,  backlinks is totally broken.   So we need a fix for 2.1, be it a new backlinks feature embedded into QUERY   or a new BACKLINKS macro
(10:14:12 AM) JulianLevens: I'll look at that and provide feedback ASAP. I know the principle but it's been a while
(10:14:21 AM) gac410: Thanks JulianLevens
(10:15:20 AM) gac410: I've fixed http://foswiki.org/Development/ReleasePlan ...  it has 3 accepted proposals for 2.1,  3 needing investigation,  and a whole mess of stuff that could be picked up.
(10:15:20 AM) JulianLevens: Hmm, do you have time frame for 2.1?
(10:15:49 AM) gac410: That's something we can discuss. ... not really ye 
(10:15:51 AM) gac410: t
(10:16:07 AM) gac410: afk - back in a moment...
(10:17:21 AM) JulianLevens: Can I ask everyone here to vote on the options within http://foswiki.org/Development/AddConcatOptionToAttrs
(10:19:31 AM) gac410: back
(10:20:05 AM) gac410: will do JulianLevens.  
(10:20:34 AM) ***MichaelDaum clicks, JulianLevens
(10:23:26 AM) MichaelDaum: I still prefer param="string1" + "string2" as well as param="string1" .... param+="string2"
(10:23:50 AM) jast: same here
(10:23:55 AM) MichaelDaum: as this is what most programming languages and i.e. perl have
(10:24:25 AM) JulianLevens: Then please vote accordingly :)
(10:24:43 AM) gac410: so that's a vote for "both" ???  
(10:25:01 AM) JulianLevens: yes that's both
(10:25:15 AM) MichaelDaum: JulianLevens, what exactly is the use case for a -=
(10:25:27 AM) gac410: okay ...  got more confused the more I read and then skipped ahead to the bottom line  :)
(10:25:45 AM) gac410: ah... prepend  vs append
(10:26:32 AM) gac410: hm    indeed why would you need to prepend,  vs just list the values in the right order?
(10:26:48 AM) jast: yes, I find -= confusing, too
(10:26:56 AM) jomo: also
(10:26:58 AM) MichaelDaum: -= for prepend?
(10:27:04 AM) jast: voted
(10:27:07 AM) gac410: That's what it says
(10:27:11 AM) JulianLevens: There are use-cases for prepend but they do tend to be contrived
(10:27:27 AM) MichaelDaum: woops my comment has been deleted
(10:27:52 AM) jast: and I'd like to note that the voting table took a while to understand :)
(10:28:14 AM) JulianLevens: The implementation allows easy addition of new operators; so prepend could be added later very easily
(10:28:26 AM) gac410: lets use comment plugin and then summarize into table.      Otherwise stepping on each other.
(10:28:30 AM) jast: oh, I've figured out why bulk_copy skipped a lot of attachments here
(10:28:47 AM) gac410: jast  wonderful   Is it bad news?
(10:28:50 AM) jast: the attachments had a leading '_', and the RCS iterators in 1.1.x (not sure about 2.x) skip filenames starting with _
(10:29:06 AM) gac410: Yes   thats true.    _ is defined as a hidden attachment iirc.
(10:29:26 AM) jast: but there's already a FILEATTACHMENT attr for that
(10:29:40 AM) jast: and you can't tell the store to not skip over _ :(
(10:29:43 AM) gac410: hidden from store,  vs hidden from user
(10:30:07 AM) jast: so I can upload _foo via the store but then not find it...
(10:30:07 AM) gac410: How did they get there?    Manual upload? 
(10:30:17 AM) jast: plugin upload using store API
(10:30:51 AM) jast: Func API, in fact
(10:31:03 AM) gac410: hm   yeah that sounds like a bug then.     I think historically     _blah was used to hide webs     like _default.     
(10:31:10 AM) gac410: but not sure about attachments
(10:31:13 AM) jast: it's specifically in the getAttachmentList function
(10:31:57 AM) gac410: MichaelDaum:  Lavr: ... any old timer info about why RcsStore  hides filenames with leading underscore?
(10:31:58 AM) jast: at least it's not an issue in bulk_copy itself
(10:32:11 AM) gac410: y.   And cannnot be fixed.    
(10:32:15 AM) MichaelDaum: gac410, my 2cent: OMG
(10:32:34 AM) gac410: easily.  ...    as it would need to patch 1.1.x source store.
(10:32:48 AM) jast: 2.x has the same behaviour
(10:33:11 AM) MichaelDaum: never knew about _underscore_attachments being treated special
(10:33:59 AM) jast: seems like the iterator is the only place that happens
(10:34:19 AM) jast: FILEATTACHMENT entries with _ prefix in the name are not skipped
(10:34:24 AM) ***MichaelDaum pictures Foswiki becoming an archeological artifact of research sooner than later
(10:35:23 AM) gac410: skips any files named with dot, asterisk or underscore prefix.       grep { !/^[.*_]/ && !/,v$/ && -f "$ed/$_" } readdir($dh);
(10:35:48 AM) jast: so how do we address this? conceivably bulk_copy could be rewritten to combine info from store iterator *and* FILEATTACHMENT entries
(10:35:59 AM) jast: but it's probably not going to be trivial
(10:37:42 AM) gac410: Not fixable in bulk copy.   It is by design using the Store API.     Better would be to patch the source store.
(10:38:39 AM) gac410: bulk_copy is intended to handle DB based stores, etc.   fully portable.    
(10:39:00 AM) gac410: If it starts digging into the file system that all falls apart.    
(10:39:51 AM) jast: sure, that's why I suggested something different ;)
(10:40:21 AM) jast: but it's still not really a change worth making for this, probably
(10:40:31 AM) jast: so, perhaps we should just document this?
(10:41:23 AM) gac410: yes.   Document for sure.    And maybe include a suggested patch.    And maybe a task consider this hidden-from-list  a possible 2.0 bug.
(10:42:31 AM) gac410: PlainFileStore behaves the exact same way:      grep { !/^[.*_]/ && !/,pfv$/ && ( -f "$ed/$_" ) } readdir($dh);
(10:43:12 AM) gac410: the "dot" i understand.  Don't want to reveal possible .htaccess files as attachments.   
(10:43:24 AM) gac410: I don't understand the asterisk or underscore restriction
(10:44:26 AM) JulianLevens: neither do I and Store.pm does not document this behaviour for eachAttachment
(10:44:34 AM) gac410: But imo this is still possibly a nasty inconsistency.   I believe PlainFile is much more tightly coupled to the attachment %META:FILE...  
(10:48:18 AM) JulianLevens: My 1st thought is to add include and exclude regex parms to eachAttachment with defaults as-is
(10:49:13 AM) jast: we're calling this through 2-3 indirections
(10:49:35 AM) JulianLevens: bulk_copy could pass more inclusive values. Still like to understand why * & _ are excluded though
(10:49:36 AM) jast: so that's a lot of changes, and probably not urgent enough to do on 01x01
(10:50:00 AM) gac410: y.   This is a mess.      And really needs a 1.1.9 patch to make a conversion work.
(10:50:34 AM) ***MichaelDaum backing off ... still listening but ...
(10:50:37 AM) MichaelDaum is now known as MichaelDaum_
(10:50:38 AM) JulianLevens: Which is the achiles heel of bulk_copy
(10:51:56 AM) gac410: Same code is in twiki.   Omits _, . and * prefixes
(10:52:13 AM) gac410: So this dates way back.
(10:53:04 AM) JulianLevens: bulk_copy should use latest code with all appropriate fixes in place - or we'll be forever applying back patches
(10:53:31 AM) gac410: Okay everyone.   Lets wrap this up.    ...  it sounds like 2.0.1 can go ahead with a very strong warning about bulk_copy and hidden files.   
(10:54:00 AM) gac410: with suggestion of a "find" command to find them  and a one-off patch to copy them
(10:55:25 AM) gac410: And for the next meeting on the 10th.   lets plan a feature review for 2.1.    I will unfortunately have to either bow out early,  or suggest moving to Tuesday.  I have a 10am Dr's  appt.   
(10:56:48 AM) gac410: We clearly have some discussion points to hash out with CDot  on configure and store.
(10:57:16 AM) gac410: Any other last minute.   Or declare the meeting closed?   
(10:57:50 AM) jomo: imho the restriction reason for the * (asterix) is - the windows doesn't allows asterix in the filenames. And probably we don't want have attachments what arent downloadable to ANY OS....
(10:58:42 AM) jomo: just guessing...
(10:58:49 AM) gac410: okay,,  but that ought to be handled in NameFilter    but good guess for sure.
(10:58:55 AM) jast: there's a whole number of other characters that aren't allowed on windows
(10:59:01 AM) jast: those are not excluded in this list
(10:59:06 AM) jomo: < > : " / \ | ? *
(10:59:39 AM) gac410: the underscore   used to be the documented way to hide attachments.   ChartPlugin  caches generated graphics using _ChartPlugin as the prefix
(11:00:46 AM) andreli left the room (quit: Ping timeout: 240 seconds).
(11:02:45 AM) andreli entered the room.
(11:03:41 AM) gac410: yeah it's a twiki historical thing.   For ex.  in the docs: 
(11:04:39 AM) gac410: Prefix the filename with an underscore (the leading underscore avoids a name clash with files attached to the same topic)
(11:05:02 AM) jomo: IIRC the ImagePlugin uses the _filename for the cached (generated) thumbnails too...
(11:05:06 AM) gac410: It's describing how to handle generated cached type files.  
(11:05:13 AM) jomo: :)
(11:06:02 AM) gac410: In theory cached files should be relatively safe to omit,  as they get recreated on view.   But if it's (ab)used for other stuff. ... oops
(11:07:33 AM) gac410: DirectedGraphPlugin ... same thing.
(11:08:08 AM) gac410: well.  old versions.  Newer code doesn't do that.
(11:08:10 AM) jomo: main problem - anyone could upload a file _something - and it isn;t a cached data. so two possibilites: 1.) deny allow upload such files and use _file for the temporary-cached files or 2.) allow underscore on the filenames, but need another mechanism to detect the temporary-caches files...
(11:09:01 AM) gac410: With the move away from file based store,   web accessible cache needs to eventually end up somewhere outside of the Store managed structure.
(11:11:00 AM) gac410: DojoToolkitContrib .. ships lots of _ prefixed files too.
(11:12:51 AM) gac410: jast,   in your testing,  if you could see what happens if the _ is removed from the getAttachmentList filter... that would be helpful
(11:13:03 AM) gac410: as a temporary patch to 1.1.9 source store.
(11:16:25 AM) jast: I tested that... the attachments get copied
(11:25:43 AM) gac410: okay great.   So we can include a suggested patch.    I suppose another option is to monkey-patch the old version  :(
(11:26:21 AM) gac410: jast ... did they end up polluting the META on the target system?
(11:29:40 AM) gac410: though that will get complex.  Multiple versions to patch :(  source could be 1.1 or 2.0   RcsLite, RcsWrap or PlainFile   
(11:30:43 AM) jast: RcsLite and RcsWrap share the same code for this
(11:30:52 AM) jast: in 1.1.x, Foswiki::Store::VC::Handler
(11:31:00 AM) gac410: okay that's good 
(11:31:09 AM) jast: same in 2.0, only it's Foswiki::Store::RCS::Handler
(11:31:24 AM) gac410: Could also be source from PlainFile 
(11:31:30 AM) jast: right
(11:32:25 AM) jast: as for "polluting the META"... there were META entries in the original topics, so "polluting" isn't exactly the right word
(11:33:34 AM) jast: in fact it seems to be copying the embedded store form with no changes whatsoever
(11:33:56 AM) gac410: okay that's fine.   I guess it will depend on the original file source.   Attached with API,  or external to the store.
(11:34:03 AM) gac410: good.
(11:37:07 AM) jast: oh, dang
(11:37:09 AM) jast: correction
(11:37:19 AM) jast: bulk_copy claims the files got copied but they're not there
(11:37:48 AM) gac410: ug.  that's even worse.
(11:38:20 AM) jast: looking into this
(11:38:34 AM) gac410: thanks.
(11:38:38 AM) andreli left the room (quit: Remote host closed the connection).
(11:39:38 AM) gac410: Also by not copying files like .htaccess,  sites that might have applied custom apache access will have a miess.
(11:39:41 AM) gac410: mess
(11:50:56 AM) jast: I can't figure this out right now
(11:51:13 AM) jast: and my mind is pretty much gone... I'll look again tomorrow
(11:51:18 AM) gac410: In the release notes, I'm adding:  +<div class="foswikiHelp foswikiAlert">%X% *Caution:*  There are known limitations to  =bulk_copy.pl=.  See the Foswiki:System.UpgradeGuide for details. Hidden files  (filenames with underscore (_) or dot (.) prefix) will not be copied.</div>
(11:51:29 AM) gac410: Okay thanks.   I'll shoot an email to CDot as well.
(11:51:48 AM) gac410: And I'm adding detailed information to the UpgradeGuide as well.