Feature Proposal: Simplify non-developer editing of release managed topics.

Motivation

RichMorin's attempt to trivially make some copy fixes to a plugin's topic shows that the "update SVN from Extensions/System web" workflow needs to be simplified.

TODO: RENAME to 'auto create tasks for RM-managed topics and open edit to all registered users'

Description and Documentation

Ideally, everyone should be able to come along to foswiki.org and edit, expecting that their changes, or something derived from them should go into the next release.

Because topics are release managed, there are technical complexities that we've never solved.

This proposal is an attempt to get some way along that path.

Workflows

user workflow should be:

1. user edits foswiki.org release managed topic, then saves.

backend automation should:

1. create or add to a task
2. record changes staged for a developer to merge and release/close.

Implementation ideas

use a git repository that tracks the topics on foswiki.org

onSavePluginHandler that calls git commit %FILENAME%

Then we add some post-processing code in a cron job that every hour looks at the git log and creates tasks for any extension that has had changes - listing who, what, where, why.

sven rejects this as more complex that necessary (but what a shame, i like git)

use a restHandler to analyse the .changes and to raise Tasks with a patch file attached.

just a simple little resthandler that creates a task and attaches a diff, run by a daily/hourly cronjob.

no changes need to be made to the running and upgrading of foswiki.org

next step might be to add code to allow developers to 'accept merge' from the task, and if the patch succeeds, it gets commited to our svn (ideally using some magic so its commited by the userid that originally edited the topic)

Impact

world peace.

WhatDoesItAffect:

Implementation

-- Contributors: SvenDowideit - 05 Mar 2012

Discussion

tbh, now that i've listed it, i might just write it this week



-- SvenDowideit - 05 Mar 2012

Sounds great, are you doing this in a plugin?

Also: will you use repo-per-module for this? I fear that taking edits in Extensions web will be hugely problematic.

Programmatically speaking, it would be easier to direct users to edit Eg. Export/JQueryPlugin/System/JQueryPlugin - rather than the version shipped in System or Extensions

Using repo-per-module, we can avoid problems affecting one extension from stalling the whole user-editing-doc effort.

-- PaulHarvey - 05 Mar 2012

Extensions will be tough because the Extensions web version has the Form, updated during the upload, and the expanded BuildContrib macros. "un-releasing" the released version back into the repo will be interesting.

On a slightly different subject, to encourage Commenting on system topics, for users who don't want to edit, or for guests, and others we don't want to open up for full access, consider the following view template: (btw needs the trunk version of CommentPugin)

%TMPL:INCLUDE{"view"}%
%TMPL:DEF{"formtop"}%%TMPL:END%
%TMPL:DEF{"formbottom"}%%TMPL:END%
%TMPL:DEF{"attachmentsattop"}%%TMPL:END%
%TMPL:DEF{"attachmentsatbottom"}%%TMPL:END%
%TMPL:DEF{"simpleheader"}%\
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
<div style="margin-bottom:1em; padding:1em; background:#99ddaa;" >
<p>%ICON{warning}% This is an official distribution version of *<nop>%TOPIC%*. and is managed by Subversion revision controls.</p>
<p>%ICON{support}% Members of the Main.DocumentationGroup can edit this document.  Anyone is welcome to submit feedback, suggestions or other comments under the Discussion tab.</p>
</div>
%TMPL:END%
%TMPL:DEF{"edit_discussion"}%<span class="foswikiRight foswikiButton"><a href='%SCRIPTURL{"edit"}%/Sandbox/%BASETOPIC%?t=%GMTIME{"$epoch"}%%TMPL:P{"url_param_editaction"}%%IF{"context TinyMCEPluginEnabled" then="" else=";nowysiwyg=1"}%' rel='nofollow' %MAKETEXT{"title='Edit the discussion' accesskey='e'>&Edit Discussion"}%</a></span>%TMPL:END%
%TMPL:DEF{"bodyclassname"}%patternViewPage%IF{"'TOPIC' allows 'CHANGE'" then=" foswikiHasChangePermission" else=" foswikiHasNoChangePermission"}%%IF{"context authenticated" then=" foswikiLoggedIn"}%%IF{"defined raw" then=" patternRawViewPage"}%%TMPL:END%
%TMPL:DEF{"content"}%
%TMPL:P{"simpleheader"}%
%TABPANE{select="%URLPARAM{"tab"}%"}%

%TAB{"Documentation"}%
%TEXT%
%TMPL:P{"form"}%
%ENDTAB%

%TAB{"Discussion" id="discuss"}%
%TMPL:P{"edit_discussion"}%
---++ Feedback and suggested changes
%IF{"istopic 'Sandbox.%BASETOPIC%'" then="$percntINCLUDE{\"Sandbox.%BASETOPIC%\"}$percnt" }%
%COMMENT{target="Sandbox.%BASETOPIC%" type="returntab"}%
%ENDTAB% 

%IF{"%QUERY{"length(attachments[NOT lc(attr)=~'h'])"}% > 0" then="
$percntTAB{$quotDownload$quot}$percnt
$percntTMPL:P{$quotattachments$quot}$percnt
$percntENDTAB$percnt
"}%

%ENDTABPANE%

%TMPL:END%

-- GeorgeClark

er, no thats not what i mean.

the reason we've never solved this, is because we add all these complex things that are not what the end user wants.

everytime someone has gotten up the courage to ask about it, they've wanted to edit the topic that is there. not some complicated future release, not add a comment, when all they wanted was to fix a simgle >sic< selling mistake. just edit as they did on their own wiki.

so all i'm proposing, is to track the topics and attachments using git (one day we can talk about replacing rcs..) so we can simplify generating changesets.

and in the process of saying that, its obvious that git is gratuitous, and that all i need to write, is a restHandler that analyses the .changes and makes tasks for us to merge into the next release. (updated proposal)

-- SvenDowideit - 05 Mar 2012

Not sure if this first is objection-worthy ... in principle I like the proposed solution, however I can imagine trying to figure out the why of some simple changes, and/or tracking them backwards to existing tasks. Do we have a plugin that can require comments during save?

• (mandatory) Reason for change:
• (optional) Related Tasks:

For example, Item11606 was just opened. It explains that the documented $user format token should really be $username. Without comment and context, the developer will have to figure out why the change was made. A simple save during comment "$user doesn't expand, $username does. See example below" would make all the difference. Maybe the handler that processes the changes would push into the open "Documentation Task" a list "Topic, Modified-by, Timestamp, (Task), Description, Merged(no)" So the developer handling the changes updates SVN and sets the Merged flag.

And I still believe that similar to Developers, we should authorize documentation changes rather than allow any registered user to edit. Yes we should invite changes and make it easier, but at least initially some level of throttle is needed. Maybe it's a much lower bar. (Just ask). But something that might discourage spammers to have a go. If we make the model - everyone can edit, just lock out the bad guys, we'll just get more spam registrations.

I'd also like you to address the foswiki.org upgrade process. We still should re-sync back to the tarball version as part of installing new releases. It's an important part of assuring reliable releases. I'm concerned that what is currently a simple process of overlaying System web with the tarball version will turn into hours of merging. Currently I have to check every diff of the released tarball file with the web version. TOPICINFO changes are bad enough, but I really don't want to see us complicated the web upgrade process too much. If we go this way, I think we need:

• Some way to turn off the updates while preparing for the upgrade
• A commitment that the updates will not all be left to be handled during the site upgrade.

Ideally, I'd prefer that a diff of the System web against the tarball shows no modifications at the time of the upgrade.

The more I think about the time it takes to do the upgrade, even with the small number of modified topics that developers edit already, I'm raising an objection. When someone installs an updated extension onto foswiki.org, there can be hundreds of differences to resolve. Ugh.

And how will this be resolved if the Extensions Installer is used to install an updated release of an extension. Yet the VarBLAH topics have been edited. Currently the installer will just overlay all the topics.

Actually if you think about it. We ship the System web update restricted to the AdminGroup, and strongly advise everyone to not update topics in System, so the upgrade process isn't a nightmare. I think turning this on will turn foswiki.org maintenance into the nightmare that we advise against. An upgrade with an un-modified system is a 5 minute untar of the -upgrade- tarball. Modified topics are just overlaid. With the Colas script, it's 10 minutes - verifying any diff's when there are typically only a small handful.I think there were 6 files I had to check to install the 1.1.5-Beta1. If we turn this on:

• Every changed topic, even when up-to-date will have a TOPICINFO difference
• For each difference, need to verify
  • Was the change incorporated
  • If not, was it rejected, rewritten, or missed.
• And then decide:
  • Keep the original (and foswiki.org is not a complete representation of the "beta" or RC)
  • Keep the new (any changes will be in the rcs history)
  • Or manually merge, and then sync back to SVN

-- GeorgeClark - 05 Mar 2012

In a rather lengthy IRC discussion, I learnt that Sven clearly favours editing foswiki.org/System 'in-place', and tracking foswiki.org's data/System directory in a dedicated repo unrelated to any of the core/plugin/foswiki repos.

I tried to argue the merits of editing a separate web (the most trivial implementation would result in edits of something like "foswiki.org/Export/FooPlugin/System/FooPlugin" rather than "foswiki.org/System/FooPlugin"), where users would be editing the actual .txt in a dedicated branch of the module repo. We could push those to github; Eg. you could go to github.org/foswiki/core and checkout a 'Release01x01-userdoc' branch which contains the edits from the "Export" web.

But we both agreed Sven's current proposal is simpler to implement and is a worthy first-pass.

I think what should happen is:

• whoever upgrades foswiki.org, simply tags (or runs a script which tags) the repo tracking foswiki.org/System as "prior to upgrade", or something.
• And then simply dump the release upgrade on top, obliterating the changes users have made.
• And then commit the result.

The point is we'd still have the user's edits in those commits, even if the HEAD of the branch has lost them.

I am wondering if Sven intends to make convenient links to user edits (as per the trac integration we have now with Foswikirev) on the tasks it generates.

As for comments, I guess this means installing RevCommentPlugin, and perhaps Sven's plugin could use its commit message as the commit message to the git repo.

I guess it's possible to do an afterSaveHandler which restores the original TOPICINFO line, but that sounds hugely hack-tastic.

Perhaps it would be better to fix BuildContrib so it tries to cook TOPICINFO into something acceptable, at build time.

-- PaulHarvey - 05 Mar 2012

TOPICINFO updates done on the wiki can just be ignored when merging into SVN. We can just skip the first line when calculating the diff to report to the developer. And buildcontrib is probably okay. IIRC it does handle TOPICINFO correctly, forces rev 1, etc. The bigger issue for me was that changes to TOPICINFO causes a "false positive". It meant that the site update needs to stop to review the changes.

Actually if the upgrade just ignores local changes and simply overlays all existing topics, the displayed rev info will flip to "UnknownUser" and the old version will be obtainable from RCS. So actually that may not be a bad approach. I'm not sure we even have to tag the topics as "Prior to upgrade". The act of replacing the topic makes it newer than the RCS file, and identifies it as replaced, and rev -1 is the version prior to upgrade.

So maybe this works out even better. Starting from a base, no changes, every System web topic is rev=1 "ProjectContributor"

• Over time, topic is updated
  • Transaction logged for possible merge (comment, rev, by ...)
  • Developer merges change into SVN/GIT
  • Repeats: updates, merges ...
• Next "Release" is built from SVN and installed
  • Copy in files ignoring any local changes. They are lost in the "cache" .txt file but remain in the .txt,v database
  • The .txt,v file becomes a full history of the topic across releases and minor updates. That's another good thing.
• Next "edit" occurs - now using newly installed release files as the base
  • Change is logged
  • Diff generated - cumulative System .txt file against SVN .txt file

I'm removing my objection. I believe this is even easier. I get to update without even caring if changes are made. Crawford's changes to ignore TOPICINFO and detect that .txt file is newer than .txt.v file makes this possible.

-- GeorgeClark - 06 Mar 2012

oh dear, and then we can think about using this cleverness for upgrading distributed topics that a user has modified - like SitePreferences.

(ok, that will be a separate proposal_

-- SvenDowideit - 07 Mar 2012

Developer is no longer contributing. Changed status to 'ParkedProposal'

-- Main.CrawfordCurrie - 11 Jan 2016 - 18:00