Foswiki NOT Heading For Doom (Maybe)

Below are the opinions of one apparently disgruntled developer. Is the Foswiki "glass" half full or half empty? As the Release Manager who took over from Kenneth and has managed the releases starting with 1.1 or thereabouts, I don't agree with many of his assertions. The project is indeed a bit more focused on internals right now, but the 20-year-old core has reached the point where it needs a refresh. The new development is exciting and promising. And we are thrilled to have had the contributions from several new developers over the past year or so. Foswiki 2.0 and beyond would not have been possible without the efforts of new contributors working with some of the "old guard" for the next generation of Foswiki.

I would like to explain very clearly why I am no longer contributing to this project and why I think decisions are being made at the moment that makes the project moving towards its own destruction.

To anyone not knowing who I am

• I am Danish and 51 years old (2016). I have worked for Motorola Solutions for 28 years.
• I manage a very busy Foswiki installation in Motorola Solutions and I also have a private one on which I run a couple of open source projects.
• I was release manager on TWiki until we forked
• I was part of the fork to Foswiki
• I was release manager on Foswiki on all the 1.X releases for several years
• I was testing and testing for many hours a day and spent long nights. I have a day time job and all work I did was in my free time at nights and weekends.
• I am not a programmer. I am an educated electrical engineers and I have 28 years experience in radio design and in project management and department management.
• I am able to find bugs and fix them in code that already exists.
• And I am have actually coded some pretty complicated plugins with great value. It took me probably 10 times longer to do than it would have taken for a pro programmer. But I did it. I can program basic Perl. I know a little C. I have zero knowledge of Javascript and Java and have no experience is object oriented languages. The only OO I know is from the Foswiki Perl code and mainly by example.
• I had to take a break from Foswiki because I was getting sick from not having enough sleep and I was getting frustrated with always having to convince programmers not to break compatibility. I just could not take the pressure any more.
• I made a come-back and spent significant time testing the 2.0/2.1 releases. And that was OK.
• And I have been updating and releasing plugins all the time. My DelayMacroPlugin, ExpandTopicContentPlugin, ExtendedWebListPlugin, FieldHistoryPlugin, ModifyLoginPlugin, MultiSearchPlugin, MultiTopicSavePlugin, SetFormValuesPlugin, and TimeCalcPlugin are plugin I created and maintain. And in addition I have done two plugins that are not shared because they related to some internal IT systems and they have zero value unless you have these systems.
• And I have also been maintaining some old TWiki plugins that we converted over.
• So even though I am not a pro programmer I have managed to write quite many lines of code on this project. The current plugin framework makes it not too hard to write straight forward Perl without knowing any object oriented programming. That has been a strength of TWiki and Foswiki. Any idiot could make quite powerful extensions with very little knowledge.

So what am I so concerned about?

• Lately the project has not has much focus in adding new value to the endusers.
• We have had a lot of focus on improving the installation. The admins hopefully likes these improvements but to the end user this is totally invisible.
• We have added a nicer Raw Wiki Text editor - which was mainly to include Michael Daum's wonderful work in the NatEditPlugin
• We added a nice syntax enhancement to macros
• We added the indent syntax
• The code runs a little bit faster (after we removed some nasty bugs that made it slower)
• UTF8 support added. Important. But the upgrade from iso to UTF is still to this day a pain requiring hours of manual work.

  I would love to hear a proposal of how to make this better. The old store made no attempt at keeping the data iso* "clean". If users paste in crap, then it's crap, and needs manual resolution. It's one-time pain that will hopefully not happen again. -- GeorgeClark

• EditRowPlugin was added (after a lot of bug fixing from bugs that I found).

Looking at how many years this took, it wasn't really that much that was added

I guess this all depends upon your perspective, and your user requirements. If your primary language is non-latin, then Foswiki has become much more usable. (Unfortunately we can see that by the increasing amount of non-latin wiki spam that's been hitting us.) So for your use case not that much was added. But for other regions, Foswiki has finally become useful.

What has not been done

• Wysiwyg Editor based on TinyMCE Editor has not been maintained at all. It is buggy. Especially when people copy/paste from misc sources a lot of garbage ends up in the topic. TinyMCE has been developed a lot since.

  I've tried once to upgrade and had to revert. TMCE is rather a thankless task. And the TMCE project broke backwards compatibility. It needs a developer who uses Wysiwyg to work on Wysiwyg. Whenever I've touched it I've only caused pain. -- GeorgeClark

• Tables are still a pain. EditRowPlugin did not really make a difference we has hoped. After a year I can see that the users hardly use it. What people need is a GUI to create tables in Wysiwyg and set columnwidths by dragging the dividers. AND then having this saved as widths in a TABLE tag so you maintain the TML format of the table. A lot of wikiapps depend on tables being in TML and not HTML

  Tables are indeed an issue. HTML is much richer than TML. It's a huge challenge to make tables round-trip through an HTML editor without loss of the fancy markup. I have no idea how to fix it. Needs a developer to figure out something creative. We have to preserve TML or we break wiki applications.

• User management is a pain. Removing a user when a person leaves the company requires hacking files and even then there is really no good way to do it.

  Not True! Take a look at ManagingUsers. It has a dialog & REST handler which removes a user, removes them from Groups, and even finds and kills any active CGI Session files. Been in the code since 2.0, and was previously implemented in AntiWikiSpamPlugin.

• People constantly ask for ways to make simple drawings in the pages. Small flow charts. Timelines. Mostly boxes and arrow and lines and a little text. Nothing fancy. The only offering we have it JHotDrawPlugin which was OK but it depends on Java applet running in browser and we all know what is happening with that in all major browsers. We have nothing to replace it.

  Not part of core. It's a migrated / abandoned / orphaned plugin. Either needs a developer who wants a replacement or a business willing to fund it's replacement. Many of these features were done as paid development by businesses with a need.

• We have more than 100 feature proposals in the queue.

  Yep, sure do. Several of them are yours, and I've asked several times if you plan to do them for 2.1, 2.2, etc. I'd like to get them onto the Release Plan. but if they sit in the queue with a commitment, but no work, not much I can do.

What is the project working on?

• All the now very stable Perl core is being rewritten totally using a CPAN lib which is not standard in the Redhat repository. Total rewrite!
• Best case we end up with Foswiki being able to do the same it does today.
• Worst case we break all sorts of things we do not see until it hits the endusers. I expect 200-300 bug reports with all the changes made and noone to beta test. I for sure will not expose my users to this major rewrite
• And last - now I discover by chance - in a checkin note - that we plan to DEPRECATE THE ENTIRE EXTENSION API (Foswiki::Func) as part of the rewrite

Addressing these 4, The Object Oriented branch is a (very promising) experiment being done by a new developer who probably would not be contributing other than scratching this "itch". It is not currently on our ReleasePlan. It started out with the premise that:

1. The Foswiki core dates from nearly 20 years ago, and in some areas is nearly unmaintainable.
2. It's procedural. It's spaghetti, It's been layered with patches and kludges.
3. New OO trained developers don't want to work on it. And there are areas where even experienced dev's are afraid to tread.
4. 100% backwards compatibility has been a boat anchor dragging down any hope for modernization.
5. We are completely unable to take advantage of well proven reliable code frameworks like PSGI / Plack.
6. There are areas that we just cannot fix because of the architecture. Modern authentication mechanisms for example. Our user mapping code is very convoluted.

So, we agreed - let's see where a modernization of the core could go if we cut all the roadblocks. We can most likely add back in a compatibility layer AKA the old TWikiCompatibilityPlugin. But that comes after the restructuring is complete. So far the rewrite really is looking promising. If we don't try, we certainly can't succeed.

• Most of the core coders seem to have left the project

  I don't think we should be asserting any underlying cause from "most of the core coders" leaving. In the first place I don't think it's true. There have indeed been some attrition, but several important losses have very clearly been job changes, layoffs, family changes, etc. Life does go on, and every organization has some attrition over time. And yet we have gained some new developers too. The OO rewrite is one of note. All of the UTF-8 / Unicode was driven by a new developer (who has stepped away again waiting for PSGI/Plack), working with one of the old timers. So there indeed has been change, but it's been a mix of some loss and some gain -- GeorgeClark

• All the test contributors that ran large installations are gone

  Really? Not from what I've seen. Large installations are slow to adopt, and for good reason. We've seen several large installations move ahead with 2.x, with reasonably good results. -- GeorgeClark

And you have to ask yourself

• What is in it for me?
• Not just me Kenneth. Me generic. What is in it for anyone spending time on this project?
• I cannot find any motivation anymore. I only see trouble being created. I cannot point to a single thing that makes Foswiki better for me and my users. Nothing!

This project is doomed if this is the direction it continues. It will be a few code geeks that are left behind that like to rewrite code. Noone will want the result. Noone will contribute if there is no gain for them.

• I am not saying that improving core code is wrong.
• It is wrong ONLY to focus on that and make the rewrite so massive that it drains the project for resources
• It is wrong to not respect the API. Do not deprecate Foswiki::Func! Find a way to work around it.
• Be innovative and create new exciting features for the users. Look at Confluence, yes even the old Twiki project has some good stuff going that we should include too.
• If there is something in it for me and my users I will gladly spend some time again testing and reporting bugs. But right now my motivation is ZERO.

Sorry to see you go. I believe that you are really wrong in your conclusions. -- GeorgeClark

-- Main.KennethLavrsen - 04 Aug 2016 - 14:52

Sorry, Kenneth, I had to bring up the "Don't touch my TXT files" cartoon from 2008 again, the one that you had on your t-shirt once. Take care.

-- MichaelDaum - 04 Aug 2016

From email from Vadim

Quote "Anyway Foswiki::Func itself has to be deprecated." Unquote

So can you agree with each other? Are you trashing the Foswiki::Func API or not? Will hundreds of plugins suddenly be using deprecated API or not? Will we all need to rewrite all plugins? Will they ever be updated? Most have no active maintainers anymore. The deprecation of Foswiki::Func is what makes me explode because this is touching something I consider a contract with the extension developers. This is heartblood.

The rest of what I said is still valid. If all you do is rewriting code for all the right reasons you cannot expect anyone else to invest time and resources. There has to be something in it for us. Right now I see a 3.0 which is only bugs and trouble for me.

Michael. I loved my T-shirt and wore it until it got too many holes. I need a new nasty T-shirt now. You should have plenty of inspiration above.

I know you are all pissed off at me but sometimes it takes a good shake to wake up geeks

-- KennethLavrsen - 04 Aug 2016

Kenneth, let me answer you with a question on your question on deprecation: how long do deprecated save/readTopicText exist? Do you know of any plans to remove them in near future?

Same gonna happen to Foswiki::Func. Because most of the time 'deprecation' means: "Guys, this piece of sh... code smells. Don't use it and move on to more advanced interfaces."

I would add to this that replacing a call like 'Foswiki::Func::someAPIMethod' with '$Foswiki::app->someAPIMethod' won't hurt anybody. Except that its more clean and doesn't hide out the fact that eventually the call manipulates with some kind of an application object. Thus code becomes more readable. However, there is choice between those two forms anyway.

Now, closer to your requests. You wanna new userland features? What about server push? PSGI makes it very easy to implement except for one quite minor details you would never think about: it requires 2+ applications to coexists in the same address space. And this is where only complete data incapsulation would do the job properly. Otherwise we risk ending up with even bigger load of bugs dragging us down and down again.

You're worried about plugins? Me too. Some years ago I developed DBIQueryPlugin and lost interest soon after finishing it being tired of fighting with the procedural approach mess where problems pile up faster than I fix them.

And this is not to mention that problems with FastCGI /mod_perl/etc support alongside with session and authentication handling as well as some other functionality could simply be unloaded on shoulders of PSGI middleware developers and we can benefit from joint force of developers of this community and have our hands free for more Foswiki-oriented coding.

PS. I wonder where FreeBSD and Linux would be nowadays if they be THAT worried about compatibility. And I do remember what a pain in the ass was some of major version upgrades of my FreeBSD servers even though there are only few of them.

PPS. And note that I'm a real conservative with TWiki 4 installation surviving 6-7 years until making me really mad about multi-language support. One of my internal servers still running FreeBSD 6 and as soon as it is not exposed to the outer world I'm not gonna bother myself upgrading it.

-- VadimBelman - 05 Aug 2016

The unfortunate reality (or maybe fortunate for code using deprecated interfaces) is that we almost never actually remove deprecated code. Seems that it only happens when the code actually breaks a new feature. The most recent removal that I can think of was the old [[Space delimited square bracket links]]. I'm sure there have been others but it often takes many years before the deprecated code disappears. Our official policy is here: DeprecationProcess

-- GeorgeClark - 05 Aug 2016

QUOTE I would add to this that replacing a call like 'Foswiki::Func::someAPIMethod' with '$Foswiki::app->someAPIMethod' won't hurt anybody. UNQUOTE

Yes. It hurts all the extension developers who will have to change the code. ALL plugins with no exception use Foswiki::Func. ALL. Every single one. They use it because it is the official API and we have been telling and repeated the message to all extension developers for all the years only to use the official API because only by using the official API you can be sure the extension will continue working. This is the contract with extension devs

It also hurts all the end users and admins that use perfectly working extensions which are no longer actively maintained. Maintenance that is not needed because they work just fine and there is no reason why they would stop working.

You cannot deprecate the entire Foswiki::Func!! And as some have observed. The few individual Foswiki::Func calls that people tried to deprecate were deprecated after a careful analysis of which plugins that used the function. Often it was less then 5 and the developer that did the deprecation also altered and re-released the extensions. Are you ready to rewrite and re-release ALL the Foswiki extensions? And are you willing to update the unpublished ones you will break also? Obviously you will not.

Foswiki::Func is not a piece of shit that smells. It is a fine API. It works well and is easy to use. Even a hardware guy like me have been able to write plugins.

It is OK to add a new API to Foswiki and recommend people to use that for new plugins - which they will if the new API becomes more powerful and can do new things. But only 30 % of the extensions will be updated the first 2 years. Most are not actively maintained but they work just fine and add great value to us users.

You cannot deprecate the entire Foswiki::Func. Drop it!.

And the DeprecationProcess has not been followed yet. We learn about it in a checkin message. And even if you follow it, we would face a deprecation process where it would be totally unacceptable to announce deprecation of our entire API in one version and remove it 2 releases later. You can do that with things noone uses or where we could fix it everywhere. But this is not the case here. You want to convert all Foswiki::Func:Bla to Foswiki::Api->Bla. And that is not all. There are additional things in the API. You will smash the whole project if you attempt this. You have to stop for a while and rethink what you are doing so you can handle the transition

I owe George some answers on the other issues.

• Remove users - I may have missed something here. I will have a look again. I think my problem was that the removal happened in a bad way. I cannot remember.
• All the items added to 2.0 and 2.1 are valuable. But if you look at it from a Latin end-user perspective not much happened. We still have the same annoying editor and for an end user - it is the daily process of creating and changing content - that is important. What is under the hood is invisible to them. Foswiki needs work in the Wysiwyg and it needs a way to make drawings (I do not expect a feature compatible with JHotDraw - that would be unrealistic). Foswiki needs someone to write an extension that integrate JS based open source drawing framework and it should be added to the set of core plugins. And that work would not need a new OO Core. It is mainly JS and integration in the normal handlers we already have. And it is way above my level of skills. But testing I can do. For hours and hours and I can assign volunteers to test with me here in Motorola.
• Conversion to UTF8. The conversion scripts are not easy to use and if your users are on Windows (99% of all business users are) you cannot just convert from iso to utf. The normal case is case 3 in the UpgradeGuide. People have mixed encoding in the topics. When people visit the IRC we recommend people to not use bulk_copy when upgrading and to install a Charset converter manually and do a command line process which is not well documented either. The upgrade to UTF8 is really difficult and in reality I see you all recommend to upgraders to stay with the RCS store and only use the text store for new installations. It is all good work and it is like the last little step was just never completed.
• Tables and Wysiwyg. Well. The simple way is to convert html tables to TML more aggresively and turn the column widths in the HTML into column widths in a TABLE macro above. Today you have to guess on a columnwidth and add it to the TABLE tag (if you know it exists) and save and edit again and adjust and ... argh! It is a pain to get a table right with the columnwidths that match the content. Something the users do in Google docs by dragging the divider line in the table. If I knew how to code that - I would code night and day. I just can't. But if someone does I will test it night and day and help making it stable and I will get users to beta test it.

Summary

• Give up that deprecation all of Foswiki::Func. Keep it and instead of copying it to Foswiki::App make a new Foswiki::App which is even better and more powerful and easier to use. Keep Foswiki::Func as it is functionally. Actually it is Foswiki::Meta as API that I am more worried about because as API is it poorly documented and dangerous. Watch out also for the few other API areas. The official API must continue to work. Only deprecate stuff very few plugins uses and be prepared to fix those plugins. * Foswiki 3.0 with out the OOO stuff is worthless to the users. You need to have bandwidth in the development team to put value for the users. Otherwise noone will beta test. Noone will upgrade. All because there is nothing in it for us. It is like upgrading a 2 year old car to the same model with the motor upgraded to a different one with the same horsepower but it is easier to repair. Noone does that. Give me a climate system, an autopilot, 20 horsepowers more, seats that adjust themselves depending on the key. Then I will upgrade. You need to add value to the users to motivate people like me.

If you can respect the extension API and add some meat to the product, then you can OO convert the core all you want. If you can do that then the NOT you added to the topic name is justified. And I will come back and help testing 3.0.

Michael - I was wrong. I still have the T-shirt. I will wear it with pride this weekend and I will post a picture of it!

-- KennethLavrsen - 05 Aug 2016

Kenneth, briefly: you read only what you wanna read; you see only what fits your view. You didn't even read carefully what I wrote here – not to mention the proposals I made or the discussions we had back then.

And you don't understand what impact bad core design makes to other code.

-- VadimBelman - 05 Aug 2016

Kenneth, One more try:

1. Development of Features: Development on Foswiki basically happens by 2 ways:
   • A) A developer is really interested in scratching an itch, makes a proposal (for core) or writes an extension, and we have a new feature. The effort is voluntary and depends upon a volunteer to contribute free labor to the effort.
   • B) A business desires a new feature, and contracts a developer/consultant to work on it. Many of the extensions and quite a few core features as came in via this path.
2. The OO Rewrite: This is an experiment. A very promising experiment. We intentionally told Vadim that he should take his vision forward without being fettered by the compatibility boat anchor. Once we see his completed vision, the project will look into how we deal with compatibility. It's not yet on the release plan, It's not merged into master.

So for your missing features. In the absence of A), then if you really need something, B) is the answer. You've been upset about JHotDraw for a long time, and the deprecation of client side Java in browsers has been well known. If you really require a replacement for Java, something completely and utterly outside the control of the Foswiki project, I suggest that Motorola step up and sponsor a replacement extension. Same for other major features that you want.

You were once the RM. There is no "magic wand" that we can wave and demand development on areas that are uninteresting to the current developers. And there is no way to demand completion of proposed features. And your whining about it isn't exactly motivational.

You yourself have two features that you committed to: NumbersAsUpperCase and AddADDWORKINGDAYSToSSP. I've deferred them for 2.0, 2.1, 2.2, and finally took them off the planned release. So how would you suggest the project get you to honor your commitments? Dock your pay? Threaten to fire you? Complain to your boss? And now you whine that the project is not working on what you think is important? We are in the same situation with every proposed feature.

And for your hissy fit over deprecations, You've seen a note in a commit on an experimental branch that is intentionally going forward without compatibility. Big deal. It's intentional, and it's experimental. We will address compatibility later. Once there is a proposal to merge the Item branch into a new OO master, then we go through what it takes to be compatible. Maybe it will be easy, maybe difficult, maybe impossible. We will figure it out at the end of the experiment.

From developers we know and trust there has been general consensus that there will be huge benefits from a restructured core and migration to a well tested platform & framework like PSGI/Plack and Moo / Moose. CGI itself has been deprecated upstream. It is no longer in Perl core, and will gradually fade to oblivion. Ultimately the restructuring of Foswiki is unavoidable. Or... we can stick our heads in the sand until we are hopelessly obsolete.

So please please. Stop with the vitriol and poison that you are pouring on the project. How do you think google searches look seeing topics like FoswikiHeadingforDoom. What do you think competing wiki's will do with your comments.

-- GeorgeClark - 05 Aug 2016

Yes. Again. Coders are gold. Testers are trash. You would think that a working hour is a working hour. But sadly that role is no longer respected.

On NumbersAsUpperCase I have written that I am stuck because I need help with the Javascript part. Noone will help me. So I have given up. And again. I am not going to spend 1000s of test hours which is hard work just like coding if I get nothing out of it.

On Vadims work I think you try to talk the problem down. It is clearly the plan to merge it in. You do not let a guy spend that much time and then tell him that his work is not welcome.

On funding. I wish I could. I am under a constant pressure. Plenty of people push to move to Atlassian products and to Google docs. And as time goes and I have nothing to offer I am loosing the battle slowly.

I stop here. I cannot say more.

I will look at how things evolve. Maybe some other people with better diplomatic skills can explain the same thing. If 3.0 is just a code rewrite you will find yourself to be 2-3 very lonely programmers.

If you are afraid of this page - delete it. You have received the message

-- KennethLavrsen -- 05 Aug 2016

Commenting is disabled while not logged in