Foswikisaur Extinction Event or Why Foswiki Maybe NOT Heading For Doom

Everything bellow is based on my own observations and feedbacks from my users. Highly subjective.

The Alpha-Omega of every software project is: Provide solutions for the user's needs. So, let's define the users and their needs.

Again, talking about the my users, (maybe your experiences are different) - here are two basic categories

1. "The Word user" - aka the user who mainly uses MS Word & Excel or OpenOffice - so, when need enter something to the wiki, it is mainly pure texts and tables. Such user:
   • is looking for excellent WYSIWYG environment
   • and don't care about the macros at all
   • and even more he is frustrated with any macros - even when see some already inserted predefined macros from the template topics
   • and never will learn how to use "macros" - even when he actively uses excel macros. This is strange but this my experience.
   • also, never will learn TML - the maximum what i could expect: they (with great resistance) will learn how to use the NatEdit.
2. The "IT-person" - aka the user who has some "IT-background" and understands (at some level) how "things" works in the computers. Such user:
   • could learn macros
   • have no problems with TML (but got a feedback, "why our wiki couldn't use the standard syntax " - and they mean: markdown)
   • but, need to say - they're all frustrated with the syntax and with the try and will see approach
   • also, never creates any wiki-app (but could use them if/when someone develops some)
3. The third category of "users" are wikiadmins, who makes the decision to use Foswiki. They:
   • mainly happy with it
   • and could "hack" it to achieve their needs
   • admins creates some wiki-apps
   • mostly could write some plugins
   • myself is in this category
4. The forth category (added by Kenneth)
   • The wiki-app developer or hacker (hacking an existing app or creating a copy of one and hacking it)
   • has "seen the light" after the admin shown him how to make an app that solves a problem
   • typically software architects or project managers with boring administrative routine jobs that sees a chance to be creative
   • does not create the really advanced stuff. Very simple apps to track things. Like do/done lists, managing a test resource, meeting minutes
   • needs help from the admin when he/she gets stuck
   • not afraid to use macros
   • if a software developer - will often fill the app with in-line javascript making him the only one that understand what is going on
   • There are around 20 of them in my user community out of 1000s of users. Small group with a big impact on the business flows. Most are young software engineers from Krakow, Poland.

Also, all of the above users uses some web-based tools, staring with Facebook, Linkedin, google-apps, and zilion other modern and powerful web-apps ...

And the above sentence is very very important. Unfortunately, Foswiki loosing the battle when we talking about:

• nice design (honestly, foswiki has a terrible 1995-like design)
• new design adaptability (aka using "easy theming" and "Foswiki" in one sentence means an oxymoron)
• mobile friendliness - Foswiki looks even more terrible in the tablets and mobile phones
• user-friendly features, like: inline commenting (like msword commnents), charts, diagrams (like Kenneth already (very precisely) said), even our handling of images has zilion problems.
• one more example for all: we couldn't event include content from another topic into the table cell, e.g. the extremelly simple thing doesn't works

| some content | %INCLUDE{"SomeOtherTopic"}% |

• and could create another ten bullets
• but the reader probably already knows the Foswiki shortcomings

Kenneth view is absolutely true and correct when talking about the user-firendly-ness and advanced user-wanted features in the Foswiki.

So, the first conclusion: Foswiki needs provide more advanced and very-very POLISHED(!) features out of the box (aka as the core)

Now, switching to the developer's view - e.g. to an person who should develop such features.

Myslef knows enough deeply only two languages. bash (could qualify myself as expert), and perl (could qualify myself as advanced perl programmer - but (unfortunately) i'm not an expert - now mean expert in compare with well-known CPAN/p5p developers, like: ikegami, Borodin, choroba and many tens of others). I'm able (and already done) many Foswiki hacks, internal extensions. I managed an ( mostly ) successful utf8 trasnition on the Foswiki 1.0.5, /and even when i uploaded the patch files to the Tasks.Item5437 in 2010 - nobody care - so the Foswiki team managed the utf8 transition only in 2015, 5 years later :(/

The only thing what a want to tell with the above: I know foswiki internals enough "deeply" to be able talking about it competently.

Why the above? Because I need to say: The Foswiki current internal design is so terrible that it isn't possible (or more precisely - is possible, but the cost of the "spagethi" code is extremely high) to transform it successfully to be able provide the wanted more modern features. Foswiki needs a revolution. The step-by-step process is doomed to failure. The only way is usable:

• trow away the whole thing
• start over with an brand-new internals
• and after (and only after) - we could create an compatibility layer with the current Foswiki - aka the current TML, Macros and/or Foswiki::Func.
• Such new advanced core easily could fulfill any current Foswiki feature, but the development should be done "from the begining" - especially must repair all wrong design decisions from the past.

Conclusion 2: Vadim's work is only the 1st small step - here is needed throw away even more parts of the current design and processing logic.

Want one example? Out basic rendering "logic" - aka using regexes for the topic content manipulation is totally wrong. Also, the line-based approach is doomed. We already mixing the line-based approach and the pure html-tables. Such design is spagethi and doomed to failure. You can check any other wiki engine, (and/or any other CMS or such engine which deals with the content-manipulation - they already knows, it must be based on some DOM. (Confluence, Xwiki already knows this). (and please understand i'm not talking about the DOM in sense of javascript or jQuery and even about the HTML DOM!). But analysis of this isn't the right place in this topic - needs extensive developer talks.

The only thing what i want to tell: Vrurg's work is exceptional and absolutely needed for the project.

He doing an really heroic work - and all his steps are needed and welcomed. I sometimes getting a feeling than he is the only person who really knows how an perl software should be developed.

Examples?

• He is the only(???), who knows - using the BEGIN blocks in way as Foswiki using it is an TERRIBLE practice.
• He is the only who knows, our $session variable is an terrible mess, which contains circular references. (very-very-very bad design).
• He is the only who knows: we need total encapulation.
• He is the only who know: the global LCS hash is an TERRIBLE THING?!?
• He is the only who know how the software should be tested (using already existing a good perl-testing tools).

Morever,

• He doing the development totally alone - without any help from the community.
• Moreover, the "community" (when he already done tens hours of developemnt) only says:_ you doing it wrong_
• Guys, this is TERRIBLE. You really shoud read the original VADIM's proposal.
• But nobody reading, nobody care.
• Only when he invested many hours of development - he gets only criticism.
• This project NEEDs a FORK!

The second thing is - unfortunately - the lack of knowledge. It is really wrong starting some development when someone doesn't knows some "basics". And only gac401 (and Kenneth) is enough brave to tell: "I know nothing about PSGI and or Moo(se)". Even Vadim learns the PSGI and Moo "on the way". Guys, you really should first learn how PSGI and Moo works and only after make some design decisions - and or talks. Michael - with an deep respect to all your great development and to your really big Foswiki assets - but sorry, sentence such in the irclogs:

uwsgi would be best technically speaking due to the wsgi protocol ...

is bullshit. Here isn't such thins as wsgi protocol. WSGI is an pythonian programming API. Like the PSGI is in an perl API. You probably mean uWSGI protocol, which is an really existing "wire" protocol. (very similar to SCGI). But the uWSGI protocol has nothing (and in the near future probably will not have) anything with the Foswiki. Please, finally STOP mix CGI, SCGI, FastCGI, uWSGI and such things with the PSGI. PSGI "sits" elsewhere. Period.

Clarification:

• uWSGI is an open source project developing a full stack for building hosting services
• uwsgi (all lowercase) is a binary protocol
• WSGI is a gateway interface specification, a protocol between a web server and web applications. See PEP 3333.

Jozef's above quote refers to a longer report of mine on irc diving into uWSGI while searching a viable easy to install and maintainable production environment for a Foswiki-3 engine.

-- Main.MichaelDaum - 07 Aug 2016 - 22:17

Using Moo is an great way simplify the code. But we need use it's full potential. For example the "Roles". (Moose would be even better, because its ability to have traits - extremely powerful feature). Moo(se) isn't any black-magic. It is just pure perl. Everything what you can do with perl can do with Moo(se) too. Just more easily. Much much easily. Unfortunately, here is a learning curve - for some (old procedural perl programmers) could be steep. Perfect way of learning is checking the sources of some already existing more thousand of CPAN modules.

While (now) me not sharing fully Vadim's vision (like one parent object for everything) - his work is marvelous. Although he yet not using the full potential and possibilities Moo - his work is the biggest thing in the existence of the Foswiki. Although it is currently more or less only an "window decoration" - like jast said in the the irclog. It is only "window decoration" because (unfortunately) Vadim's hands are tied with the current "backward compatibility". Vadim, please don't get be dis-motivated. Use the FULL Moo (Moose) potential. Use the Roles. Use CPAN. Use modules from the Plack:: namespace. Go further and adopt Dancer2 :). Break the whole current design to an new (redesigned) pieces, which will perfectly align, and in the future and will form an new and modern Foswiki core!

Me time-to-time checking Foswiki-logs and the FO. I saw a discussion about the "reverse-proxy-problem". Guys, we aren't the first persons in the universe who need solve such problems. Just check again the CPAN. Here are plack middlewares which solves the problem perfectly. Just Foswiki needs to be able allow different "mount-points" by it's design. Unfortunately, we currently (even the new Vadim's branch) doesn't uses the reference implementation modules - aka modules from the Plack:: namespace. Bad thing, and again once we heading to the bad design.

This topic is terrible. Many things are mixed together without any clear logic and flow. But I want said many such thing long time ago. WE NEED TALK ABOUT THE FUTURE FOSWIKI. Really need TALK. We are NOT TALKING. The current approach: "go, and start hacking and we will critise your work after - is terrible". Even Michael said in the irc:

to me it seems most of the discussion is between him and George 

and this is exatly the problem. Here really nobody want talk DEEPLY about the future Foswiki. Enough to check the irc-logs. Every initiative or idea is slammed with the "backward compatibility" hammer, without allowing blooming the idea and solving the compatibility later... Guys, please read somewhere the basics of the true "BRAINSTORMING". Want know the main rule? Here is: No idea is wrong. Please, read this: http://www.brainstorming.co.uk/tutorials/brainstormingrules.html .

Moreover, the "core" team really should read and shoud TRY TO UNDERSTAND the comments from the community. One great example - citing it as a whole from the Development/AddTakeOutPutBackBlocksToFunc topic:

It has struck me for some time that both Foswiki::Func and Foswiki::Meta are rather unwieldy and crufty, both are growing to nearly 4000 lines long.

Often in discussions we talk about having clear separation of concerns in our code — well these don't qualify in my book.

Therefore, I've been wondering about creating an API directory under Foswiki to give us Foswiki::API::whatever to give us a chance of gaining this separation.

I say chance because although Foswiki::API::RenderBlocks is a little better then adding to Foswiki::Func; the real benefits will only accrue by taking extra care in the design of what we expose there. The actual code will, in most cases, only be a veneer of some sort.

I am aware that while ideally, for this particular case, Foswiki::Render would be completely redesigned and written from the ground up to offer this as a very clean API. However, as Render is quite a large ocean we have to take smaller steps forward.

The Foswiki sub-directory could alternatively be called Func rather than API of course.

Are there not multiple APIs in Foswiki-space? Have we defined all of them? I'm assuming here that we mean plugin API.

As for design specifics, because we might actually redesign Render, should we even add takeOut and putBack at all to the API? That is, will allowing plugins to takeOut and putBack blocks hinder a potential redesign. If not, then sure add this capability. However, what is the underlying problem being solved? Aren't taking-out and putting-back the verbs describing the current implementation?

-- JulianLevens - 10 Dec 2015

This is an perfect example of an comment which should read many times and needs think about it. I love such comments.

For the end - Vadim needs help and support - and we should allow him to change EVERYTHING in the core. We could solve the thin (or fat) "compatibility layer" later - when his "fork" will fully working.

Guys we all (counting myself too) really badly need LEARNING things. The world around moving fast, the modern technologies are needed and demanded by the users (indirectly, they demanding features - unfortunately, such features are provide-able only by using modern technologies). We need learn and use them.

The Foswikisaur should die - and from his ashes could born an new human-friendly product - FoswikiNT aka FoswikiV aka Foswiki 5.0.

And now, before making any comments - please - take a break, drink one good (Czech) beer - and after a hour make a comment - if really needs to add something.

And also, please read the topic more times. Not because it is an great text (it isn't!) but because my perl is 10x better than my english and probably me said many things grammatically (and politically) incorrect. So please, try dig the real meaning what I want to tell - if it is possible. Thank you for reading.

-- Main.JozefMojzis - 06 Aug 2016 - 23:48

I agree - Foswiki (and TWiki) havn't moved on enough, both as UX, or internaly. At work, we use Confluence and Jira - and yup, Foswiki is still a more useable UX (for me ) but there's not a major difference between the 2.

The number of times the non-user facing internals have been worked on has really badly distracted us from continuing the user-facing ideas that made TWiki interesting in 1999-2005.

BUT, now, you're also needing contributors - which really means using cutting edge, modern tools - and being really, really helpful to anyone that feels like doing things, and teaching you new toys.

So - I'd say YES, and run with it

-- SvenDowideit

Jozef, thanks for your piece of opinion. It really is important to communicate more and exchange different ideas. But for the sake of your own argument it would be better to be terse and precise. Otherwise you put yourself in danger of watering the point that you were actually trying to drive home.

Sometimes I wished people would code at least as much as they ventilate.

In that sense: Don't just say so, make it happen. Let there be code.

-- Main.MichaelDaum - 07 Aug 2016 - 21:56

I have to second Michael's comment here. I have remained silent and watched Vadim's work for a while now, because I think it's interesting, and represents the best chance Foswiki has of rebasing on a modern technology. He's been very brave to take on the issues that Josef iterates above, issues that have mostly been known about (and acknowledged many times) since at least TWiki Cairo (2004/2005?)

Bottom line for Foswiki is you can either incrementally improve on what you already have - slow, and largely unrewarding, unsexy, work (e.g. upgrading TMCE) or you can bite the bullet and build something that does what Foswiki mostly does, but better, using good design and modern tools, accepting that something is going to have to break but it's for the greater good. In either case it takes people to step up to the plate and code (or test, or document).

For the record, I haven't been contributing to Foswiki for a while now because Unicode support, and getting configure working again, nearly killed me. Hero programming is exhausting, unsustainable, and without constructive support, it's going to be too much for Vadim as well.

-- Main.CrawfordCurrie - 08 Aug 2016 - 12:07

In addition to all the above and the damned compatibility talks: that's not possible to just get everything broken as it means a new product. And a new product it's few years of development for a couple of devs. Nobody has such resources. This is why step-by-step development is more appropriate for Foswiki.

-- VadimBelman - 08 Aug 2016

I added a 4th category above. These web-app developers are worth gold. It is a small group but they have a big impact. They need help and nursing and often the admin needs to actively plant a seed that makes them "see the light". In my organisation they have made applications for software version tracking in large system releases (tracks the version of more than 100 individual software products that makes up a large radio system). Also feature requests, and management of Service Enhancement Releases is done in an app made by a few of these app-geeks. The regulatory department built a small app that traces in which countries we have registered our products and which approvals we have. Instead of an Excel sheet from hell, the guy built a cool simple app that enable the 2-3 people in the team to maintain the overview. I have helped with advice and ideas but the apps are not made by me. I sometimes hack their app to make it better or faster. I sometimes see some apps where I think - "I did not know you could use Foswiki for that".

The users I feel are left behind are not these app developers. We have a huge and great set of features when you include the many plugins that enable doing great things with small and simple effort. Where I think we are behind the "competition" is the UI for the "Word User" that concerns me. It is not the features under "More Topic Actions" that is a problem. It is the WYSIWYG editor that is too weak and buggy. Try and copy text from a Word Document or a web page viewed in IE and paste it into the WYSIWYG editor and look at the garbage this produces. Garbage that leaves the user behind with a broken page and no way to fix it other than deleting the whole thing. The lack of ability to change the columnwidths of tables by dragging the column-dividers and saving it as TML + TABLE macro. The lack of ability to make a table cell background green. And we lack a simple way to draw simple line drawings in the browser directly on the wiki page. All the tools we compete with can do these things today. It is not how the skin looks or where the menus are. That is easy to tailor and we have a cool alternative done by Michael. It is what happens when you hit "Edit" where we lag behind.

I have nothing against OO work done, as long as we commit to provide a way to not break ALL the plugins we have. Surely, this project can find a technical solution that both provides a very cool new API with more power and more ease of use for the developers AND maintains compatibility with the many many 100s of extensions we have. I will surely love to start using a better API once it is available.

BUT. The message I sent in my blunt and provoking way was... If the only thing this project does the next two years is the OO rewrite then the project will sink into irrelevance. And the work needed in UI is mainly Javascript related. The core we have may be difficult to maintain. But the truth is that the main issues we have with the UI requires very little poking with the core. It is all about integrating more modern JS based tools for the editing, drawing etc. On the perl side it is parsing the resulting HTML and turning that back into TML which is the condition for being able for the app-devs to make the powerful applications. It is our predictable way to make tables etc that enables so many features. We just have to hide it away from the "word user" who does not care for macros and wiki-markup. Most of that work will be done in plugins. And unfortunately the Javascript domain is out of my skill-set. This is difficult stuff even for a pro-programmer. But I can test. I can spend time trying all sorts of crazy things and exercising the UI. You all saw how many bugs I found in just a few days in the EditRowPlugin that the developer did not see. I may not be the provider of code to a great extent but I know I make a difference when I start testing and sometimes put a whole group of people here at work on the task of testing (often without them knowing).

I'd love to see a 50-50 split in effort between working on core and improving the experience for the "word user". Right now the split is more like 95-5.

-- Main.KennethLavrsen - 15 Aug 2016 - 13:44

Lead by example.

-- MichaelDaum - 15 Aug 2016

I recently had to use a well-known commercial "competition" wiki which also uses TMCE. It was well-integrated, and has clearly received a lot of the kind of love to which Kenneth refers. However, it was not all roses:

• It discarded all of the formatting when I pasted content copied from Word.
• It was hard to find documentation for the "macro-like" functionality and several of the macros were undocumented.
• Some of the documentation was only available for the "wiki text" form of the macros, with none available for the "WYSIWYG" equivalents.

We have subsequently switched to Foswiki. Utilization only took off after the switch (despite UI issues).

Please don't oversell the state of the competition

-- MichaelTempest - 16 Aug 2016

Commenting is disabled while not logged in