Feature Proposal: Support <blockquote> and paragraph indenting in TinyMCEPlugin

Motivation

See Support.Question293.

Apparently using the indent buttons on paragraphs (not lists) used to result in <blockquote> in 1.0.6.

In 1.0.7 this is no longer the case, paragraph indenting is lost.

Propose that we add a button to apply <blockquote> ( done ) and figure out a reasonable way to have indented topic content that's not inside a list.

Description and Documentation

1. Implement the new markup in Render.pm with unit tests
   1. Emit as foswikiIndentLevelN classed <p>s, or style="left-margin: (n x 30)px;" attributes?
      Probably implement as classes, so rtl locales can use right-margin in CSS?
2. Implement the new markup in WysiwygPlugin - tml2html, html2tml - with unit tests.
   1. How should WysiwygPlugin handle older versions of Render.pm which don't support the new markup?
3. Test with TinyMCE. Test how indenting applied to non-paragraph block elements behave.
4. Documentation
5. Review how application of indenting works or doesn't work with existing PatternSkin, NatSkin, WidgetSkin CSS.
6. Develop a SEARCH (or script to be shipped in the /foswiki/tools directory) which will report existing topics which already have content that may be interpreted as paragraph indent
   1. Foswiki release upgrades: Should the presence of this tool simply be advertised in the release notes, or should we add a configure checker that adds a warning/error until you run the tool? NB: Upgrade packages are simply a tar -xzf of the upgrade tarball, nobody runs an actual upgrade script as such...
   2. Those upgrading WysiwygPlugin on older Foswikis: Disabling the indent feature should be enough?

Examples

Impact

WhatDoesItAffect:

Implementation

Implement a MediaWiki inspired style of indents, as proposed for the old project, based on the current behaviour for TML lists:

Example 1

   : one level of indent
      : two level indent
      text following the above indent
      more text that is part of the above indent
         : three level indent
         : three level indent, but rendered as a separate para

Rendered as:

one level of indent

two level indent text following the above indent more text that is part of the above indent

three level indent

three level indent, but rendered as a separate para

Example 2

   1 An ol item
      : two level of indent. counting list level
         : three level indent, counting list level
         text following the above indent
         more text that is part of the above indent
         1 Another ol item three levels in
         1 Yet another ol item three levels in
            : four level indent, counting list level
      : two level indent
   1 An ol item, continued numbering in sequence after the first

Rendered as:

1. An ol item

   two level of indent. counting list level

   three level indent, counting list level text following the above indent more text that is part of the above indent

   1. Another ol item three levels in
   2. Yet another ol item three levels in

      four level indent, counting list level

   two level indent

2. An ol item, continued numbering in sequence after the first

Notes

• TinyMCEPlugin should receive these as <p style="padding-left: (n x 30) px"..
• =::Render should translate them to <p class="foswikiParaLevelN">. This class would be appropriately styled in CSS ( {padding-left: (n x 30)px;} or maybe 3em?) to achieve the desired indentation effect.

STICKYBITS preferences

Moved to WysiwygContentPolicies. We could avoid doing much at all (just use <blockquote> as normal) by making it easier for users to more easily specify different levels of tolerated HTML in their topics.

<blockquote> function

Add blockquote button (next to indent button).MediaWiki doesn't have special markup for this.

• Done, checked in to trunk and release branches Tasks.Item2341.

Alternative Business as usual: no support for indented paragraphs, only <blockquote>. -- Contributors: PaulHarvey - 08 Nov 2009

Discussion

• New toolbar layout:
  

---

The removal of blockquote as a feature in 1.0.6 was in principle a bug

Adding it again is fixing a bug. Even if we do the UI a little differently.

I think this can be done as bug fix without need of feature proposal decision.

Adding an indent feature is another monster. It never becomes a clean TML documemt once we start adding a lot of CSS stuff in the code. That concerns me a bit and I would like to think that through. But the blockquote feature should be added as it has been there until 1.0.6 removed it.

-- KennethLavrsen - 11 Nov 2009

Adjusted the proposal hopefully to satisfy Kenneth's concerns. As a side-effect, I actually think per-topic setting of a STICKYBITS_POLICY is a nice idea (Eg. WebHome and fancy topic pages which have a lot of presentation effort, it would be nice to set to liberal. I know that I would like to set some topics to strict to help pasting tables from word/excel staying as TML tables).

-- PaulHarvey - 15 Nov 2009

I'm with Kenneth on the nastiness of an indenting feature (see http://twiki.org/cgi-bin/view/Codev/TWikiMarkupForBlockquotes) but otherwise cool with this.

-- CrawfordCurrie - 19 Nov 2009

The useful part of that conversation was apparently Mediawiki's syntax:

   : one level of indent
      : two level indent
         : three level indent

Should this proposal be updated to work this into new TML syntax?

-- PaulHarvey - 19 Nov 2009

Reset the clock because the proposal has been changed substantially...

-- PaulHarvey - 19 Nov 2009

I really would like a way to do muti-level, bulletless indenting without having to fart about with CSS, and the MediaWiki syntax looks pretty good. Tentatively supported, though I'd like to hear some other voices on this.

-- CrawfordCurrie - 20 Nov 2009

i think this formatting markup is not bad at all! it's consistent with current TML and, as a bonus, leverages people's MediaWiki knowledge. i am concerned, however, that several of the revisions to TML and WikiWord definitions have lead to grief before, so a high degree of caution is required.

perhaps some of you with large topic collections could execute the following command to see if this is feasible or if it would "break" "too much" existing data?

grep -RPe '^( {3}|\t)+:' data/

Possibly a better recipe:

egrep -r --include=\*.txt "^( |$tab)+: " data | wc -l

Who	# Topics	Hits	Notes
PaulHarvey	4832	0
AndrewJones	46189	1	(however, it is in a pre tag)
UrsBeyerle	2798	0
PetrMatejka	825	0
AntonioTerceiro/1	9381	0	used PetrMatejka's suggestion for counting matches
AntonioTerceiro/2	3592	0	used PetrMatejka's suggestion for counting matches
ColasNahaboo	60213	15	java code directly in the page 6 - text in a html table cell (document is one big HTML table) 4 - java code in verbatim 5

-- WillNorris - 21 Nov 2009

I used find data/ -type f -name '*.txt' |wc to get the total number of topics.

-- PaulHarvey - 23 Nov 2009

What would happen if it was found in a pre?

-- AndrewJones - 03 Dec 2009

I wasn't able to use your grep on debian so I used following:

find data/ -type f -name '*.txt' | xargs pcregrep -M '^( {3}|\t)+:'

• or use: egrep -r '^( |XXX)+:' data where XXX there means typing ctrl-v ctrl-i on the keyboard - CN

-- PetrMatejka - 03 Dec 2009

IMHO, the statistics above are meaningless even in the unlikely case that they should be representative -- this is a good opportunity to introduce a policy ( w/o separate proposal) whereby any change regarding the basic formatting (especially TML codes) requires some sort of 'upgrade script' which -- upon confirmation by the administrator who's upgrading the foswiki installation -- makes sure that no existing content will be changed unexpectedly. (In this case, existing "{_}" sequences would have to be escaped.)

The 'right' way to verify this, btw, if I'm not mistaken regarding the order in which TML codes are expanded, isn't to just scan the TML text files, but to actually render all pages in order to catch 'reformatting instructions' and check whether the new logic would change the output (think GluePlugin etc.).

NB this doesn't mean that I'm against this proposal.

-- MarkusUeberall - 03 Dec 2009

I agree with Markus, but would be satisfied by just a detection script and a pointer to a foswiki.org page with recipes to fix things, for the cases where an automated upgrade script is not possible to do easily. Ah, and I agree with the proposal, even if it means fixing pages by hand on my wiki.

Note: I used to count: egrep -r --include=\*.txt "^( |$tab)+: " data | wc -l which is better because:

• it excludes RCS files otherwise you count many times one occurence
• it looks for a space after the : I had matches such as :peserver: otherwise

With this I find 15 hits instead of 86.

-- ColasNahaboo - 03 Dec 2009

Though I personally prefer not using <blockquote> for purposes other than quoting text, I imagine this approach is the simplest way to integrate the indent functionality with TinyMCE (and probably any other generic WYSIWYG HTML editor?). -- IsaacLin - 04 Dec 2009

---

At one time I favored the syntax as proposed:

 
  : one level of indent
      : two level indent
         : three level indent

Since then, I've been fairly divorced from TWiki and Foswiki, using TWiki markup with some personal variants to maintain a "personal" free-format database that I'm hoping someday do move back to TWiki/Foswiki and put on the web (without the really personal stuff).

I've found that I do an awful lot of quoting--I mean multiple paragraphs per Foswiki topic / page (I call those records in my database). The markup I use (which has at least one problem--below) is more of a start blockquote, end blockquote markup--I use single quotes on dedicated lines (sometimes more than one, and sometimes in combination with equal signs (to denote preformatted / monospaced text--e.g., code).

For example:

'
A paragraph intended to be blockquoted.
'

So, I can easily blockquote long passages of text. I would find it inconvenient to mark each paragraph of a long passage with a preceding 3 spaces, colon, space.

The problem with my syntax is that it is ambiguous at times (for nested quotes) (I've learned that while trying to work out a parser for it (for Scintilla)). I have thought about modifications to it, like using ' for the start of a blockquote and something else (`?) for the end of a blockquote.

I just thought I'd mention this. I'm not going to have a major problem if Foswiki implements the proposed syntax. Some day, I may come back and request / propose something more like I use. (And, of course, I can always write a converter, although that has its drawbacks.)

Hmm, should I give some examples of the ambiguity that can result from my markup? Some of the ambiguity can be resolved by smarter parsing (lookahead), but some of it cannot. Ok, here's a simple example of ambiguity:

'
Looking at this, you'd initially think that this is a single level blockquote, and ...
'
...this is a non-blockquoted followup paragraph.

But, if you later find two single quotes on a line as the last blockquote markup in a given Foswiki page, you'd realize that I want the paragraphs here to appear as a nested blockquote, as shown below.
''

The way it should (based on my intent) appear (after looking ahead far enough).

Looking at this, you'd initially think that this was a single level blockquote, and ...

...this is a non-blockquoted followup paragraph.

But, if you later find two single quotes on a line as the last blockquote markup in a given Foswiki page, you'd realize that I want the paragraphs here to appear as a nested blockquote, as shown here.

Some similar examples of ambiguity cannot be resolved by looking ahead, they are just plain ambiguous.

-- RandyKramer - 04 Dec 2009

Randy, for doing citation/quote markup, I strongly believe <blockquote> and <q>/<cite> are the right tools for the job.

TinyMCE provides the ability to indent paragraphs by applying a style="left-margin: attribute to <p> (and other?) blocks, however, and that's what we can't represent neatly in TML.

A legitimate use of this is the user who desires the content below a minor heading to be indented (visually distinct) from the rest of their content.

I have updated the proposal's work plan to cover implementing a script which will examine the user's topics for any text that will be accidentally interpreted as indented paras.

Unfortunately I don't see an easy way to do more than grep through the topics - checking the final rendering of topics incl. macro expansion would entail a huge amount of work (and CPU time).

I am willing to hear other opinions about that, however.

-- PaulHarvey - 05 Dec 2009

Paul,

Thanks for your comments!

This page is a little hard to follow because it addresses both blockquotes and indented paragraphs.

If you use the proposed markup or something like it for indented paragraphs:

   : one level of indent
      : two level indent
         : three level indent

That leaves the field free for a markup like I prefer (a begin / end type markup) to be used for blockquotes either soon, or sometime in the future. I have a strong need for a *wiki (FosT[wW]iki) markup for blockquotes, and, how can I say this--something that makes the intent obvious in the non-rendered raw *wiki text (like some of the various structured text markups that are intended to be read in raw or rendered form).

-- RandyKramer - 06 Dec 2009

You're right, this page needs a cleanup. It was written from the point of view of features in TinyMCE that are crippled due to lack of TML representation.

So to clarify: "support blockquote" was simply adding a button in the WYSIWYG editor; "support indenting" was just trying to find a way of doing indenting without messing about with style attributes on <p> tags.

At a stretch, I would consider extending this proposal to something like:

   : one level of indent
      :" two level indent, blockquote
         : three level indent, within the blockquote

But I would have to reset the clock and ask everybody to come back and review again

Which is a terrible reason for not rewriting this proposal, but it doesn't satisfy your need for start/end style blockquoting which is not something I was thinking about when I wrote this up, and probably deserves its own proposal.

Randy, are you happy to make your blockquote syntax a separate proposal?

-- PaulHarvey - 06 Dec 2009

Yes, sure it can be a separate proposal. I don't know how soon I'll bother even making a proposal, as my current focus is getting folding for *wiki markup into the Scintilla based family of editors.

BTW, from my point of view, I don't see any particular need to extend the proposal as you describe--maybe somebody else does.

-- RandyKramer - 06 Dec 2009

Looking forward to seeing the Scintilla work, it sounds very interesting. So I suppose I can set this to AcceptedBy14DayRule

-- PaulHarvey - 07 Dec 2009

Updated examples.

-- PaulHarvey - 07 Dec 2009

Updated with another example to demonstrate behaviour in lists.

-- PaulHarvey - 09 Dec 2009

AcceptedBy14DayRule, created Tasks.Item2516

-- PaulHarvey - 21 Dec 2009

The TinyMCE part needs a bit more work, but Crawford has implemented the new syntax in Tasks.Item2516

-- PaulHarvey - 13 Dec 2011