Item2174: WysiwygPlugin removes line breaks

WysiwygPlugin (I presume) removes line breaks from the middle of paragraphs. Being OCD, I like to put in line breaks at sentence (and sometimes phrase) boundaries, to keep the line length under control. I REALLY dislike it when an overweening program elects to remove them.

-- RichMorin

It is a tough call with these line breaks.

We have to remember that Foswiki (as well as the old project) is made to be able to mix Topic Markup Language and HTML. And in HTML a line break in the source is not visible. You need to provide a tag to break a line. Either br or p. Or some HTML element that creates new lines like bullets.

This is also why TML uses two line breaks to create a new paragraph. The Wysiwyg TinyMCE (TMCE) is an HTML editor and when you save the WysiwygPlugin translate the HTML to TML.

When you enter a new line (hit return) in Wysiwyg mode you actually add a paragraph (p) and it is converted to an empty line (ie two newline characters). And this remains also when you edit again.

But if you edit the raw text first and write some text and apply some new lines (single new line character) and then view the saved text you see everything on one line as you would expect from the rules of TML (and HTML).

When you open such a topic in Wysiwyg mode the two lines merges to one line as you have noted. The conversion happens already when the topic is loaded.

I checked with some quick debug code. The TML2HTML conversion is not the one that removes the new lines. It is the TMCE editor itself that does that. This also means that it is very very difficult for us to fix this. The TMCE is an advanced project in itself that Foswiki uses as our editor and changing it so it does not do any processing of the HTML when a page is loaded seems near impossible. This reformatting is a price we have to pay. All we can do is handle the newlines in those cases where the HTML2TML would otherwise be wrong.

Foswiki has a special tag to be used by people such as you that want to enter some TML in raw mode and prevent the Wysiwyg editor from changing it. The tags are <sticky> </sticky>

If you put sticky tags around a section of TML it will not be altered by the Wysiwyg editor and it will be shown as red in Wysiwyg mode. You can actually also add the sticky tag in the TMCE. The format "protect forever" is the sticky tag.

If you want line breaks preserved in a Wysiwyg editable topic you should use the sticky tags.

I will let other follow up before claiming this "No Action" but I am afraid this is going to be the outcome of this one

-- KennethLavrsen - 29 Sep 2009

Ahem. I don't really care about the difficulty of making WysiwygPlugin stop breaking my source code formatting. It's the responsibility of new "features" to not break existing behavior.

To be specific: TML is source code. An editor which BY DEFAULT messes up the formatting of source code is broken.

The real answer, given that FW wants to offer Wysiwyg editing, is to make WysiwygPlugin work properly. (Not to put too fine a point on it, this should have been done before introducing the "feature".) However, this may take a while, so why not provide a "raw edit" button?

-- RichMorin - 29 Sep 2009

The "source code" view of TML is important to many long-time users of Foswiki and its predecessor. It is largely irrelevant to many users. Where I work, most users are software developers. To them, the "source code" view is neat, but they prefer the Wysiwyg editing.

BTW, there is a raw-edit button in PatternSkin; it is at bottom of the page.

-- MichaelTempest - 30 Sep 2009

I have been thinking about this. WIBNIF if WysiwygPlugin sent TMCE specially marked up <br/> tags representing \n newlines in the TML. Then we modify save_callback handler to replace these with \n's again before submitting back to save/html2tml/etc?

Any thoughts?

-- PaulHarvey - 20 Oct 2009

As I noted on Item2619, some WYSIWYG use-cases require eating of newlines e.g. the non-technical user wants the editor to present paragraphs in the same way as they are rendered, without extra linebreaks. For this reason, WysiwygPlugin should not convert \n to <br/> tags, universally. Instead, I was wondering about converting them to some kind of hidden marker (e.g. an empty span), for ordinary text (i.e. non-protected text), so that the \n's end up in the same places but are hidden in the WYSIWYG editor.

-- MichaelTempest - 03 Mar 2010

It is very true that <br /> users have put in a topic but never be converted to plain newlines. People put them everywhere when they want a new line and no extra space between lines.

I really do not think we can fix this without breaking something else. I am deeply concerned about the consequences. If people care for source code in a document they should surround the section with the already existing codes that prevents Wysiwyg from altering the format. Or disable Wysiwyg on that page. Watch out.

-- KennethLavrsen - 03 Mar 2010

Perhaps it should be easier to disable WYSIWYG (at the click of a button? it would be nice to set lots of things from the editor, kind of like what NatEditPlugin can do: permissions, etc).

Michael, I really like the idea of the hidden spans.

-- PaulHarvey - 03 Mar 2010

I implemented hidden spans to preserve line-breaks and associated indenting in the TML. There is a serious problem with hidden spans.

As this example shows, TMCE hides the hidden span (which contains " X " i.e. space-X-space) and the cursor jumps over it, but it is still really there. To see what I mean, put the cursor between C and D and press left-arrow to move to between A and B. No surprises so far. Now put the cursor between C and D and press backspace until B is deleted - several additional keypresses are necessary even though nothing apparently happens.

AB<span class="foswikiPreserveNewline" style="display: none;"> X </span>CD

I don't think this will be acceptable at all.

-- MichaelTempest - 15 May 2010

On balance though; do you think eating newlines is the more acceptable of the two situations?

Hmm. I wouldn't give up just yet: TinyMCE allows us to provide an onExecCommand callback quite easily - it could check to see if the delete command was used, and then also remove a hidden span to the left (assuming LTR language) if present.

My only doubt is whether TMCE actually uses execCommand when delete is pressed, at all or consistently

-- PaulHarvey - 15 May 2010

The newline-eating behaviour is acceptable to non-technical users (because, for the most part, it does not affect them at all). It is frustrating for technical users, but technical users can disable wysiwyg for the whole wiki (if the only users are technical users), for a specific topic, or just for a section of a topic. There are a few work-arounds for technical users.

I suspect the weird backspace behaviour when deleting hidden spans will be confusing for all types of users. Technical users might be prepared to put up with it, but then we would have to educate them about the cost/benefit tradeoff we made. If there were a way for the admin to choose between these two options, then the hidden spans might be acceptable. I am concerned that the odd behaviour from deleting hidden spans will cause a higher support-load than the newline-eating behaviour.

On balance, I do think that eating newlines is more acceptable. That is at least documented, and seldom affects non-technical users.

-- MichaelTempest - 16 May 2010

I've had another idea that I'll think about: replace TML newlines with a visible span that contains a single space, and use class to track it, and to keep track of leading-spaces on the TML line that follows.

-- MichaelTempest - 16 May 2010

I think this is a viable way forward. I was thinking of how we'd document this, probably something like:

When it comes to paragraphs and newlines, there are three scenarios to consider:

• Paragraphs (emitted as <p>some text</p> in HTML). You usually use the enter key in the WYSIWYG editor to create a new paragraph. In WikiText mode, a new paragraph is started whenever there is a blank line preceding some text.
• Forced line-breaks (<br/> tags in HTML). You usually use shift+enter in the WYSIWYG editor to force a line-break inside an existing paragraph. In WikiText mode, a forced line-break is achieved with the %BR% macro or the <br/> HTML tag.
• Hidden line-breaks (not rendered when you view a topic). For example, a new paragraph is generated if there are two more consecutive newlines in the WikiText; if there is only one newline, it is ignored. These line-breaks do not directly contribute to the final layout of topic text. Often they are used to help make WikiText TML easier to read.
  • In previous versions of Foswiki, these newlines would be removed by the WYSWIYG editor as they have no real equivalent in HTML. Now, they are represented by a special WYSIWYG character %ICON{"wysiwyglinebreak"}%. You may insert a hidden line-break when in WYSIWYG mode by clicking the %ICON{"wysiwyglinebreak"}% button in the toolbar.

The last sentence is something we haven't discussed yet

-- PaulHarvey - 01 Jul 2010

That is an interesting idea - some people won't want the TML newlines shown, so I suggest there should be a toggle-control for showing or hiding these things - similar to many word-processor's fancy "paragraph-P" button.

BTW - We can store details about the amount of whitespace via JSON in a span's style attribute - that survives a round-trip through TMCE.

Rich - I would value your feedback on this idea, so I've marked this task "waiting for [your] feedback".

I am also raising this from "Low" to "Normal". It bugs me

-- MichaelTempest - 18 Jul 2010

First, thanks to everyone for paying attention to this and working to find a solution that meets both the needs of WYSIWYG and TML users.

As I understand it, the proposed change would keep the current behavior for WYSIWYG users, but allow TML users to add single line breaks for formatting. If so, great! My only worry is that existing topics may not be handled properly - do you have a strategy for handling and testing this?

-- RichMorin - 19 Jul 2010

I plan to store the line breaks that do not affect rendering in meta-data in the HTML whilst the topic is being edited. This is intended to protect the TML formatting in existing topics. It would be great if these can be made visible in the WYSIWYG editor, but (to me) that is a nice-to-have.

I have an automated test that loads each topic that ships with Foswiki into the WYSIWYG editor and saves it, and then checks that the TML is the same. As a generalisation, I plan to make this test pass without modifying the topics. I may have to apply <sticky> or <literal> tags in some places, but those should be the exception, not the rule.

Changing status back to "confirmed".

-- MichaelTempest - 20 Jul 2010

Rich - a status update: I have implemented the change I had in mind (details here). It helps for a lot of whitespace, but not for all whitespace. For example, line breaks before and after ---+++ -style headings are still messed up. The change has required that I tweak the TinyMCE configuration, too, otherwise I run afoul of workarounds for Firefox bugs. Nevertheless, it is better than it was

I am unwilling to commit this change just before Kenneth tries to create a release candidate for 1.1. The change is too risky to put in shortly before a release. I have attached a diff so that Kenneth may decide if the change should go into 1.1 at all.

• wysiwyg.diff: Diff for encoding TML whitespace when converting to HTML, and restoring the whitespace when converting back to TML
• tmce.diff: Change to TinyMCEPlugin so that it is possible to delete the span-with-one-space in Firefox

-- MichaelTempest - 03 Sep 2010

This is planned for 1.1.1. We might release (on the Extensions web) versions of TinyMCEPlugin and WysiwygPlugin that contain this fix before the release of 1.1.1.

-- MichaelTempest - 05 Sep 2010

Deferred to 1.1.2

-- MichaelTempest - 29 Oct 2010

I never saw the patches that were meant for me.

-- KennethLavrsen - 29 Oct 2010

Right - this is checked in on trunk When Kenneth is happy that is is stable, I'll check it on the 1.1.x release branch, too. Kenneth - I have tested this in a browser (and I did find problems, and those I have fixed), but there are always more tests that can be done and I seem to miss the ones that fail. So I am unleashing this on trunk.foswiki.org.

-- MichaelTempest - 06 Nov 2010

The good news is that I haven't had any complaints about this change on trunk. The bad news is that I turned some existing newline-eating behaviour into a bug. I need to fix that before I check this change into the release branch.

   * Foo

     Bar

used to become

   * Foo
Bar

which is different in the TML but renders the same - like this:

• Foo

Bar

Unfortunately, now it becomes

   * Foo
     Bar

which renders differently:

• Foo Bar

-- MichaelTempest - 19 Nov 2010

Michael, I merged this over. WysiwygPlugin and TinyMCEPlugin should now be in sync wrt trunk vs Release01x01. I did this under the impression this task could be closed, is that true or was I dreaming?

-- PaulHarvey - 06 Jan 2011

I must still fix the newline-eating behaviour (for newline-after-a-bullet) I described above. I have a fix for it, but I must still make sure all the tests pass before I check it in.

-- MichaelTempest - 06 Jan 2011

Okay - wysiwyg editing no longer eats empty lines between bullets and the paragraph that follows.

-- MichaelTempest - 13 Jan 2011

Happydance++

-- PaulHarvey - 13 Jan 2011

The first real bug to report against this, Item11312

-- PaulHarvey - 30 Nov 2011

Defer to 1.1.5

-- GeorgeClark - 13 Dec 2011

Commenting is disabled while not logged in