Feature Proposal: Add markup to exclude content from Rendering

Motivation

This has repeatedly been requested. We need something better than <-- html comments --> for hiding content from users. Foswiki 2.2 adds support for #{ style comments }# which prevent all macro expansion. My objective:

• Content should go through the full macro expansion process
• It is then removed from the Rendering process, preventing any HTML generation.

This extends development done in ImproveSupportForComments, and is a much simpler implementation than that proposed in AddHideOptionToSTARTSECTION

Description and Documentation

Add a <norender> block which behaves exactly like <verbatim> for all macro and preferences processing, but is not put back into the rendered output.

This is a on-line change to Foswiki::Render, adding a single call:

  +    $text = Foswiki::takeOutBlocks( $text, 'norender',  $removed );

I've tested this, removing large blocks of html comments from the UserRegistration page.

Examples

• %{ ... }% comments:
  • Function as comments only in templates
  • Removes surrounding white space from the template (spaces, newlines, ...).
  • Are displayed regular text in topics.
  • Macros internal to the comments are rendered normally when used in a topic
  • Ex: Used frequently in Template Topics %{ <verbatim> }%
• #{ ... }# comments: (Foswiki 2.2)
  • Functions as comments in topics
  • Are processed as if they were a macro.
  • Preferences inside the comment are set
  • Any macros inside of the comment are ignored. (There is no inside-out for the comment.)
  • The macro is removed before rendering.
• <norender> ... </norender> comments (This Proposal)
  • Contained macros are fully expanded.
  • Preferences inside the comment are set.
  • The content is removed from the Rendering process and is not generated into HTML
• <-- html --> comments
  • Contained macros are fully expanded
  • Preferences inside the comment are set
  • Content is fully rendered
  • Content is sent to the browser client.
  • The browser hides the content.

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: GeorgeClark - 10 Feb 2018

Discussion

I'm struggling to see the point of this; if you are just going to remove the HTML from the final rendered output, why do you want to expand macros (unless, horror!, they are side-effecting macros)

If it's been repeatedly requested, there must be some links/references to those requests?

-- Main.CrawfordCurrie - 10 Feb 2018 - 08:49

We've added the %SET macro as a way of setting preferences. A case in point is the UserRegistration page, where the content of registration, fields being solicited, group membership, etc. is all established in a block of html style comments. (or depending on foswiki version, as a hidden verbatim)

1. There is no good reason to sent all this stuff to the browser
2. It might have some security implications - why should guests know how the page is configured. (yes, this is a pretty weak justification, but ...)
3. HTML comments are somewhat dangerous, in that they must not be nested, and it can be difficult to discover rendering issues, even with some cross-browser differences.

The only explicit request for this feature is in AddHideOptionToSTARTSECTION. I clearly overstated that point. However it does come up in discussions in some of the other requests.

Another way to look at this, which you point out in your proposal, and I'll expand on... the phases are:

1. Template expansion %{ }% comment is removed during template expansion
2. Macro expansion #{ }# comment is removed during macro expansion
3. TML rendering (<verbatim> ... maybe. Not rendered, but not removed)
4. Client side rendering <-- --> is removed by the browser

We don't actually have any way to allow a block of code to be fully processed by 1 & 2, and then remove it at the start TML expansion, and not put it back. <verbatim> prevents the rendering, but then encodes, and leaves the content in the output.

Other techniques I'm finding are:

• <verbatim class="foswikiHidden">
• <div style=".. display:none">

Anyway, I just think it's a missing piece.

-- GeorgeClark - 10 Feb 2018

Another example where there is a lot of undesired HTML output in the page. Page design that uses translcude to include hidden blocks into the displayed part of the page. The only ways we have to hide those components are with either html comments or css style hiding. Or by not transcluding, but by splitting the components into a separate topic.

-- GeorgeClark - 10 Feb 2018

Another way to reduce the rendered output to only the desired amount without bits and bytes around it is to have a view.include.tmpl skin/cover that replaces the standard %TEXT% placeholder in the template with an %INCLUDE{"%WEB%.%TOPIC%"}%. Anything that should be rendered will be located in the %STARTINCLUDE....STOPINCLUDE% area while additional sections stay out. I know this is the other way around what you envisioned, George, yet still very practical ... and this without any futher ado.

-- MichaelDaum - 10 Feb 2018

I was hoping this was going to work. Maybe I'm doing something wrong, but while it does work, none of the %SET macros outside of the %START/STOP include section are processed. So this technique also omits the Macro Expansion phase.

So this is worth documenting, and we may want to even ship a view.include.tmpl file. But it won't work for the UserRegistration page, which has a lot of %SET statements in the hidden areas.

-- GeorgeClark - 11 Feb 2018

For the UserRegistration page I'd strongly recommend to move out complexity into a UserRegistrationViewTemplate, and use TMPL:DEF/TMPL:P mechanisms for modularity instead of nested transclusions.

-- MichaelDaum - 11 Feb 2018

There's a handy little table in https://foswiki.org/System/TopicMarkupLanguage that shows what gets expanded in different tag contexts.

	TML	HTML	Macros	White Space	Auto Link WikiWord	Auto Link Email
<verbatim>
<literal>
<pre>
<noautolink>

i.e. it's a bit of a mess. It might be better to discourage all this and support the use of feature enable/disable, e.g.

   * < verbatim> - <feature tml=off html=escape macros=off whitespace=preserve autolink=off>
   * <literal> - <feature tml=off whitespace=compress autolink=off>
   * <noautolink> - <feature autolink=off>
   * <sticky> - <feature wysiwyg=protect>

Use of feature would clean up a lot of legacy, and future proof it too.

-- Main.CrawfordCurrie - 11 Feb 2018 - 10:07

Had to "break" the nested verbatim above. One issue with our blocks is that they cannot be safely nested, even when brought in with an INCLUDE. It would be nice if our include processor could somehow mark included blocks, so that they can 't break blocks at the higher level.

I like the idea, but I think it's clearer if all the features were a simple on/off, rather than compress, preserve, escape, protect, etc. My proposed norender feature would map to "hide" or "redact"? Maybe html=escape becomes escapehtml=on/off? Maybe not.

Anyway, this bikeshed's a simple way of redacting output from foswiki into a major rework . I like it, but it changes a one-line feature into a rather large effort. Certainly nothing that we could tackle for 2.2.

Part of me keeps wanting to think of this as linear markers rather than blocks, but that would completely break our whole topic expansion process. So blocks it is.

---

Re: fixing UserRegistration. The problem with skin templates is that they are arcane and really difficult for the new user to understand and modify. If we could use them, but do it through a simple list of fields to collect on registration, that's good. Asking a new admin to Foswiki to edit templates is just too high a bar. I'll review what's changed in 2.2. for Registration and see what might be done moving stuff into templates.

-- GeorgeClark - 11 Feb 2018

At the risk of going off track, (well further anyway), I see we support another style comment - <? ... ?>. However it is a single-line construct only.

• <? ... ?> Stops all TML rendering. Macros are expanded. Single line only
• <!-- ... --> No impact on TML or Macro rendering, Multiline

Both are passed through to the browser. Firefox marks the <? format as invalid. Any closing html tag breaks the comment.

And a couple of more side-effects I've found while trying to understand/document our topic -> html pipeline.

• Any macro inside a <dirtyarea> that sets state used elsewhere, ie %SET, will not function as expected. If I'm reading the code right, dirtyarea rendering happens once everything outside those areas is completely expanded and rendered. So %SET would have no impact. There probably ought to be a caution that any macro that sets state should never appear in a dirtyarea. I can't find that we say that anywhere.
• For some reason we "put back" the <literal type blocks twice, once as "removed" and again as "protected". I can't see why this would ever happen. (Render.pm line 573)

I can't find anywhere where our whole macro expansion / TML rendering process is documented. "Timeline of a topic" would probably be a good thing to add to our Developers bible. And now I've taken this completely off the rails.

-- GeorgeClark - 11 Feb 2018

Bottom line is there is no way to prevent a block of topic text from passing through the Render process. Whether or not that is either needed, and/or useful, is the question to be answered with this proposal. The proposal AddHideOptionToSTARTSECTION is another way to accomplish this. It seemed to be widely supported, but broke down into a lot of discussions on terminology, and the proposing developer decided it was outside his expertise.

So I think that this is a duplicate proposal. %STARTSECTION{"blah" definition=hidden}% and <norender> would have the same impact. Process the section per the standard macro processing rules, but redact it from the Render process.

-- GeorgeClark - 11 Feb 2018

And digging into the STARTSECTION proposal, This is indeed a can-of-worms to implement. The *SECTION related macros are all "null" macros. They are simply removed upon macro expansion. Sections are only parsed in 3 places:

• The INCLUDE macro runs the _parseSections code on the named topic, not necessarily the current topic.
• The UI::View will run _parseSections if called with the query param section=" ". Happens before any expansion/rendering.
• And expandOnTopicCreation runs the section parser, only during topic create.

The way to remove the SECTION ... ENDSECTION block. Once the macro parser runs, the tags are gone. To fix this:

• Implement handlers for the START and STOP/END section macros
• Handler recognizes the START and STOP of the hidden defnition type, and changes them into <hidden> or <norender> blocks, otherwise returns ''
• And Render then discards those blocks.

I'm going to merge this proposal into AddHideOptionToSTARTSECTION as the implementation strategy. The settled upon implementation was:

• Add an option definition="hidden" to remove the section contents from view. The section will not be shown when the base topic is viewed or INCLUDEd; the intent is that the section will be hidden unless explicitly called for.

%STARTSECTION{"blo" definition="hidden"}%
contents
%ENDSECTION{"blo" definition="hidden"}%

When a section is hidden, the macro will expand into a marker, including the section name. This will allow render to remove the matching start/end markers.

• <!--${REMARKER}HIDESECTIONname-->
• <!--${REEND}HIDESECTIONname-->

Render.pm will remove s/<--${REMARKER}HIDESECTION(.*?)-->.*?<--${REEND}HIDESECTION\1-->//g;

This could be extended to use the *SECTION macros to set other rendering controls. Left for another proposal.

-- GeorgeClark - 12 Feb 2018

The existing documentation for the topic view process is very sparse:

1. Expand macros
2. Render TML

It would be useful if you documented your research above (all <tags>, %MACROS, \escapes, %{comments etc that exist for controlling macro expansion and topic rendering) in a single readable topic.

-- Main.CrawfordCurrie - 12 Feb 2018 - 08:54

Indeed. I've been working on establishing a LifeOfATopic timeline that shows the order of events from when the UI asks to load the topic through to the finishedPage. Right now it's a text file. I'm still pondering ways to present the information. Maybe in a graphical phases with a more detailed breakdown.

Anyway, marking this as a discarded proposal. Thanks for all the feedback / ideas.

-- Main.GeorgeClark - 12 Feb 2018 - 14:45

See LifeOfATopic for a detailed look at the topic rendering pipeline.

-- GeorgeClark - 13 Feb 2018

Commenting is disabled while not logged in