SafeCommentPlugin

Anonymous Comments

I am frequently asked for this feature on our wiki at work. It would also be a critical component in making Foswiki a viable blogging platform.

This topic describes my ideas so far which would enable a safe way for anonymous or otherwise not-logged-in users to post a comment in a reasonably painless way, and minimise exposure to bad guys who are constantly looking for a wiki/blog/forum to plant malicious code and/or take over another server via security exploits...

Existing commenting solutions we can use with Foswiki

• http://disqus.com/ - 3rd-party hosted comments included on any web page via javascript
  • Loose ownership of the comments. Now they live on a proprietary service.
  • Comments at a particular URL can only be owned by one individual. Bad for project environments where you want shared ownership of comment moderation.
  • Javascript only.
• CommentPlugin - in a special comments web where ALLOWTOPICCHANGE=WikiGuest
  • Now WikiGuest (exploit/spam-bots) can wreak havoc editing any comment topic they like in that web.
  • No captcha challenge
  • Still have a spam problem (AntiWikiSpam goes some way to helping...)
  • WikiGuest can inject unfiltered content into topics (SafeWikiPlugin mitigates the impact of successful injection of nasty content)
• CaptchaPlugin (TWiki 4.1 only, IIRC) + CommentPlugin - the code hacks the core, and is apparently unmaintained. Doesn't do much for security other than perhaps reduce spam slightly.
• There are other disqus alternatives out there that might be better suited to project work, but I haven't found them yet.
• New: MichaelDaum's MetaCommentPlugin is in svn. Need to study.

Proposal

Key features:

• WikiGuest can't modify topics, only create
• WikiGuest content is filtered aggressively before it hits a topic.
• WikiGuest should be challenged via CAPTCHA
• Desirable: E-mail validation (click a link sent via E-mail to validate the comment)
• Desirable: E-mail notifications (for other comments on the thing being commented on, or replies to the comment)

Components

REST handler

Or a dedicated script. Or a plugin that hooks into the save script.

• HTTP POST to /bin/rest/SafeWikiPlugin with target=/BlogWeb/CommentsWeb
• WebPreferences in the target (CommentsWeb) web that sets some variables on:
  • Whether the current user (WikiGuest) can use the script
  • How to name the newly created topic
  • The template topic to use in creating the topic
• Shouldn't have to remove the rest script from AuthScripts

Captcha

Components

• Image generation
  • Provide %SCPCAPTCHAURL% and %SCPCAPTCHAURLPATH%. WebPreferences and/or {} params specify dimensions.
• Seed id generation
  • Provide %SCPCAPTCHASEED% to be used in the HTML form in a hidden <input> field that will be POST'd back to /bin/rest/SafeWikiPlugin

The image would be generated along with its seed info in the initial rendering of the topic that should be commented on. The seed number/hash would be rendered by the server in the HTML comment form in a hidden <input> field. Perhaps via working_dir or a hash validation function, the seed value POST'd in the hidden <input> field + captcha text offered by the user should be recognised by the server as a valid/invalid combination (to proceed with save, after having checked that the remote user is actually allowed to use the script).

http://recaptcha.net/ is one option, which I would recommend since it is free and easy to set up. -- RaymondLutz - 22 Sep 2009

Metadata

The HTML form should be able to POST to metadata fields in the target topic by appropriately name 'd <input> fields.

The WebPreferences should specify a template topic that has the appropriate form attached.. the location to insert the submitted topic content - unsure of details of this. I suspect the approach taken by CommentPlugin should be studied first. %TMPL:P{}% stuff...

Should be possible to set topicparent on comment topics to simulated threaded conversations.

E-mail validation

If SafeCommentPlugin could somehow fire an E-mail to the provided E-mail address, the link provided in that E-mail go link back to /bin/rest/SafeCommentPlugin?validate=S33DHASHTH1NG or PERMALINK_ID. It could then adjust a field in the topic - perhaps ALLOWTOPICVIEW - to make the comment visible.

-- PaulHarvey - 22 Sep 2009

Thanks Paul, for taking the initiative to get this started.

One approach that was not mentioned would be fairly easy to implement would require review of comments before they became live. This can be included with the other options such as Captcha.

1. Create a topic for each comment in the comment web, which is writable by WikiGuest. May need to go a bit further here and not allow arbitirary writes/edits, but only creation of comment topic and editing that topic.
2. Field on the comment would index back to the topic where it is relevant.
3. Approval field would allow moderator to review comments before they are posted. Such review is common practice in blog platforms like Wordpress.
4. Topic where comment was entered lists the comments using a SEARCH over the comment web, pulling out the comments that are indexed using that topic name and have been approved.

It may be worth considering several levels of users, which is common on forums. Basic users are not trusted at all, even though they may have registered. I do this already by manually authorizing users using the WikiGroups lists, and that has eliminated google spam robots who are intent on adding hidden urls to topics.

The bad part of this setup is that it is more difficult to refactor comments into the body, but perhap that is not a goal of this sort of guest comments.

I would also like to see tight integration with email, such that comments are emailed to anyone else who commented, akin to what is currently done on Facebook. Click the link and it brings you back to the topic to see what was said. Add a comment and others who have commented are informed, etc. The next step after that would be to allow submitted emails to be automatically added to the comments. I looked into the MailIn addon and it is almost what is needed, but not quite.

-- RaymondLutz - 22 Sep 2009

Hi Raymond,

I think we're pretty much on the same page.

1. Commenting demands a new access control action. We already have RENAME, VIEW and CHANGE. Micha's MetaCommentPlugin implements COMMENT, so you can do things like ALLOWTOPICCOMMENT = WikiGuest. I would absolutely require that random WikiGuest (and other) users can't arbitrarily edit comments. This is easy to do if commenting is a separate action to CHANGE, as it is in Micha's plugin. However I would prefer that this action resulted in a new topic (using existing templating techniques we already use for creation of new topics). Until I read Micha's code thoroughly (and perhaps understand Foswiki internals a lot better), I'm nervous about how attackers to could inject specially crafted comments to do more than they're allowed.
2. Exactly (although, I'm not sure that DBCacheContrib can do the indexing we're talking about here). Regardless, should be able to do this with proper construction of the template topic, where some macro would populate the metadata field by expanding to the proper Web.Topic the comment belongs to. Then, threaded conversations could be achieved by setting topic parent meta. DBRECURSE can render the comment thread appropriately in the view template.
3. & 4. I am hoping that the moderation/review (viewable) status of each comment would just be standard data forms app techniques, ie. appropriate construction of a DBQUERY to only include topics with desired criteria in the view template. Something like:

%DBQUERY{"target='%BASETOPIC%' AND 'approved'='true'"
   web="%BASEWEB%"
   format="   * $dollarformfield(TopicTitle) by <nop>$dollarexpand(info.author) on $dollarformatTime(info.date)$n
      $percntDBRECURSE{\"$topic\"
         web=\"$web\"
         filter=\"parent='$dollartopic' AND approved='true'\"
         format=\"   $dollarindent* $dollarformfield(TopicTitle) by <nop>$dollarexpand(info.author) on $dollarformatTime(info.date)\"
   }$percnt"
}%

... obviously with some extra magic to show/hide comment text (probably with some jQuery)

-- PaulHarvey - 28 Sep 2009

We have several general options that have been implied in the contributions above, so let me try to present these options.

1. Embed the comments as part of the main topic, such as at the end. This the way the CommentPlugin works (or is at least one way to implement it.) Drawback is that it implies that the user will have edit rights to edit the topic, and this eliminates the concept that we can have a guest commenter. It also does not lend itself to threaded discussions, javascript-based expand/contract, and replying to a specific comment. It is also quite difficult to integrate with email reply and notices because that requires some special strings in the email to direct it to the threaded conversation. The advantage to this approach is easy refactoring of topics to eliminate repetition or to finally bring the conversation to a head. That aspect, in my view, is so important that it must be kept if possible.
2. Place the comments in a separate web with DataForm fields to allow them to be tagged with the original topic they relate to. This operation is implied by some of the comments earlier in this topic, when they state that we must allow topics to be created but not edited by Guest. Allows sorted comments using existing SEARCH macros and a report of the comments can be easily placed in the main topic (as you suggest above). Comments can be included in general searches and the base topic could be access through a DataForm field link. There is no mechanism to delete all the related comments if the main topic is deleted. Comments can be approved individually by listing all unapproved comments on a page for the administrator so they can be approved. There is no existing mechanism to approve all comments, etc. This design results in an explosion of small topics in the comments web, something that is highly inefficient on popular disk architectures with very large minimum file sizes. Refactoring the comments into the main topic is not straightforward, but at least a two-step process.
3. Include the comments in the main topic but "out of band" so they are not considered part of the main topic, but include them in the same web and couple them with the main topic (somehow). This eliminates the explosion of files and maintains DataForm fields for use by the user application and not for the comment infrastructure. There are at least three possible ways to do this:
   1. Use META fields in the main topic. These likely will require extensive escaping because the content must be quoted and all in one line.
   2. Use a marker in the file to start the comment section at the bottom and different from META fields so the section could be defined as XML, for example, and content could include HTML, etc. as well as other tags to provide for threaded discussions and email updates, etc. The nice thing about placing them in the same file is it is quite easy to refactor the comments using a super-raw edit mode which would show all the comments and their related tags.
   3. Place comments into another file with optimal structure, such as XML, as described above, and couple it with the main file perhaps with a single META field, much like attachments are today. In fact, the comments could be considered an attachment with known structure. This is advantageous because it makes it is much easier to confine searches to exclude comments or to include them. This would need to be an additional checkbox in the advanced search if comments were to be searched as well. I am tending to embrace this approach because it allows extensive flexibility without resulting in a multitude of tiny files, isolates the comments from public view until they are approved.

Any of the options can allow the comments to be extracted and placed into an mysql-based structure in the future when foswiki eventually migrates that way.

As an aside, I used to use a tool called Hypernews. (Remnants of this project are still at http://www.hypernews.org) It was popular in the 1990s but I think went out of use in 1997 or so. They had extensive discussions on the topic of collaboration, threaded comments, etc. (although I can't find these now) and their structure was primarily a base topic which was under the control of the moderator and a large number of comments which were tightly coupled with email (i.e. if you subscribe to the page, you get any comment added in your email box and a reply added your comment as a reply to that original comment on the page.) The comments could be listed in full or contracted.

Looking back at that work, I think we can learn about the hooks required in each comment and at least one way to tag them so email can be tightly integrated. For example, a typical topic might be this one: http://www.hypernews.org/HyperNews/get/computing.html so you can see the options for reading all the comments, etc. In their heyday, the hypernews "forums" were heavily used, some with thousands of subscribers and differed from popular forum tools that are still a popular hang-out, and have morphed into social-networks like Ning and Facebook. These forums and social networks do not have the characteristic of collaborative work which will eventually can come to conclusion which is the reason wiki tools are superior for collaboration (however, foswiki, etc. has lacked the tight email collaboration which is key to continued conversations and a culture of cooperation and engagement.

With all that said, I agree with the concept that we need a new operation that is specific to comments. I don't want to make the mistake of trying to replicate hypernews, since it was a monolithic monster, poorly coded, and we need to stick with the flexible macro approach.

I think we should discuss and think about our options before we start any implementation. I have set up a server and foswiki install on the server (at http://www.pubwik.org) so we can experiment there without upsetting foswiki.org or some other working install. I need to study the existing CommentPlugin and probably take a quick look at Hypernews to see what they did before I can make a solid proposal.

-- RaymondLutz - 28 Sep 2009

Hi Raymond,

After I write some docs for Micha's MetaCommentPlugin, I'll enumerate here the things it does and what it's missing.

Somehow in my poor little mind I didn't like that unauthenticated users are submitting content into the topic. But perhaps I can learn to live with that if I study the code more and ensure submitted content is aggressively filtered.

Using attachments is a very interesting approach, and I like it. But I'm not sure it will scale well either, if my recent problems with performance on DirectedGraphPlugin and AttachContentPlugin are anything to go by (Rcs versioning + attach API = slow).

It would be nice if comments in an attachment could be done append-only. You could always cache the sorted/pre-rendered comments in work_area. Or maybe each topic could just bypass the attach API and just have some SQLite file sitting in /pub

However, a serious question: we keep saying that keeping comments in the wiki is a valuable thing because comments can be refactored into stable atoms of knowledge. If that's the case, then if we don't have comments stored in topic text - but instead escaped out into a metadata field or encapsulated into an obscure attachment - would it really be a big loss to look at seriously picking up the commenting system from some other Eg. php project and including it near the original topic as an iframe somehow?

I had been attracted to the topic per comment idea because it maps well to the philosophy of the rest of the wiki, and could essentially be a traditional data forms application leveraging some new plugin.

Is scalability to 100s, 1000s of comments necessary? That's not the type of use case I had in mind, but perhaps it is for many others?

(15 minutes later)

I just had a tiny thought ... What if submitting a comment didn't actually go to a topic or attachment at all. Well, not directly.

What if we used work_area or tmp to hold the comments temporarily. Then some other action - maybe viewing the original topic - triggers a check for new comments which should be merged into (probably a dedicated) comment topic. Then, hopefully, it's just a matter of INCLUDing the comment topic from the original topic.

Format of the flat comment topic - keeping in mind E-mail functionality, rendering for CSS flexibility and javascript toggles. Comment IDs encoded in-line with comments (delimited somehow ) could be matched up with topic META that would hold E-mail, subscription info etc.

...This is why this topic is under brainstorming!

-- PaulHarvey - 29 Sep 2009

I'm continuing to think about this and my mind is starting to get to a resolution that I sort of like. The issues about my other ideas were that

1. we want to keep things in topics that are generally quite readable and editable. This is a genaral philosophy of such a wiki.
2. we want to leave things the same as much as possible, and yet allow for other options that will progress us toward perhaps an optimal database design in the future.
3. The current philosophy of the comment plugin should be possible, i.e. place comments in the same topic, if that is desired.
4. It should be possible to split the comments to a subweb (probably) where we can avoid searching over the comments unless we want to.
5. Although the comments should be directly readable, they should contain structure so we can treat them as data objects, i.e. list all comments from given users, mark comments as needing review by the moderator, render them in threads, by date, by topic or tag, etc. Thus the structure of the comments must be readable but also processable so they can be rendered in any arbitrary form.

The idea of placing them in an attachment is not a great idea, although I thought it was OK at first. Instead, I think it is best to shoot for the implementation that the comments are in a separate subweb in a topic of the same name as the primary topic. So, assume we have a topic "News" and then there would be a subweb called Comments, i.e. News/Comments that would (potentially) contain topics of the same name as the primary topic, that is if there are any comments at all.

When someone wants to add a comment to a topic, then they would click the comment button, and an edit box would come up. In my view, this should be the same wysiwyg editor (or raw editor) that is installed to edit any full topic. But in essence, the user is editing a "section" of the Comments topic, and the result is that the edited text is inserted into the proper place in the TopicName topic with additional (hidden) markup that explains who inserted it and when, and also some magic strings that let us match this up with email in and out.

In essence, this resolves to a section edit mode, which is perhaps just a subset of the regular edit script. The result of the edit is not to replace the entire topic, which is the usual course of action, but to insert the text into a given topic with information about who inserted it and when.

I hope to see the Comments subweb configured so that each commenter can request to be informed if someone makes a comment on theirs, or if anyone makes any comments by email. Indeed, it would amount to sending the result of the (approved) edited section to the user, with link to the page if they want to comment on that comment or to add a new comment to the end of the comment thread.

It may be feasible to allow a user to simply reply to the email with their comments, and using the MailInAddon, notice the magic numbers in the email and then perform an "edit section" command.

I have to move on to other tasks but that is where I am now on this an I would appreciate your thoughts and input...

-- RaymondLutz - 01 Oct 2009

Apparently MichaelDaum did this type of comment system (topic-per-comment which can be in a subweb) with his Blog plugin. On IRC I discussed the pros and cons with him a little: http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2009-10-01,Thu&sel=276-292#l276

Section editing would be a great addition to Foswiki (I would love for EditChapterPlugin to be able to lock just a particular section instead of a whole topic). However that's a bigger problem that I'd like to stay away from in addressing the comments situation (section editing would bring with it pretty big changes to the way we address topics, get locks/do versioning, conflicts/merge etc).

I have been quite busy but should be doing proper evaluation of MetaCommentPlugin "any week now"...

-- PaulHarvey - 14 Oct 2009