Feature Proposal: Centralize HTML form handling with an API for developers and end users

Motivation

This proposal originates from my work on FormPlugin and discussions I had with EugenMayer.

Although Foswiki is a web application platform, the way data is passed from web page to data storage - so through HTML forms - is not developed or standardised.

1. The lack of central form handling leads to possible security issues (for those who want to create immutable, non-wiki forms).
2. It leads to a lack of possibilities for extension developers.
3. And end users who are generally encouraged to change wiki pages themselves are faced with the task of writing HTML forms.

Some examples:

• It is not easy for topic creators to adapt html forms, for example for creating topics based on a template. Not to mention how to populate a category list dropdown from a search.
  • Even the templates for CommentPlugin are filled with html form tags, for example:

    <textarea %DISABLED% rows="%rows|3%" cols="%cols|50%" name="action_comment" 
    id="action_comment" class="foswikiInputField" wrap="soft" 
    onfocus="if(this.value=='%MESSAGE%')this.value=''" 
    onblur="if(this.value=='')this.value='%MESSAGE%'">%MESSAGE%</textarea>
    

• It is too easy for form users to change the form on the fly. For example to remove the check if the topic already exists is a matter of right-click the form (Firebug, Safari Webkit, Google Chrome) and type a different value:
  

  

• Foswiki core does not perform any checks on the input (except for the CSRF validation key).
• HTML forms are outside the realm of plugins (unless they write HTML in perl or in a template) (and Form.pm is for rendering DataForms).
• Foswiki does not provide a method for form validation and feedback.

FormPlugin does a part of this: it allows the end user to create html forms, it performs validation. But it does not provide centralized form handling with required security measures.

My proposal is:

• Create a programming API for creating, rendering, changing, validating and saving forms, to be used in core and extensions
• Expose the API with macros to end users

Proposed essential features:

• Create form object
  • Store form data in server cache before rendering to the page, then use the cache for processing the form data on submit
  • Provide "alter hooks" (from Drupal): extensions can add and remove fields or pass additional data
  • Change forms based on user, group or access rights
• Render form object to the page
• Create macro tags to create forms and fields in topics (like FormPlugin)
• Validate the form (backend):
  • Check if expected fields are in place (form is not modded by page visitor)
  • Check on empty fields or the type or format of values
  • Provide feedback on mandatory fields and the result of submitting
• Formatting and layout
  • Populate multi-select fields with list, for instance from search
  • Add typed fields, for instance for dates
  • Render fields and labels using formatting parameters

Additional nice to have features:

• Connect to frond end validation for fast interface response
• Javascript logic and behaviour
  • If this item selected, show that field
  • Behaviour onfocus, onchange, etcetera
• Value substitution (from FormPlugin)
  • Populate fields based on other field values

A good API to start with is Drupal's Form API.

Description and Documentation

Specification to be developed.

What is the best way to create/store a shared form object? What do we currently use for similar objects?

Examples

Impact

WhatDoesItAffect:

Implementation

-- Contributors: ArthurClemens - 16 Mar 2010

Discussion

What exactly do you refert to:

• (a) HTML form or
• (b) DataForms.

I guess this is about the latter.

From the drupal docu - as far as I red it - their FAPI is about creating a model + interface. Well, in that sense we already have a Form API, that's our DataForms. It creates a data model and defines an interface to enter data for it. The form is generated automatically by Foswiki::Form::renderForEdit() which in turn calls the renderForEdit() method of each formfield part of a form. There's a new plugin which leverages that API to wiki space - called FlexFormPlugin - which adds a convenient wrapper arround renderForDisplay() and renderForEdit() to use and modify the forms generated by the core Form API in various ways.

Admitted Foswiki's current From API seriously lacks features but that's what we have right now.

I am really not convinced that we need caching for forms.

It would be very cool to have a better server-side validation module.

We already have quite good cliend-side validation using JQueryPlugin 's form plugin. See this example how to create a validating user registration form (validates if wiki name and login name are still available, email address has got the proper format, etc).

Influencing the display of a form based on access rights is a beast of its own called: field level access rights. As soon as we can revoke view/edit rights per form field, they won't be displayed during renderForEdit() or renderForDisplay() presumably.

The current Form API is actually quite extensible implementign a Foswiki::Form::FieldDefinition. For instance JQueryPlugin comes with new formfield types: color, textboxlist, rating. ClassificationPlugin implements cat, and tag to edit hierarchical categories and tagging with autocompletion respectively.

In effect, wiki application developers will never really write HTML forms or macros to generate HTML forms, but use the DataForm schema and let the core render a matching form for it automatically (with means to diverge from standard layouts).

-- MichaelDaum - 16 Mar 2010

The proposal is called "Centralize HTML Form handling". It is about HTML forms.

-- ArthurClemens - 16 Mar 2010

This reads more like a brainstorming topic than a proposal. It doesn't relate itself to the existing Foswiki::Form (core) API and how it might be changed/extended/re-architected.

The scope is so wide, it should be broken up into at least five proposals.

With the existing core API, I am working on a 'richtext' formfield type for wysiywg on formfields. It will render for display differently to textarea too: Eg. $formfield() in a SEARCH will show fully-rendered, newline/TML table friendly output.

The main weakness I would personally like to solve is one of avoiding too much template glue, when organising and displaying for view/edit any topic which represents a record that is made up of one or more other topics (think one-to-many, one-to-one, has-many-through relationships). It should be trivial for users to define relationsihps between instances of forms and have their edit/view screens magically render everything nicely without hacking view templates together with convoluted SEARCH queries.

This goal alone will require a better or at even fully formalised concept of a Form object. And so in designing a "better Form API", there is more to consider than just HTML handling - we need to consider implications from a Foswiki::Meta perspective; and I assume the backend DBs that are in progress (CrawfordCurrie's XML+SQL, SvenDowideit w/MongoDB) would like to "know" about our topic object schemas (DataForm definitions).

There is a lot to think about here, and I also would like to see some of the improvements mentioned. But I don't see anything to accept or reject.

So Arhtur, please remove the date of commitment to stop the 14-day clock, or better yet - let's make this a brainstorming topic, and out of this we should be able to create solid proposals with workable specifications that can actually be accepted or rejected by the community.

Meanwhile, I will try to expand on my idea of "relational" forms.

Also check out BrainstormingDataFormDefinitions and ListFormatViewAndEditTemplate

-- PaulHarvey - 17 Mar 2010

OK, sounds like there are two topics here already. I read this as relating to GUI forms (i.e. using macros to generate input forms that capture data from end users). Arthur didn't address the second half of the problem, viz. how to get that data back into topics, and how to store it there.

The first question that appears in my mind is how you want to ask people to build these GUI forms. There are a range of approaches, ranging from the sublime to the ridiculous.

The idea of having some sort of language that defined a UI form has been explored in depth in many 4GLs which include GUI builders - examples include eDeveloper, Progress 4GL AppBuilder, Sculptor etc. To maintain the strengths of TML, careful thought has to be given to interleaving this sort of language-based approach with TML. Most would regard this as overkill.

A second approach is to build a form generation tool into TML via the use of macros that define simple GUI components, that can then be interleaved with TML (and HTML) to provide representational flexibility. This is the approach we see with FormPlugin. This approach has an attractive balance of simplicity with flexibility; however it requires almost as much knowledge of the form-building macros as many people have of HTML, and has a significant learning curve for newbies.

A third approach is interactive GUI builders, such as we see with NetBeans. In this approach, the end user interacts with a GUI which allows drag-and-drop and dialog based definition of GUI forms. There are loads of GUI builders out there. The main problem I have personally encountered with these tools is the difficulty of customising the look and feel - they are great for simple applications that fit the model, but a PITA for anything outside the norm.

It would be wise not to ignore the second aspect of this toolkit viz. how to get the generated data back into topics. Foswiki forms is one way to store data coming from UI forms, and a form definition topic might be extracted nicely from the schema generated by a form builder. However this is only one way to store user data; it might be used in other ways, for example passed in parameters to plugins, stored in topic text, even passed to entirely different (not Foswiki) services. For this reason I think that tying the GUI form builder to this representation would be a mistake.

As I understand Arthur's proposal above, he is proposing a refinement of the second approach described above viz. a FormPlugin-type macro-driven GUI builder, integrated into the core. On this assumption, I'll observe that Perl has a bunch of support that can help such as CPAN:CGI::FormBuilder, CPAN:HTML::FormFu, and others (see http://www.perlfoundation.org/perl5/index.cgi?page_name=form_processing). Any of these could provide a substrate for this approach, and help define the user-facing language. My main concern with this approach is that while it makes life easier for those immersed in TML, it still presents a considerable barrier for newbies.

In an ideal world, it would be integrated with an interactive GUI builder that would support interactive GUI building, but generating the macros as output. Perhaps a StudentProject?

-- CrawfordCurrie - 17 Mar 2010

As PaulHarvey said, this is a brainstorming topic and not a clear feature proposal with a spec that people can say yes or no to.

Paul wanted the 14-day clock stopped, so I added both his and my name in concern field.

I would love to see new features that can help people to construct those submission pages we all make for applications and have to make in HTML today.

Several plugins exist to aid people but most replace complex HTML by almost as complex Macros.

I also agree with Paul that there are many individual proposals in this.

Personally I see a number of things that needs to be addressed.

• Our API for plugin developers is very limited when it comes to accessing form definitions and you end up either calling non-API code in core or you end up writing your own parsers. We need a formal API for this.
• We need a good way for end users to build HTML forms without knowing they build a form.
  • I miss a few core (not plugin) Macros to access form definition topic content for the more expert users that can do a little HTML but end up doing searches to get for example the options for a multiselect field.
  • I miss a set Macros for building forms where the macros are a lot simpler to use than the HTML. Some plugins are already nearly there. Often it is just the documentation that lacks in those plugins. I am not sure it is correct to have this in core. They are features that can live well in plugin land.
  • I miss a graphical way to build the forms. For example by selecting formfield types in the Wysiwyg that you can drag around and where the user interface is geared towards passing information to Foswiki topics and creating Foswiki topics. Maybe also here plugin land is more natural as this allows multiple implementations to evolve and maybe even "compete".

All this starts with getting the API right so this is where we should start.

-- KennethLavrsen - 17 Mar 2010

As far as I see, the TML macros of FormPlugin are isomorph to the HTML form it generates, that is the user goes through similar steps from top to down to create a form. Either she writes down HTML, or uses FormPlugin's TML interface. I don't see this as a big advantage. It does not provide any new concepts.

DataForm have a HTML live as well. While they define a data model, they also define a gui to display and edit this data. Maybe this aspect should be leveraged to define HTML forms using a DataForm definition without it being used as a data model for data stored in topics.

For example here's hypothetical "DataForm" definition for a user registration:

Name:	Type:	Size:	Values:	Description:	Attributes:
Login details	heading				m	edit
LoginName	login	40		The login name you use to sign in.	m	edit
E-mail	email	40		Your email A valid email address. Will be kept confidential.	m	edit
Your password	password	40			m	edit
Retype your password	password+equals(YourPassword)	40			m	edit
Personal details	heading					edit
First name	text	40			m	edit
Last name	text	40			m	edit
WikiName	text+wikiword(FirstName, LastName)	40		Your name as displayed online. Note, this must be a unique WikiName. By default it is derived from your first and last name.	m	edit
Country	select	1		Select the country where you live		edit

A user would use it using %RENDERFORM{form="RegistrationForm"}% similar to what is in FlexFormPlugin and will get something like this automatically.

The main difference to what Arthur proposed and to the above is, that it builds upon our current From API, but extends it to fit additional needs. Then, users do not create any HTML nor anything isomorph to it (STARTFORM...TEXTAREA ... ENDFORM). Instead, they just call a RENDERFORM and the form will be displayed.

That's what my initial comments were trying to illustrate.

For instance, there's a DataForm definition, that is used to render a complex search form for it to search for topics that have this DataForm attached to it. Again, the form definition itself is reused for different wiki application needs.

I don't see the connection betweem FormPlugin and Drupal's Form API. Drupal's and Foswiki's data form API are motivated for very similar reasons.

I am by far not as enthusiastic about Drupal as a lot of its developers with regards to their technology.

-- MichaelDaum - 17 Mar 2010

It looks we cannot make this actionable soon as we have much problems with definitions.

I have changed this topic to Brainstorming. So no concrete proposal yet.

I see almost all has misread the proposal. It is not about Data Forms, it is also not focusing on creating a macro language.

To have macros, there first must be a good API for creating html forms. I would like to first focus on the flow of data:

• creating a form object through code
• making this a shared object so it can be altered
• having this rendered on the page for viewing
• handling validation
• handling feedback of validation
• handling succesful submit: passing on the data to other modules
• handling feedback of submit

When we have this in place we can look at the interface we present to end users.

-- ArthurClemens - 17 Mar 2010

Arthur, I think we only misread because the proposal was very broad in scope.

So to start with, let's talk about form objects and manipulating them in code.

What is required of a form object? Ultimately it will have to be represented in the wiki topic in some way. Is the current table approach sufficient? What enhancements are necessary?

Personally I have a great need to better separate the data model from GUI concerns, but I understand why we may choose not to do this in Foswiki (more steps for ordinary users to create DataForms apps).

• Form fields have an identifier (a name). In my work they are always matched up with a topic (not uncommon practice). Perhaps of note though, this field topic also has a form attached to it to help formally describe what belongs in the field, and what the data means.
• Later when we export our data set as XML (for consumption by aggregators and rigid, curated database stores), not only can the field data be marked up, but also the definition of what the markup means is delivered in the XML bundle. This is the way of the future. Linked data that also encapsulates meaning.
• Which brings me to an inconvenient point. Currently the field topic also defines its type and GUI attributes (textarea, radio, checkbox, mandatory, size, etc).
  • When a user needs to build up a new collection of data documenting a set of things, they need to make a form. The way they go about it is by creating these "term" definition topics as needed, but hopefully mostly choosing from existing term definitions (a rudimentary namespace mechanism is implemented to handle conflicting meanings for same term/field names).
  • After creating terms or choosing existing ones, their list of desired terms (really just a list of topics) is finalised and a button is pressed to create a traditional Foswiki DataForm (among other things). With some template trickery, it is immediately available for data entry in their target web.

What I wanted was a "drag and drop" UI for users to build their forms. There is a similar builder for term definitions. Problem is that the GUI appearance of the terms when finally instantiated into a form, is tied to the definition topic.

If a user wants the same term (in a semantic sense) to be used in different ways (with different representations - eg. even just "80x10" textarea in one instance, and "100x2" in another), right now their only option is to hack the DataForm "out-of-band" (outside my formbuilder app).

I do apologise for the rambling explanation. Hopefully I have conveyed why I am interested in being able to have separate schema model to GUI representation.

My question, Arthur and others: Is this out of scope for this work we are considering here?

Or would it be useful to have this separation of data model + view + ... in Foswiki? Perhaps it is not possible to accomplish without hurting complexity for ordinary users. I would like to give it consideration though.

Ultimately, it would be highly useful for the final data entry to decide what representation they are going to use to populate a field.

-- PaulHarvey - 17 Mar 2010

Hello folks,

i thought i might clarify some things, as i know both APIs pretty well.

FAPI == Dataforms?

No, not at all. The Drupal FAPI has absolutely nothing to do with the data-model. If you want to see it in a MVC pattern, FAPI is arround the model and the view, while the view is seperated as the "FAPI renderer".
You can use the FAPI to show any forms for any actions, not only on topic (node) meta data.

FAPI - what is it?

The FAPI does abstract the complete formular handling on the backend and it also has a layer for rendering (which can be exchanged and partially overriden). Ever form element we know from html (select, checkbox, textfield ..) is a element in the FAPI. Each element has its options, data and its renderer. You can add new complex elements using the api (lets say and image upload + crop + title + datefield) which will be then rendered using your renderer, and the data (get data, set data) is handeled using your element controlers. The whole API is enormous and really flexible. Its where all, every application is build whe you use drupal.

Runtime on the code scope you have a huge associated array, holding all those elements, which are themself embedded in the "form element" ..which has attributes like action and so fort. The form element also has tokens to secure against CSRF and much much more concerning form-hacking-security.

FAPI - worth it?

If i would have a top-3 why i love using drupal, FAPI would be on the first place. FAPI #1 with the developer hat on, FAPI #1 with the applicaiton build hat on and FAPI #1 as end-user (due the consitent look and feel ).

-- EugenMayer - 18 Mar 2010

A use case/story/narration/paragraph describing the problem that this feature wants to solve could be useful to focus the discussion.

(note that "centralized html form handling" is describing a solution, not a problem).

-- RafaelAlvarez - 18 Mar 2010

Rafael, what would you like to see besides the examples in the Motivation section?

-- ArthurClemens - 18 Mar 2010

From what I see drupal's form api is quite similar to foswiki's. Admitted, our form api was initially motivated by the Structured Wiki idea. Today, we can use it to reuse these form definitions for other purposes as well.

-- MichaelDaum - 19 Mar 2010

So a first step could be: make the Foswiki form API generic to create any web form, through code and user interface.

-- ArthurClemens - 19 Mar 2010