There needs to be a general refactoring of all the different table parsers into a central service (or, better, a topic object model that allows view of a topic as a sequence of alternate table and text blocks)

This impinges on a lot of code. The rest of this topic collects together comments about various affected areas.

There are all sorts of difficulties here, because topic content changes throughout the rendering pipeline. Expansion of a macro can easily create a new table, for example.

EditRowPlugin

There is already a reusable table parser in the EditRowPlugin that might be useable. It is closer to what we need (it slits topics into alternate text and table sections)

EditTablePlugin

TablePlugin

TWiki::Data::DelimitedFile

(was reported as "TWiki::Data::DelimitedFile needs exposing in TWiki::Func" by MartinCleaver)

TWiki::Data::DelimitedFile has functionality that should be usable throughout TWiki and its plugins. However, it is not well documented nor is it exposed. Sooner or later someone will end up wasting their time duplicating the functionality because they need it and don't know it's already been implemented. This will be bad because then we'll have wasted time and have incompatible versions.

It will read from a file into a tabular data structure or save a file from the tabular data structure containing delimited (e.g. tab, comma or pipe) entries, ensuring that the lines read conform to the heading row.

sub read {
    my (%settings) = @_;
    my $filename = $settings{filename};
    my $content = $settings{content};
    my $delimiter = $settings{delimiter} || "\t";
    my $rowstart = $settings{rowstart} || '';
    my $rowend = $settings{rowend} || "\n";
    ...
    return (\@fieldNames, %data);




sub save {
    my (%settings) = @_;
    my $filename = $settings{filename} ||
      throw Error::Simple( 'Save: filename parameter is mandatory' );
    my $delimiter = $settings{delimiter} || "\t";
    my $rowstart = $settings{rowstart} || '';
    my $rowend = $settings{rowend} || "\n";
    my %data = %{$settings{data} ||
                   throw Error::Simple( 'Save: data parameter is mandatory' )};
    my @fieldNames = @{$settings{fieldNames} ||
                         throw Error::Simple( 'Save: fieldNames parameter is man
datory' )};

There is no testsuite for this module but it is exercised as a consequence of testing bulk register in the RegisterTests.pm

There appears to be an adverse call to Data::Dumper in save() - perhaps this has not been reported as an error because only read is used? I don't recall.

-- MC

Crawford kindly factored out TWiki::Data::DelimitedFile during http://develop.twiki.org/trac/changeset?new=twiki%2Fbranches%2FMAIN%2Flib%2FTWiki%2FUI%2FRegister.pm%4013373&old=twiki%2Fbranches%2FMAIN%2Flib%2FTWiki%2FUI%2FRegister.pm%4013364

This code provided TWiki with an API to process a TWiki table line by line.

Is there something else that does the same?

Ultimately I want to create a set of topics, one per line, from a TWiki table. Just like Bulk Registration does.

-- MartinCleaver - 16 Oct 2008

There's some history here. From an architectural perspective, we really need to get away from the plethora of island parsers/readers we have in TWiki. I originally looked at TWiki::Data::DelimitedFile with a view to re-using it wherever table parsers were required in TWiki. I also looked at using the CPAN equivalent, CPAN:Text::Delimited. However both these readers suffer from being very limited and plagued by "special cases" (for example, row extension using \, embedded | etc. etc.), and neither is event-driven, so I couldn't find any reasonable way to re-use either of them anywhere else. In the end I decided just to flatten out that one application of Data::Delimited (15 lines of code) because I wanted to discourage re-use of this approach elsewhere in the core.

My intention was to move towards use of an object model for topics (part of the TopicObjectModel) that would support a "table view", but this work has been stalled since the SF summit. What we really want to head towards is:

my $tom =  $session->{store}->readTopic(undef, $path, undef );
my $tit = $tom->tables(); # get tale iterator
while ($tit->hasNext()) {
    my $table = $tit->next();
    my $rit = $table->rows(); # get row iterator
    while ($rit->hasNext()) {
        my $row = $rit->next();
        ...
    }
}

-- CrawfordCurrie - 18 Oct 2008

If you end up with one central 'tom' object, what't the benefit of using event driven table (or topic) parsing?

-- ArthurClemens - 08 Mar 2009

It's a question of what can be achieved in the available time. It would be neat if we had a topic object model that presented "views" of the topic - for example, allowing you to view the topic as a list of sections, or a list of text lines with embedded table objects. However the issue with that is that you can't know in advance all the possible views that will be required. So the idea is to allow views to be attached to the meta object. I originally envisioned that the views would themselves be event driven, and would be implemented using an event driven parser. On reflection, a simple iterator model might be better (this is how EditRowPlugin works) than event driven.

my $topicObject = Foswiki::Meta->new($session, $web, $topic);
my $view = Foswiki::TableView->new($topicObject); # instantiate a view of the object
while ($view->hasNext()) {
    my $obj = $view->next();
    if (ref($obj)) { # is a table object
       ... do tabley stuff
    } else { # if a line of text
        ... do texty stuff, or just ignore it
    }
}
$view->save(); # save the modified table view back to the topic

Nothing is required in Foswiki::Meta to support this; Foswiki::TableView would just use a Foswiki::LineIterator to iterate over the text, build up the tables, and implement a standard Foswiki::Iterator interface for callers. it's also relatively easy to see how to do mixins, where different views are stacked to create new views. For example, a Foswiki::SectionView could be written to work off any other iterator; so you could pipe a Foswiki::TableView iterator into a Foswiki::SectionView to get sections and tables.

On balance this is probably a better approach. Event driven parsers can be tricky to understand, tricky to write and trickier to debug.

-- CrawfordCurrie - 09 Mar 2009

Woo hoo, didn;t even realise this was here. Closed this a while ago.

-- CrawfordCurrie - 12 Jan 2015