Integrate generated content

It is trivial to use Foswiki with topics stored as text files and it is straightforward to produce text files to be used as Foswiki topics, especially if the conversion is uni-directional or once-off.

Meta data - TOPICINFO

Foswiki assumes that every topic has a %META:TOPICINFO{...}% tag that specifies the revision info, who last edit the topic, etc. To see this metadata, append ?raw=debug to the URL, e.g. for this topic. You can get away without this, but the resulting topics look odd compared to regular topics (e.g. the topic date looks wrong).

It is really easy to generate this in perl. The date format is as produced by perl's time() in scalar context. I generate it like this:

  print $fh "\%META:TOPICINFO{author=\"GeneratedContent\" date=\"".time()."\" format=\"1.1\" version=\"1.1\"}\%\n";

Foswiki expects a username, so consider adding a user called GeneratedContent or something similar, so that it is obvious when content is computer-generated. And then create the corresponding user topic to explain this.

Revision history with RCS

You can also get a revision history on your generated topics if you are using an RCS-based store (either RcsWrap or RcsLite) and RCS is available to your scripts.

The RCS commands below are based on (default) values from my LocalSite.cfg, and assume a (debian/ubuntu) linux platform, although they should also be portable to Windows simply by changing the paths - YMMV.

The RCS file must be initialised before the first checkin:
/usr/bin/rcs -i -t-none -ko /path/to/foswiki/data/Web/NewTopic.txt,v

If the RCS file already exists (because the topic was checked in before), then the RCS file must be locked:
/usr/bin/rcs -l /path/to/foswiki/data/Web/NewTopic.txt

Then check in the file:
/usr/bin/ci -mPutACommentHereIfYouLike -t-none -u -wGeneratedContent /path/to/foswiki/data/Web/NewTopic.txt

If you are checking your generated topics into RCS, then the topic version in %META:TOPICINFO{...}% should be correct. Parse the output from RCS rlog /usr/bin/rlog -h /path/to/foswiki/data/Web/NewTopic.txt and extract the current version from the head: line.

Example code

Here is how all this might be done:

Here is a routine that writes a %META:TOPICINFO{...}% tag to a file, using the next version if the file was previously checked in:

sub write_next_version_meta_topicinfo
{
  my $fh = shift; # filehandle of temporary output file to write to
  my $filename = shift; # /path/to/foswiki/data/Web/NewTopic.txt
  
  my $version = "1.1";

  if (-e "$filename,v")
  {
    # file was previously checked in
    my $rlog_command = "/usr/bin/rlog -h $filename";
    my @rlog_output = `$rlog_command`;
    if ($? == -1)
    {
      die "Cannot run '$rlog_command': $!";
    }
    for (@rlog_output)
    {
      if (/^head: 1\.(\d+)/)
      {
        my $revision = $1;
        $revision++;
        $version = "1.$revision";
        last;
      }
    }
  }
  print $fh "\%META:TOPICINFO{author=\"GeneratedContent\" date=\"".time()."\" format=\"1.1\" version=\"$version\"}\%\n";
}

After generating a topic into a temporary file, you may want to compare it with the existing topic and only replace the existing topic if the content actually changed. The %META:TOPICINFO{...}% contains the date, so here is a routine for comparing two files but ignoring this tag:

sub did_content_change
{
  my ($old, $new) = @_; # filenames

  # Read in the content of the files
  local *OLD;
  open OLD, "<", $old or die "Cannot open '$old' to read: $!";
  my $old_content = '';
  $old_content .= $_ while (<OLD>);
  close OLD or die "Error closing '$old': $!";

  local *NEW;
  open NEW, "<", $new or die "Cannot open '$new' to read: $!";
  my $new_content = '';
  $new_content .= $_ while (<NEW>);
  close NEW or die "Error closing '$new': $!";

  # Remove topicinfo metadata, since that can change even when the content does not
  $old_content =~ s/^\%META:TOPICINFO{.*?}\%//;
  $new_content =~ s/^\%META:TOPICINFO{.*?}\%//;
  return ($old_content ne $new_content);
}

Here is a routine for checking in a topic. It initialises the RCS file if checking in for the first time:

sub checkin
{
  my $txt_file = shift;

  my $rcs_file = "$txt_file,v";
  if (not -e $rcs_file)
  {
    # Not previously checked in, so initialise the RCS file
    my $init_command = "/usr/bin/rcs -i -t-none -ko $rcs_file 2>&1";
    my $init_result = `$init_command`;
    if ($? == -1)
    {
      die "Cannot run '$init_command': $!";
    }
    my $return_code = $? >> 8;
    if ($return_code != 0)
    {
      die "$init_result\n'$init_command' returned $return_code";
    }
  }
  else
  {
    # Previously checked in, so lock before checkin
    my $lock_command = "/usr/bin/rcs -l $txt_file 2>&1";
    my $lock_result = `$lock_command`;
    if ($? == -1)
    {
      die "Cannot run '$lock_command': $!";
    }
    my $return_code = $? >> 8;
    if ($return_code != 0)
    {
      die "$lock_result\n'$lock_command' returned $return_code";
    }
  }

  my $ci_command = "/usr/bin/ci -mgenerated -t-none -u -wGeneratedContent $txt_file 2>&1";
  my $ci_result = `$ci_command`;
  if ($? == -1)
  {
    die "Cannot run '$ci_command': $!";
  }
  my $return_code = $? >> 8;
  if ($return_code != 0)
  {
    die "$ci_result\n'$ci_command' returned $return_code";
  }
}

-- MichaelTempest - 15 Apr 2010