Using bulk_copy.pl script and migrating user data to upgraded Foswiki version on Windows server

I have two questions related to migrating data from Foswiki version 1.1.9 to Foswiki version 2.1.6 (a major version upgrade). The old wiki is running on a Windows 2003 server while the new wiki is on Windows Server 2016. Both wikis have existing content which I want to preserve. I want to migrate only specific webs and topics and the user data from the old server to the new server. I don't want to overwrite the System or Main webs or the preferences on the new server.

1. How do I migrate user data from an older version of Foswiki to a newer one?
2. To migrate the topics and webs, I'm using bulk_copy.pl. However, I'm running into issues with this script and missing Perl modules. See below.

Attempts to run the bulk_copy.pl script result in a missing module warning (running Perl 5 version 26.2):

Shell command: perl bulk_copy.pl

Result: Can't locate Pod/Usage.pm in @INC (you may need to install the Pod::Usage module) (@INC contains: /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at bulk_copy.pl line 39.

OK, then I'll install Pod::Usage.

Command: cpanm Pod::Usage

Result: --> Working on Pod::Usage
Fetching http://www.cpan.org/authors/id/M/MA/MAREKR/Pod-Usage-1.69.tar.gz ... OK
==> Found dependencies: ExtUtils::MakeMaker
--> Working on ExtUtils::MakeMaker
Fetching http://www.cpan.org/authors/id/B/BI/BINGOS/ExtUtils-MakeMaker-7.34.tar.gz ... OK
Configuring ExtUtils -MakeMaker-7.34 ... OK
Can't locate ExtUtils /Manifest.pm in @INC (you may need to install the ExtUtils::Manifest module) (@INC contains: FatPacked::25770201176=HASH(0x600061058) /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at /c/Perl64/site/bin/cpanm line 119.

Looks like it is missing an additional module. No problem, just install it...

Command: cpanm ExtUtils::Manifest

Result: --> Working on ExtUtils::Manifest
Fetching http://www.cpan.org/authors/id/B/BI/BINGOS/ExtUtils-Manifest-1.71.tar.gz ... OK
==> Found dependencies: ExtUtils::MakeMaker
--> Working on ExtUtils::MakeMaker
Fetching http://www.cpan.org/authors/id/B/BI/BINGOS/ExtUtils-MakeMaker-7.34.tar.gz ... OK
Configuring ExtUtils -MakeMaker-7.34 ... OK
Can't locate ExtUtils /Manifest.pm in @INC (you may need to install the ExtUtils::Manifest module) (@INC contains: FatPacked::25770201176=HASH(0x600061058) /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at /c/Perl64/site/bin/cpanm line 119.

So, the ExtUtils::Manifest module cannot install because it cannot locate itself in @INC? Is my ActivePerl installation incomplete? It looks like this has been reported as a bug already, but dismissed: https://rt.perl.org/Public/Bug/Display.html?id=125195. Is there a way around this?

-- MorganGoodrich - 15 Oct 2018

Workaround: Do not try to install Foswiki on Windows. As you are already about to migrate, why not install it on a virtual Linux machine. A lot less trouble and a better overall result in terms of functionality, quality and reliability.

-- MichaelDaum - 16 Oct 2018

Thanks Michael, but I do not have the option of installing on Linux. There is already a wiki established on the Windows server I am trying to migrate to and I cannot set up a Linux server. If I had the option, I certainly would. Do you have any suggestions for migrating the user data, such as a list of the files which are critical to maintaining users' edit history, access controls, groups, and passwords, etc? I am not sure how to manually merge the files, and which files are needed to capture this information. Thanks in advance.

-- MorganGoodrich - 16 Oct 2018

A few thoughts about migrading data from an older installation. One option is, instead of useing bulk_copy.pl, use CharsetConverterContrib to update/cleanup the character encoding and then manually merge your content. (More on that in a minute.) If you go this route, you have to stick with the RCS style store, but I'm not seeing a huge benefit to converting to PlainFile store anyway. The benefit is this approach does give a bit more flexibility in copying over just what you need.

Once you're run CharsetConverterContrib, then you can manually select what webs & topics you want to copy over. Here's a few notes on what to include:

• Basically you want to copy only your local data and pub directories. This would include primarily any new webs that you've created, plus your user, group and other miscelleneous topics created in the Main web.
• You do not want to copy over the System directory (either data or pub) as it's recommended that this web is not locally modified. The one noteable exception to that is UserCommentsTemplate that contains any custom comment templates you might have created. It's possible that you might have a few other customized topics in the System web (even if not recommended). I'd recommend just holding these aside for the time being. BTW, to see what those topics might be, take a look at the System.WebChanges and see what topics were modified by your users. Then look at the individual topics to see if the changes are important. Chances are there will be none or very, very few and these might be manually copied over. In many cases, any such preference settings or customized topics would be better stored in the Main web of your new installation.
• To move the content into your new installation, you simply copy the data and pub directories for each web into the corresponding directories in new installation. Note that each web will have a directory in the data directory which contains the topic (suffix .txt) and history file (suffix .txt,v) and a corresponding directory in the pub directory that contains any attachments and their history.
• For the Main web, what I did is copy over all the topics except any of the default topics (which mostly begin with "Web..."). You could copy over SitePreferences and WebPreferences or manually copy your prior settings to those topics in your new installation (while at the same time reviewing your settings). You can also copy over the WikiUsers topic as that has not changed much. If you don't copy it, you need to at least manually copy over the list of users from that topic (as that topic is what links users login name to their user topics.
• One very important file to remember is the data/.htpasswd file since this contains the passwords and emails of your registered users.
• Two other webs that you may or may not want to include is the Sandbox web and the Trash web, depending on if there's important content there you want to keep. Similar to the Main web, I'd only copy over the topics that don't start with "Web".
• If this historical WebStatistic data is important to you for the Main, Sandbox and Trash webs (usually it would not be), you could copy over that topic also.
• You might also check Main.WebNotify topic in your old installation which contains subscriptions info to email notifications of changes.

That's basically it. I prefer this approach because it gives a bit more control than bulk_copy.pl. However, if you do want to convert the store, then you need to use that option. Most of what I've said here still applies and some is built into that script (for example, excluding the System file). To specifically answer your question about files critical to maintaining users' edit history, access controls, groups, etc. That info is contained in the topic history of individual topics, the user and group topics in the Main web and the password file.

Hope this helps clarify your options and the process.

-- Main.LynnwoodBrown - 16 Oct 2018 - 21:19

This explanation is exactly what I've been looking for. Thank you for taking the time to detail the steps so thoroughly. Really appreciate it!

-- MorganGoodrich - 17 Oct 2018

An update with some follow up questions: I was able to run the CharsetConverterContrib to convert the old wiki topics. Also updated the old wiki to the utf-8 store. Eventually the server running the old wiki will be decommissioned (which is the ultimate goal), so it wasn't really necessary to update the store, but it's done anyway. I copied the data and pub directories from the old to the new wiki. The old topics and webs integrate well with the already-existing webs and topics on the new wiki. Also migrated user topics and passwords and updated the WikiUsers list, as the old wiki contains many users which are not necessarily established on the new wiki. Haven't yet had the old users test their sign-ons in the new wiki, but I expect this will go without issue.

The issue I'm still seeing, which I've had from the start, is that the old topics aren't migrating the author and revision history correctly. The topic edit dates will be accurate, but all topics from the old wiki have one revision and the author is UnknownUser. I would like to preserve the revision history with the author names in the old topics.

In troubleshooting this, I noticed the new wiki 's data folder is different from the old wiki's data folder. The new data folder contains ,pfv folders for each topic, each containing an ATTACHMENTS folder, text files and M files with author and revision history. For example, the new wiki data folder structure is data\WebName\TopicName,pfv\ATTACHMENTS and data\WebName\TopicName,pfv\{text files with revision history and author}. WebName \ also contains AttachmentName.txt files with attachment revision history and authors. The data folder from the old wiki however does not have ,pfv folders. The old wiki is data\WebName\{.txt and .txt,v files with topic names}. Both wikis have the .changes file.

How can I update all the topics from the old wiki so their revision history contains the author and previous revisions? Is there a script which converts the .txt and .txt,v files to the new wiki format where TopicName,pfv folder contains the revision history and author?

-- MorganGoodrich - 25 Oct 2018

Oh this isn't good. You may end up having to start over with the migration.

Foswiki has two backend "storage" systems - the original RCS based store, history is contained in files named "TopicName.txt,v". And a new "PlainFile" store where history is kept in a topic specific sub-directory. "TopicName,pfv" Attachments have the equivalent history conventions.

Unfortunately these stores are not compatible with each other, and Foswiki should be complaining in "configure" that it's detecting files from both store. In addition unless you disabled an option, Foswiki should be crashing rather than accidentally saving a conflicting history. The only time you can safely change the Store type without dealing with bulk_copy is when there is no history at all - no ,v files and no ,pfv directories.

The only way to convert from Plain File to RCS or back is with the bulk_copy utility. There isn't any way to deal with a mixed store of both types. I'm not sure what to suggest as the best way to recover.

• Since you were unable to get bulk_copy running on the old server, maybe it will work on the new server. Remove all of the migrated .txt files to get back to the new installation before you merged. Then use bulk_copy to convert the new wiki to RCS.
• If that doesn't work, and you can tolerate losing history on the new server, strip out all of the ,pfv directories from the new data and switch to the RCS store. Then merge in the old RCS data again.

-- GeorgeClark - 26 Oct 2018

Well, good news is the bulk_copy script appears to be moving forward when run on the new server and actually attempts to copy instead of complaining of missing perl modules. Now I'm getting a new error - haven't researched it too much yet so forgive me if this is already well-documented.

I'm running this command on the new server, which has a copy of the old wiki next to the new one in the C: drive.

Command: c:\Foswiki\Foswiki-2.1.6\tools>perl bulk_copy.pl --xweb System --xweb _default --xweb _empty --latest '*.WebStatistics' C:\Foswiki\Foswiki-1.1.9\Foswiki-1.1.9\bin C:\Foswiki\Foswiki-2.1.6\bin

Result: xweb: System,_default,_empty latest: '*.WebStatistics' Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(\$Foswiki::cfg{ <-- HERE [[A-Za-z0-9{}]+})/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Configure/Load.pm line 189. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/^%META:TOPICINFO{ <-- HERE .*?}%/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Meta.pm line 3373. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/^%META:TOPICINFO{ <-- HERE .*?}%$/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Meta.pm line 3391. Copying to v2.1.6 (Foswiki::Store::PlainFile) Copying from v1.1.9 (Foswiki::Store::RcsLite) Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/^%META:TOPICINFO{ <-- HERE (.*)}%/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Store/VC/Handler.pm line 268. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/^%META:TOPICINFO{ <-- HERE (.*)}%$/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Store/VC/Handler.pm line 328. Useless use of greediness modifier '?' in regex; marked by <-- HERE in m/^([0-9]+)\.([0-9]+)\s+date\s+(\d\d(\d\d)?(\.\d\d){5}? <-- HERE );$/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Store/VC/RcsLiteHandler.pm line 304. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/^\*<-- HERE "Related topics:".*$/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Users/TopicUserMapping.pm line 1012. Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/^\*<-- HERE "Related topics:".*$/ at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Users/TopicUserMapping.pm line 1025. Array passed to stat will be coerced to a scalar (did you want stat $_[0]?) at C:/Foswiki/Foswiki-1.1.9/Foswiki-1.1.9/lib/Foswiki/Logger/PlainFile.pm line 57. ERROR: Could not open logfile /tools/Foswiki-1.1.9/working/logs/error.log for write. Your admin should 'configure' now and fix the errors!

Old wiki's configure shows {Log}{Dir} is $Foswiki::cfg{WorkingDir}/logs and {Log}{Implementation} is Foswiki::Logger::PlainFile, but no errors or warnings in Logging and Statistics. There are other (unrelated?) warnings to do with the environment paths, mail and password encoding. I'm not really sure where else to look.

Is this related? https://foswiki.org/Tasks/Item13466 There aren't any utf-8 characters in either of the directories.

-- MorganGoodrich - 27 Oct 2018

Commenting is disabled while not logged in