Feature Proposal: Enable internationalisation of plugins

Motivation

In order to provide localized versions of plugins, a developer has to hack the local $language.po files to add his strings, which is unmaintainable, unpracticale and error-prone.

Description and Documentation

As explained on Tasks.Item9697, an easy way to do this would be to create a directory per language, and have the core put its stuff there, and each plugin too. Core would then simply read them all.

Another step will be to have configure or BuildContrib generate .mo version of the .po using msgfmt. The core already supports loading the compiled .mo version.

Maintenance Scripts

We would need a few scripts in order to manage translations over all projects like:

• detect in which projects maketext strings are available
• enable translation in a project (adding locale directory, create template file and all language files)
• adding new/missing languages to all projects
• ...

Pootle Setup

A nice side-effect would be that pootle would be happier and no longer requiring symlink hacks that Andre did to have it work.

• Can pootle work straight on any SVN checkout we mentioned here? If so how? -- AndreUlrich
  • Yes, you can define the type of directory structure you have in your project in pootle, so the schemes mentionned here work. -- OlivierRaginel

To set up the translations a simple script could transform a SVN checkout into a format pootle expects:

• simple SVN checkout of the whole tree on pootle server
• Script to create links for every project (addon, plugin, skin) within the language directory
  • go to project directory on SVN checkout like JQueryPlugin/locale
  • link all language files into pootle e.g.: es.po -> pootle/es/JQueryPlugin.po

This shouldn't be necessary. We would simply create a pootle project per translated extension. This might be a bit of a pain to maintain (for main translator rights mostly), but shouldn't be too much of a burden, hopefully. Otherwise, I'm sure we can script it to automate the creation of a new plugin translation in pootle by duplicating another plugin.

Examples

• For foswiki core we would split them in a directory per language:

Proposal	Core German translation	Core Portable Object Base Translation	Plugin POT		Plugin German translation
			In SVN	Installed	In SVN	Installed
Current	core/locale/de.po	core/locale/Foswiki.pot	n/a	n/a	n/a	n/a
After (Alternative 1)	core/locale/de/core.po	core/locale/core.pot	JQueryPlugin/locale/JQueryPlugin.pot	.../locale/JQueryPlugin.pot	JQueryPlugin/locale/de/JQueryPlugin.po	.../locale/de/JQueryPlugin.po
After (Alternative 2)	core/locale/de.po	core/locale/template.pot	JQueryPlugin/locale/template.pot	core/working/work_areas/translations/template/JQueryPlugin.pot	JQueryPlugin/locale/de.po	core/working/work_areas/translations/de/JQueryPlugin.po
After (Alternative 3)	core/locale/core-de.po	core/locale/core-template.pot	JQueryPlugin/locale/JQueryPlugin-template.pot	core/working/work_areas/translations/template/JQueryPlugin-de.pot	JQueryPlugin/locale/JQueryPlugin-de.po	core/working/work_areas/translations/de/JQueryPlugin-de.po
After (Alternative 4)	core/locale/de.po	core/locale/Foswiki.pot	JQueryPlugin/locale/JQueryPlugin/Foswiki.pot	.../locale/JQueryPlugin/Foswiki.pot	JQueryPlugin/locale/JQueryPlugin/de.po	.../locale/JQueryPlugin/de.po

• Alternative 3 pros: Use a unique name which will not be renamed like in Alternative 2 and doesn´t depend on context directory (de/JQueryPlugin.po) like in Alternative 1. They would easily be identifiable with PROJECTNAME-LANG.po
• Alternative 4 pros:
  • No changes in the current files
  • Easy translation of plugins, just create a locale/ directory
  • Easy migration of xgettext script, as structure is very close for a plugin and the core
  • Micha's code already works for this
• Alternative 4 cons:
  • One of the goal of this proposal was to try and cleanup the core. But in fact, core's directory structure is OK. Pootle can take the same flat structure now, so no need to pass through symlinks hell.
  • 2 levels in locale/:
    • .po files, which are the core translation
    • directories, which are plugin translation
    • We could get rid of this drawback by moving the core files inside a core/ sub-directory (eg: core/local/core/Foswiki.pot)

Impact

It allows l10n of extensions, with very low impact on core. Users with currently modified .po versions will have a hard time upgrading, but they always did, so...

WhatDoesItAffect:

Implementation

-- Contributors: OlivierRaginel, AndreUlrich - 16 Nov 2010

Discussion

I would rather see a simpler structure on Foswiki, with less directories to be created (see Alternative 2 or 3). There can be one locale directory for every project (addon, plugin, skin) containing the language files named with their language id:

• JQueryPlugin/locale/foswiki.pot
• JQueryPlugin/locale/es.po
• JQueryPlugin/locale/de.po
• ...

Regarding installation I would suggest to aggregate language files of installed extensions and put them into the working directory.

-- AndreUlrich - 17 Nov 2010

I don't see any harm in leaving the .pot and .po files for the Plugins in a sub-directory of the Plugin itself.

But for performance purposes, wouldn't it be better:

• that Foswiki should not use them directly, but rather should always compile them as .mo
• that before compilation, Foswiki merges the existing .po files (for core and all installed plugins) into a single .po file (is that what you call "aggregate" André ? -> Yes -- AndreUlrich), and generates one single .mo file. That would happen:
  • every time a new plugin with translations is installed
  • every time an existing plugin with translations is removed (if possible)
    • Problem here is only plugins will register properly, not AddOns or Skins, or? -- AndreUlrich
• that if you hack the .po files in the sub-directory of the Plugin, a simple shell command and a simple button in configure are available to have the compilation regenerated

If a translation is already provided for given string in core, the proposed translations in the plugins should not have precedence. Merging the translations forces us to deal with this problem at this time, instead of realising that because a plugin has been installed, it has high-jacked a given string everywhere.

Maybe also the translation process should also be taken into account here.

At translation time, the translator working on a plugin should be aware of existing translations in core and maybe also know about suggested translations of the same string available for other plugins, which may, or may not be installed in a given configuration. Precedence between plugins could be something difficult to handle. But if the translators know what they are doing when they translate, it would certainly make things better.

-- RaulFRodriguez - 17 Nov 2010

It would really be a great step forward to make this plugin aware. Having it in wiki apps land would be even better, by let normal wiki users edit a translation table in some topic. Otherwise, i18n will still remain a perl thingy. Besides it needing a webserver reload for persistent perl envs.

Any idea how to proceed on this one ?

-- MichaelDaum - 13 Jul 2011

I've got some changes to I18N.pm to load additional lexicons of plugins from

locale/<plugin-name>/<lang>.po

Note that this scheme differs from the ones proposed above, which is sort of happened by accident as I hacked this on the train with no internet access to review the above proposals. Now I wonder why the above did not come with the obvious solution. Or am I misunderstanding the proposed locale directory schemes?

I really need to get something done here as people are constantly asking for proper translations and are hacking their stuff into core po files ... which is very very bad.

-- MichaelDaum - 18 Dec 2011

Alright, so the attached patch is imho the simplest possible thing that could work. Any objections that I check that in to foswiki/trunk?

-- MichaelDaum - 20 Dec 2011

Yes, one strong objection; there is absolutely no documentation of what you have done, or how an extension author would use it., whether it works for extensions or only for plugins etc etc etc.

If that documentation is provided I would have no objection. Without it, I object on the strongest possible terms.

-- CrawfordCurrie - 20 Dec 2011

Hm, documentation. Interesting concept.

-- MichaelDaum - 20 Dec 2011

As Micha wrote some code, and triggered a new discussion on IRC because he used his code for SolrPlugin, I took the liberty to update the proposal. I think with Andre we aimed too high trying to make everything right. Proposal #4 has minimal impact, but still allows easy translation of plugins.

It still needs work, of course:

• Documentation, which Micha started in Item9697
• xgettext modifications, which I'm working on
• pootle integration, which will be a manual step anyway

Anything else?

-- OlivierRaginel - 09 Jan 2012

Made some changes to alternative 4 as I don't see the point of having the translation files in different locations when in svn versus when installed (kiss). So when pseudo installing the translation files in a development environment, you will only need to soft-link <plugin>/locale/<plugin> to core/locale and you are done. This is only one link versus one link per translation file. Likewise, getting rid of a plugin's files is easier as well.

-- MichaelDaum - 20 Dec 2011

Thanks Micha. I thought I wrote it like that too, but I must have been confused by the other propositions. Now we need comments from Andre and Crawford, who were the ones expressing concerns.

-- OlivierRaginel - 09 Jan 2012

yes, one thing - please add locale stubs to the create_new_extension tool - perhaps with them commented out in the MANIFEST - thus encouraging others to at least know about the new feature.

-- SvenDowideit - 09 Jan 2012

Commenting is disabled while not logged in