Item8227: Groups in webs other than Main do not work

Have a fresh install of Foswiki 1.0.6 where I am unable to use permissions like Somewebotherthanmain.SomeGroup only Main.SomeGroup works. Same issue in an otherwise great 1.0.5 install.

-- DrewStevenson - 03 Aug 2009

this is not a bug. (though we should discuss if the tradeoff is worth it)

At this point, the WikiGroups topic (and the underlying eachGroup code) only searches the %USERSWEB% for topics ending in Group. Changing this so that we search all million webs for topics whenever we enumerate which groups a user is in has to date been considered a bad idea

this has been the case since the initial creation of the groups system - not only for performance reasons, but to stop someone form creating a group called AdminGroup, and setting the permissions to that (thus misleading users, or even worse)

(I'm not entirely positive, but iirc groups are identified by the topic name, not including the web)

-- SvenDowideit - 04 Aug 2009

Worked without modification in TWiki 3.x - 4.1.2. I'll have to dig but I think it was actually in the default docs at one point. In any event I've got 800 webs with multiple user groups (in webs other than Main) that say this works in 4.1.2.

-- DrewStevenson - 04 Aug 2009

The "User code" was (completely) rewritten at that time.

-- OliverKrueger - 04 Aug 2009

while it was re-written in 4.0.5 and 4.1, the Cairo (3.0) Access Control docco says a vague

Groups are defined by group topics created in the %MAINWEB% web, like the %MAINWEB%.TWikiAdminGroup. To create a new group

and the ACL code (in TWiki::Access code clearly agrees with you

sub getListOfGroups
{
    my $text = &TWiki::Search::searchWeb(
         "inline"        => "1",
         "search"        => "Set GROUP =",
         "web"           => "all",
         "topic"         => "*Group",
         "type"          => "regex",
         "nosummary"     => "on",
         "nosearch"      => "on",
         "noheader"      => "on",
         "nototal"       => "on",
         "noempty"       => "on",
    "format"    => "\$web.\$topic",
     );

    my ( @list ) =  split ( /\n/, $text );   
    return @list;
}

what scares me is that you're the only person ever to report that we broke something :/

-- SvenDowideit - 04 Aug 2009

on the other hand 4.0.5 code is restricted to USERSWEB

(http://svn.twiki.org/svn/twiki/tags/TWikiRelease04x00x05/lib/TWiki/Users/TWikiUserMapping.pm)

sub getListOfGroups {
    my $this = shift;
    ASSERT($this->isa( 'TWiki::Users::TWikiUserMapping')) if DEBUG;

    my @list;
    my $users = $this->{session}->{users};

    $this->{session}->{search}->searchWeb
      (
       _callback     => \&_collateGroups,
       _cbdata       =>  { list => \@list, users => $users },
       inline        => 1,
       search        => "Set GROUP =",
       web           => $TWiki::cfg{UsersWebName},
       topic         => "*Group",
       type          => 'regex',
       nosummary     => 'on',
       nosearch      => 'on',
       noheader      => 'on',
       nototal       => 'on',
       noempty       => 'on',
       format        => '$web.$topic',
       separator     => '',
      );

    return @list;

so its probly worthwhile making sure that it works like you thing it does in 4.1.2

-- SvenDowideit - 04 Aug 2009

Frankly, there's no reason to search for groups on every click. This should be cached in a db and updated incrementally.

-- MichaelDaum - 04 Aug 2009

to be productive, Drew, I think the best way for you to proceed is to write a trivial user mapper that over-rides the TopicUserMapping and only changes the one single getListOfWebs call to SEARCH all webs..

considering most larger scale sites avoid topic based groups altogether by using LDAP, HTTPDUsersContrib, or some other DB backed mapper, cache and incremental update feels like bike shed work

-- SvenDowideit - 04 Aug 2009

Unfortunately there are a lot of undocumented "features" that may have been exploited by various people, who have been forced to "try it and see if it works". As we tighten up the code and the specs, some of these are bound to break.

Fortunately as Sven indicates, there's a simple way to work around this particular misfeature. I agree that it's not in our best interests to foster the bug in the standard release.

-- CrawfordCurrie - 04 Aug 2009

I will also say that "fixing" this so groups work outside Main (I did not know you could do that and assumed you could not) at the expense of performance is a bad choice.

Natually in a future implementation where users and groups are in a database built when you register users and save topics with a GROUP definision can make this work better. But I would say that it is a bug in 4.1 that was fixed in 4.2 and I would not re-introduce this bug.

-- KennethLavrsen - 05 Aug 2009

sub _getListOfGroups {
    my $this = shift;
    ASSERT( ref($this) eq 'Foswiki::Users::TopicUserMapping' ) if DEBUG;

    unless ( $this->{groupsList} ) {
        my $users = $this->{session}->{users};
        $this->{groupsList} = [];

        $this->{session}->search->searchWeb(
            _callback => \&_collateGroups,
            _cbdata   => {
                list  => $this->{groupsList},
                users => $users
            },
            inline    => 1,
            search    => "Set GROUP =",
            #drew changed this
            #web       => "",
            topic     => "*Group",
            type      => 'regex',
            nosummary => 'on',
            nosearch  => 'on',
            noheader  => 'on',
            nototal   => 'on',
            noempty   => 'on',
            #drew changed this
            format    => '$web.$topic',
            separator => '',
        );
    }
    return $this->{groupsList};
}

This plus a few tweaks to eachGroupMember to split the web back out and an adjustment to config to specify a web for $Foswiki::cfg{SuperAdminGroup} fixed it for us. I'm not entirely sure what the safe way is to implement this as subclass of Foswiki::Users::TopicUserMapping is.

-- DrewStevenson - 05 Aug 2009

What is the performance impact on this? Let is say a Yahoo type installation with several hundred webs and 1000s of topics in each web. How long extra will it take to check all webs for a group definition instead of just the Main web?

I fear the performance impact on large installations as Sven also pointed out. I am sure a small Foswiki with 5 or 10 webs is no problem. But some installations have many hundred webs and huge traffic constantly. I fear looking in all webs each time you have to look up a group will be noticeable.

-- KennethLavrsen - 05 Aug 2009

We have a little over 1850 webs. Not sure how many topics. No noticeable change to performance with that code in but it's also not completely working at the moment. Was just coming in to update with a note that my change now seems to ignore Main (you win some, you lose some). In any event did Yahoo not run TWiki before 4.2 (when this worked)?

-- DrewStevenson - 05 Aug 2009

And as you used it you seemed to always use a fully qualified name which means no need to look everywhere but only only by more simple rule

• If fully qualified name (web prefix given) look there to see of user is part of group.
• Otherwise look only in users web

A lookup of GROUPS should still not know any other place to look than users web. (Main by default)

That should not cause any performance hit then.

We still need to also consider any security implications

-- KennethLavrsen - 05 Aug 2009

Lavr: absolutely. I am willing to subclass TopicUserMapping or patch it if need be. We seem to have it working this AM without a noticeable change in performance. Still there's more testing to do before we'd use it locally and I'd expect it'd be a while before it made it into a Foswiki release if ever.

Additional changes. Had to modify eachGroupMember to properly split the web and topic out of the group and to use $Foswiki::cfg{UsersWebName} when the group is just a topic name.

-- DrewStevenson - 06 Aug 2009

pushing to Drew - imo a subclass user mapping is in order - at least initially - need to see if this could be a performance issue for the large group that don't use it

-- SvenDowideit - 31 Oct 2009

We have tons of groups that are defined outside of Main. We ran into this upgrading from 4.0.5 to 4.2.4, and I expected to port the same change to Foswiki 1.0.7.

It wasn't a significant performance hit for us. (ACL is pretty slow anyway) I haven't looked at the changes I made recently, but IIRC they were similar to what Drew did.

-- TimHeilig - 12 Nov 2009

I really do think that searching a larger number of webs for group definitions on every request is not a good idea. There where strong reasons why it was removed from the old code. The current search engine is not ready to do that in any way performing well. Remember this is all grep based. No indexing whatsoever. This is only acceptable for small installs, certainly not for yahoo.

IF you still ultimately need to define groups all over the place, then groups and membership information should be cached. This really does not change that frequently. So caching this information in a well performing simple db will start to pay off very soon. Any save operation of a XXXGroup topic could easily be detected to update the cache.

The next good thing to do is not to use topics for storing lots of groups and members.

-- MichaelDaum - 12 Nov 2009

My code doesn't search all webs every time... It requires groups that aren't in Main be fully referenced ie (WebthatisnotMain.UserGroup) don't know if that works for Tim as well. My thought was to provide a separate MultiwebTopicUserMappingContrib or some such. We just got 1.0.7 up on our test server and are adding back in our mods and testing. When we ship to our users I'll check in our User Mapper as an option and you can ship it with the default Foswiki distribution or not as the community sees fit. Meanwhile I've attached our dev version for anyone who wants a chuckle.

-- DrewStevenson - 12 Nov 2009

Commenting is disabled while not logged in