Item10847: FormQueryPlugin + FastCGIEngineContrib (mod_fcgid): search results multiplied by number of page requests

FastCGIEngineContrib (here: apache2, mod_fcgid) causes FormQueryPlugin to increment the search results on every page loading. This is especially an issue when the number of hits is important. Of course, tables with multiple identical rows are also not quite useful.

The "counter" can be reset by restarting or reloading the webserver.

Simple test case for FormQueryPlugin, searching for all users and outputting a table containing the first+last name

%TABLEFORMAT{
   name="STANDARD"
   sort="name"
   header="|*FirstName*|*LastName*|"
   format="|$FirstName|$LastName|"
}%

%FORMQUERY{
   name="users"
   search="UserForm.LastName~'?*'"
   web="%USERSWEB%"
}%

%MATCHCOUNT{query="users"}%

%SHOWQUERY{
   query="users"
   format="STANDARD"
}%

The built-in SEARCH with type="query" is not affected from this bug (so you could use this "feature" as an access counter ).

Setting priority to normal, since there is always the workaround of not using FGCI; otherwise, I'd consider it rather urgent.

-- PhilippLeufke - 06 Jun 2011

Nobody else using FCGI and FormQueryPlugin?

-- PhilippLeufke - 13 Jun 2011

Guessing that this Problem could as well be caused by DBCachePlugin and this way trying to get anyone's focus onto this issue...

-- PhilippLeufke - 27 Apr 2012

Another guess:

The query results variable/array doesn't get reset, as it is normally not needed, since the perl process ends after the query.

But in case of persistent-perl (FCGI), the perl process is -- yes: persistent -- and so is the query results variable. I guess the bug fix should be pretty straight-forward by either re-initializing this variable at the beginning of each query or by deleting it at the end...

-- PhilippLeufke - 27 Apr 2012

Are these duplicate entries related to the question about mailnotify/mailercontrib interaction with formquery here:

http://foswiki.org/Support/Question803

I observe similar duplicates in formquery results with my newsletter based on pages created with formquery commands. I do not, however, use fcgi.

My rather painful solution is to use the showquery to run spreadsheet calc commands that track and flag the duplicates so that they are only printed once.

My sense is that the formquery query cache is not clearing between requests when run within the same process. So reloading pages (without fcgi) works because the cache gets erased when the view process ends. Generating multiple newsletters (through a single call to mailnotify) does not work because the cache grows with each newsletter (the same queries lead to duplicates, triplicates, etc.) I assume the same thing happens with the persistent perl process.

This seems to indicate that query results are stored in memory for the duration of the process. One solution would be to have view and/or mailnotify erase that bit of memory at the end of each page. Another would be to use random names for queries, with the name fixed for the duration of a page, so that prior queries of the same name don't pollute the current page.

Out of curiosity, do you see the memory usage of the persistent perl process slowly growing because of this stored memory? Or maybe it's cached to disk and the disk cache is not being cleared between pages, but only when the calling process ends?

-- JohnRoberts - 04 May 2012

John, thanks for your remarks. Unfortunately I don't have a testing environment ready right now and I don't want to mess around with the productive system...

Could you please check, if there are similar issues in mailnotify when using DBQUERY from DBCachePlugin? I just wonder, if this is related to the DB caching or if it's only FormQueryPlugin. This is also addressing the point that you mentioned: is it in HDD cache or RAM?

Because persistent perl, caching and query searches are vital for performance (see PerformanceSupplement),I decided to follow MichaelDaum's classification (see Item11476):

"That's why I make this task Urgent, not Enhancement, as speed improvement is a must, not a nice-to-have." -- MichaelDaum - 26 Jan 2012

-- PhilippLeufke - 04 May 2012

I confirm that this bug is also present in 1.09 Codebase. Most probably caused by the fcgi plugin.

-- StefanosKouzof - 29 May 2012

I didn't read the code of FormQueryPlugin, but I hardly think the problem is in FastCGIEngineContrib. The same bug probably shows up with ModPerlEngineContrib. It seems that FormQueryPlugin uses global variables to keep its data and it's not properly cleaned after the request is gone. With plain CGI it's OK since the process dies after the request is answered, but it's a problem when a persistant backend is in place.

-- GilmarSantosJr - 29 May 2012

Happy anniversary to this task item! OK, a tiny bit too late, but who cares?

I just realized that the last release of this plugin in from 2009. Why is this plugin still available? It seems, that it's entirely orphaned?!

The idea of FormQueryPlugin seems so powerful, logical and user friendly, but as long as no maintenance is done, users should rather be warned, or they'll get stuck after investing quite some time in improving their own applications. This is what happened to us. We converted a lot apps/topics from SEARCH-based queries to FormQueryPlugin, seeking for better performance and now that we want to further improve speed by switching to FCGI, there is this fundamental bug which blocks the mission. Depressing!

OK, but now some more constructive thoughts:

How can the buffer/variable/array (I'll call it variable from now on) for the query results be initialized/purged upon each page request. Without being a Perl crack, I assume that the variables for each query are persistent in each Perl process, since this is exactly the feature of FormQueryPlugin. This way they can be recalled for further modification of a search result. Therefor, just purging all results variables when the plugin is called would be too much, as we would lose the results of the previous queries. On the other hand, the variables have to be empty on a new request of the same topic. I have two ideas which might lead to success:

• If there is a way to identify different page requests (time stamps, unique IDs) it should be easy...
• Otherwise, the plugin itself could keep a history of queries, detecting if one query had been called before and re-initialize in this case
• Or quick-n-dirty: filter each query for unique results (topics) only. I can't think of any situation where multiple hits would be welcome.

Hoping that somebody gives a damn...

CU -- PhilippLeufke - 09 Jun 2012

FormQueryPlugin is no longer maintained; please move the DBCachePlugin or use the core features.

-- Main.CrawfordCurrie - 15 Jun 2015 - 08:54

Changing to "Needs Developer". Bug appears to be confirmed, but without a maintainer for the extension, it won't get fixed.

-- GeorgeClark - 26 Sep 2016

Commenting is disabled while not logged in