Profiling slow perl code with NYTProf

Problem

You experience very bad performance on your wiki. Some saves or edits are taking a very long time (more than 10s is long).

Context

Sometimes your wiki is running very slow, and you have no idea why. NYTProf helps you pin down the culprit by showing you where the Perl code spends most of its time.

Solution

Perl has some really great tools, and one of them is CPAN:Devel::NYTProf. Devel::NYTProf uses the debugger functionality of Perl to place hooks and keep track of every function which gets called, and how long it lasts.

First, you have to install CPAN:Devel::NYTProf. It is usually packaged for your distribution, otherwise use: cpan Devel::NYTProf

• On Debian/Ubuntu, simply: sudo apt-get install libdevel-nytprof-perl
• On FreeBSD, simply: portinstall p5-Devel-NYTProf

Once installed, you need to tell the script you want to debug that it ought to use it. To do so, change the shebang line (the first line of the script) to look like this:

#!/usr/bin/perl -wTd:NYTProf

After that, you have to ensure the webserver can write its output file, so ensure the bin/ directory is owned by the webserver user. Only do this during the test; it's not very secure to have this directory writable.

You may now access the wiki and run the page which you want to debug. This should create a file called nytprof.out in the bin directory. NYTProf doesn't like to overwrite this file, so to run the script again, you have to remove the file. You may save several files for further analysis.

Finally, you ask NYTProf to generate HTML or CSV output based on the information it gathered:

• For HTML: nytprofhtml
• For CSV: nytprofcsv

In the end, you should have a nytprof folder with your data in it. If you chose the HTML version, point your browser to the index.html. The output should be pretty much self-explanatory.

Known Uses

Find bottleneck in rendering.

Known Limitations

Only one debug at a time. You can use the addpid option to add the PID to the nytprof.out file. To use this option, you have to define an environment variable which has to be passed over to your CGI. You should be able to use SetEnv NYTPROF addpid=1:file=/tmp/foswiki-nytprof.out

If you're using mod_perl, there is just one line to add in your httpd.conf:

PerlModule Devel::NYTProf::Apache

By default you'll get a /tmp/nytprof.$$.out file for the parent process and a /tmp/nytprof.$parent.out.$$ file for each worker process.

See Also

In case the script is taking too long to process, and Apache kills it before you're done, you may run it from the command line (assuming you're not uploading something), using something along the lines of (from the bin/ directory):

sudo -u www-data perl -wTd:NYTProf edit -topic Sandbox.TestTopic -username YourUsername

replace www-data, edit, Sandbox.TestTopic and YourUsername with the appropriate values.

-- OlivierRaginel - 13 Nov 2010