Quick check in the MySQL database showed that there is no really big tables which could cause some delays if not indexed properly or getting locked. So, next step would be naturally looking into the PHP code itself trying to identify bottlenecks.

First of all – copy everything to test site – I don’t really want to shut the site down or display some debug info to the visitors. My test site is actually a linux box which is pretty old one, so it probably will run slower comparing to production, but hopefully still allow me to find which areas of code are causing that.

Next step – tools. To profile PHP scripts I’ll use APD (http://pecl.php.net/package/apd). Best case scenario – you type
pecl install apd
in the shell to install package from PECL.
You may also need to add it to PHP configuration, I just created extra config file
/etc/php.d/apd.ini
with content
[apd]
zend_extension="/usr/lib/php/modules/apd.so"
apd.dumpdir = /var/apd
apd.statement_tracing = 0
(make sure directory specified in dumpdir is created and writable for PHP)

Now we need to add
apd_set_pprof_trace();
at the beginning of the index.php script and open the page we want to profile, front page would be a good place to start.

In the dump directory a file created after each page opening, something like
/var/apd/pprof.08711.0

Let’s see what consumes resources now, by running
/usr/bin/pprofp -R /var/apd/pprof.08711.0

Oh, that hurts:

Total Elapsed Time = 23.58
Total System Time = 0.65
Total User Time = 7.18

Log in to Joomla admin, go to Extensions->Module Manager, delete all modules I do not really use, but which came pre-setup with Joomla.

Total Elapsed Time = 19.51
Total System Time = 0.24
Total User Time = 2.48

Ok, I am starting to see results already

Now let’s pay attention to the actual calls and time spent by script in functions.

14 seconds spent in JDocumentRendererModule->getFeed…
Hm… I do have a feed from my blog, now I wonder what the cache settings for that block is. So, in admin Module Manager -> mod_feed, advanced settings, cache : use global, time 60 minutes (I think I am ok with blog posts delayed for 1 hour, but it’s your call, just make sure it won’t come on every page load). That slashed a lot from the page load – over 15 sec of wasted time (note – check why blog feed is so slow, there should be some reasons!!!)

JDocumentHTML->_parseTemplate – I reviewed template and got rid of calls for modules which are not required or not used (like user1, user2, user3 areas which came with template and similar) – that saved 1-2 seconds per call.

At this point I have

Total Elapsed Time = 2.82
Total System Time = 0.21
Total User Time = 2.48

Which is way better, but still makes me wonder if it can run faster.

There are other options to play with
/usr/bin/pprofp -u /var/apd/pprof.09573.15
gives an idea what subroutines consumed most user time.

Last but not least: for the profiling purposes I turned off cache (Site->admin->global configuration). For life site it should be turned on with reasonable time to keep cached copy, that shaves off another 50% of execution time on subsequent calls of the same page.

Total Elapsed Time = 1.78
Total System Time = 0.17
Total User Time = 1.55

For now it’s as good as it gets on my old server, next thing to do would be profiling the script on the production server to see what it looks like there.

Another tip: if you profile a live site, restrict profiler calls to your own IP:
if($_SERVER['REMOTE_ADDR'] == "xxx.xxx.xxx.xxx") {
apd_set_pprof_trace();
}
otherwise you may get buried in a number of trace logs.

Google Webmaster Tools

That’s a valuable data on what Google “knows” about your web site, definetely worth checking and should be a starting point of the search engines optimization.

It could be some other application, could be your own code. Doesn’t matter – just it happens, one database size grows, performance of the application is degrading.

First step to troubleshoot – enable slow queries log in the MySQL config file (/etc/my.cnf is popular location). Open file in editor and add lines

log_slow_queries        = /var/log/mysql-slow.log
long_query_time = 10

(path for the log file is totally up to you, I use MySQL 5.0*, so in other versions syntax can be different)

Those lines tell mysql to log all queries lasted 10 seconds or more. Restart MySQL now and wait.

After a while run  mysqldumpslow as root. If there are any slow queries logged, you will see summary, here is mine:

Reading mysql slow query log from /var/log/mysql/mysql-slow.log

Count: 36  Time=549.81s (19793s)  Lock=0.00s (0s)  Rows=0.0 (0)
SELECT Bug.ixBug AS ix, Bug.ixBugEventLatest AS ixChild,
       Area.nTypeAs nAreaType
FROM (Bug INNER JOIN Area ON Bug.ixArea = Area.ixArea)
WHERE ixBugEventLatest  <= N
       AND ixBugEventLatest >= N AND Bug.ixBug IN
          (SELECT ix FROM IndexDelta WHERE sType = 'S'
           AND fDeleted = N)
       AND -N =  -N
ORDER BY ixBugEventLatest  DESC  
LIMIT N

Query listed above appears to take around 10 minutes to execute. Now check the tables type in the MySQL
mysql> show table status;
In our scenario I found that all tables are MyISAM. That means, that while query is executed (10 minutes) tables used in it will be locked for writing. So, if someone tries to update bug, they have to wait up to 10 minutes.

Check what MySQL engines you have:

mysql> show engines;

+---------+----------+---------------------------
| Engine  | Support  | Comment                  
+---------+----------+---------------------------
| MyISAM  | DEFAULT  | Default engine as of MySQL
| MEMORY  | YES      | Hash based, stored in memory
| InnoDB  | YES      | Supports transactions, row-
...

If InnoDB is supported, then all what has to be done to speed up application is switching to InnoDB in this situation. This will not make query above run any faster, but, that will not lock the whole table for the time query is running. There are some benefits of MyISAM table type (well, it’s faster!), so you may don’t want to convert all tables to InnoDB at this point. But large tables with concurrent updates and reads will benefit.

If InnoDB is not supported, it can be turned on in MySQL configuration file, just comment out skip-innodb line.

In case of FogBugz I found the following tables to be worst offenders:
Bug, BugEvent, BugRelation, SorterToken, TokenAssociation.

To convert those, use:
mysql> alter table Bug engine=InnoDB;
After conversion performance of the application increased immediately, as there are no more locks on popular tables.

There are additional benefits of InnoDB engine, such as foreign keys. Read more on InnoDB:

http://dev.mysql.com/doc/refman/5.0/en/innodb.html