|
Those regular readers will have noticed that over the last few months my site has become flaky, with frequent crashes and hanging (ie no data served to your browser). I have been working with the help of a more knowledgable friend on fixing this.
For those other Joomla users who have hangs due to a high CPU load or high I/O writes I am detailing the fixes I have undertaken so that you may benefit.
This will probably bore the average reader....
First off I started by following my own advice on what to do when Joomla uses too many resources.
Also I checked my server stats and found the CPU usage climbed at high as the graphs went, 400% usage. That would explain the crashes, interesting to note that disk Input/Output (I/O) writes at this time was also extraordinarily high and of course network traffic was nil.
Turning on debugging mode (see previous link) I found that a simple add on component "Simple Quote" was generating a hell of a lot of MySQL database queries for each page load, in fact for the front page I was generating 309 database queries - heck no wonder the server was stalling when I had a few users onboard.
Unpublishing the "Simple Quote" site module did not help. Uninstalling revealed that there was a problem, it could not uninstall. Emptying the directory manually I was then able to reinstall "Simple Quote" then uninstall it. Checking the number of database queries to bring up the front page now was 27. Whew, I thought that was going to fix that..... another 3 hour outage next day with CPU loads up to 190%.
So that fixed some of it.
Going through my installed components I noticed that "PageCache" was still configured to write to the directory on the old server before the move to my current virtual private server so in effect "PageCache" was still trying to write to a non-existant directory.
Going into the directories I fixed this by putting in the new and correct path and ensuring that the directory was writable. A quick check showed that everything was now writing as it should and "PageCache" was now caching pages as static html pages to serve out faster than building the pages each time from the database, thus speeding up hits to the same page - as occurs when someone links from a busy Internet forum to a specific page and hundreds of visitors (or more) come within minutes.
I then searched the web for other causes for a high CPU load with Joomla, but came up with dead end answers. It seems that those who solve these problems do not go on to document what they did. Silly really it is free traffic from other greatful Joomla users.
Anyway in my fruitless searches I did turn up a new component called "QuerieCache" which lines up the database queries and sends them all at once, adding this and following the very easy to follow FAQ to set it up within minutes I was getting the 27 queries for the frontpage down to 15. And other pages were down about the same 30%.
On the MySQL database side of things my friend Peter in Afghanistan adjusted some settings and increased the memory usable and a few other tweaks that seem to have sped things up. Also previously we changed the default tables from MyISAM to InnoDB on the jos_opensef_sef table and the G2_sessionMap table as these were locking before (we think) as they are accessed quite often and InnoDB doesn't lock the entire table to do a write.
i have my body armour and helmet now in a bag at my side... have to take it to and from the office
there was a big car bomb in kabul this morning.... loud. thought it was in my street, but was actually probably about 5 blocks away
about a week ago our security manager found at the front of the office a piece of hot shrapnel from a grenade....
- Peter in Afghanistan
We also added eAccelerator to the PHP package to try caching the PHP as the PHP-CGI was generating a lot of CPU load.
Additionally I went into phpAdsNew (now openAds) and in the settings -> optimise I checked the "use delayed inserts" and "use database compatability mode". Meaning that phpAdsNew now waits and writes until MySQL is free then writes it's stats as a block, rather than one line at a time. Compatability mode checks the state of the DB when it starts to write and upon exit leaves it in the same state to avoid conflicting with other processes accessing the database.
So far everything seems to be running along just fine, but at this point I have only had sixusers on the site at once. If it is running OK for the next two days I will make a post on a very responsive internet forum and pull several hundred people over to the site to check the server stats and if it can deal with it.
Hope this helps, and no matter how bad your day is at least you didn't wake up to a huge car bomb.
*UPDATE:
The high CPU loads turned out to be from the RAM being all used up and the server transferring data between the RAM and the hard disk swap file. This explains why the Input/Output loads went sky high too.
To fix this I made some adjustments in the web server software, changing the maximum number of threads and the maximum number of child processes as well as lowering the maximum time to live for the processes.
This has lowered the usage considerably, though I still cannot place why this has happened rather suddenly.
The high load times seemed to be caused by search engine bots indexing the site.
** UPDATE 2:
After all the above the site was still crashing, the CPU and disk thrashing from swap file usage were lower but it was still affecting the site, making it slow and/or unreachable.
To get further ahead on this I looked into the log files, all 100mb of straight text telling each call to the database from each IP address, etc.
So spending a few hours going over the files at around the times of the crashes I found that everytime a page loaded from the photo gallery ther was 5 calls to the database for the stars that I installed on the rating system for users to vote for the best photos.
Now try Googlebot hitting a page with photos, following the links to the gallery and 'loading' and indexing 20 photos at a time, that is 100 additional database calls in under a second. Plus then if a user is on there there are read and writes to the database to get the rating and to write the new vote, all in all it did not add a whole lot to my website and cutting it out might help stop the bots crashing my site.
So I removed the rating system.
Also I noticed that the times when I was crashing most recently there would be Googlebot, Yahoo Slurp, Yahoo media indexing spider and Baidu (Chinese) bots on my site, as a test I added some instructions on robots.xt to slow down the Yahoo bots and went to Google Webmaster Tools and told it to slow down the bots too.
I also noted that on each article I had a email link to send to friend and a .pdf downloadable version for printing available too. The Search Engine bots can now read pdf documents and probably the email linked version too. That means each page could be getting indexed 3 times, possibly generating a significant load for no returns. So I removed these options.
I went into Joomla and removed all components, modules and mambots that were not being used to remove any calls due to bad coding, and to increase security.
Looking at the stats I have significantly lowered the load values but I was still getting CPU load spikes to 150%, two or three times a day. Which was better than extended spikes at 400%+
Last thing we did was upgrade the current linux distribution to the latest stable, this also upgraded the web server software and this in itself seems to have fixed a memory leak in the server software.
Currently, for the last week the CPU load has been running at about 13% with spikes to 24%, much better. I have found that I am getting more visitors and they are staying longer (more page views) - not surprising really.
This also means that I am getting more advertising revenue from my site and received my best CTR pay day to date, beating the old record by $1, but still a good indication just 3 days after seemingly fixing the problem.
I have now set the search engine bots back to normal speeds to see if there is any impact, and I found a linux command where you can change your "swappiness" factor, putting it higher means you server will be less likely to swap RAM memory out to hard drive - this would help as the default setting is too high.
|
