eZ is DOSing server, need help

Wednesday 12 January 2005 12:48:56 pm

Hello!

Yesterday I've setup my site running eZ installations, suddently eZ started to DOSing server, CPU usage went 100%, Apache "Max Servers" limit was also acquired.

I've enabled all caches / template compiling etc:

TemplateCache=enabled
TemplateCompile=enabled
TemplateCompression=enabled
Debug=disabled
ViewCaching=enabled

.htaccess:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\.(gif|jpe?g|png|css|js|htm?|pdf|swf)$
RewriteRule ^(.*)$ /index.php/$1 [L,QSA]

I've tested performance with ab utility, with -c 10 -n 10:
Percentage of the requests served within a certain time (ms)
50% 2113
66% 2445
75% 3211
80% 3865
90% 3865
95% 3865
98% 3865
99% 3865
100% 3865 (longest request)

Percentage of the requests served within a certain time (ms)
50% 676
66% 938
75% 938
80% 938
90% 1315
95% 1315
98% 1315
99% 1315
100% 1315 (longest request)

with -c 25 -n 25:

Percentage of the requests served within a certain time (ms)
50% 4952
66% 6390
75% 6697
80% 7719
90% 27715
95% 27811
98% 27836
99% 27836
100% 27836 (last request)

Percentage of the requests served within a certain time (ms)
50% 12233
66% 15263
75% 19928
80% 22600
90% 53653
95% 56540
98% 68262
99% 68262
100% 68262 (last request)

After this server is almost DOSed :/

System specs:
Dual 2.4ghz Xeon w/1GB of memory, Redhat ES v3.

What should I do? Any hints?

Wednesday 12 January 2005 2:16:28 pm

Nope, server still has 300MB of free memory, cache is cleared, PHP accelerator is enabled.

Wednesday 12 January 2005 3:45:35 pm

Debugging is always hard when you don't have access to a system. So how about some more information?

php version ?
apache version ?
ez publish version ?

The systems seems to have worked in testing, so what has changed since going live?

> CPU usage went 100%
So what process is taking up all the CPU?

> Apache "Max Servers" limit was also acquired.

This could be an apache issue. It could be caused by the system getting many hits and grinding to a halt. The hits continue coming but the apache processes are unable to finish serving the requests due to high server load.

Can you access the site at all? How long does it take to become unusable?

Cheers
Bruce

My Blog: http://www.stuffandcontent.com/
Follow me on twitter: http://twitter.com/brucemorrison
Consolidated eZ Publish Feed : http://friendfeed.com/rooms/ez-publish

Wednesday 12 January 2005 4:33:16 pm

You told us you moved the server. Does that mean you actually had a non eZ website running there before?

If this is the case try to redirect all requests that belong to a non eZ url to an error page ( e.g everything *.html )

In those cases eZ will give a module not found error those error have a high impact on the system they ea a lot of cpu time.

Besides that you may have reached a critcal limit in request per server. It might be just the case that his maschine can't handle it with eZ running on it. I have never been on a SINGLE server that can serve more then 15 eZ pages per second.

In this case block all networking traffic and do a benchmark with no traffic and compare it with your usual traffic.

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Wednesday 12 January 2005 4:41:04 pm

Is debugging disabled? It should... You want as less IO as possible

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Thursday 13 January 2005 6:17:46 am

have you checked the mysql? W
hen mysql has problems it might be that mysql requests are set to a hold state... 
during that time apache just hold it till it timeouts...

This maybe a reason. We had problem with server DOSing by eZ publish when mysql crashes. CPU usage went to 100%. Accessing main page takes very very long time ...

Personal website -> http://serwatka.net
Blog (about eZ Publish) -> http://serwatka.net/blog

Thursday 13 January 2005 8:13:22 am

I think you should let a professional look at your setup and test it. I like said before it is very hard to resolve from the outside. Root access is required too.

Please check also the disk IO, is this also running at 100% or is it just the CPU?

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Thursday 13 January 2005 9:02:29 am

Made some more test, seems that this is caching issue with disk access, with following settings:

[ContentSettings]
ViewCaching=enabled
PreViewCache=disabled

[TemplateSettings]
TemplateCache=disabled
TemplateCompile=enabled
TemplateCompression=disabled
Debug=disabled

I can run ab with -c 50, of course reply times are not so good (longest reply 11 sec) but it isn't DOSing the server... Do you have any ideas on improving eZ caching system? Any info about internal aspects of TemplateCache?

Friday 14 January 2005 12:25:26 am

There is not much more you can do to improve the caching.

An option is to make static copy of the whole site. So that you don't hit mod_php4.

What disk is actually in the server? I think providing a better and faster disk will help you a lot.

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Friday 14 January 2005 9:12:50 am

Server is running only IDE disks, even not in RAID... But managed to make some static cache system... Mayby we will improve that and send it to contributions :) Thanks for help!