eZ Publish 4 performance & Varnish

Author Message

Norman Leutner

Moderated by: Nicolas Pastorino

Monday 22 October 2007 2:02:49 pm

Hi,

we just tested the new eZ Publish 4.0.0alpha2 performance the following environment on a blank installed eZ instance:

System:

Single Server, Debian 4, 2 x Intel Xeon 5130 Dual Core 2,00 GHz, 4096 MB RAM, 2 x 146 GB SAS HDD RAID1

Webserver:
Apache/2.2.3 (Debian) PHP/5.2.4-0.dotdeb.1 with Suhosin-Patch mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8

MySQL:
5.0.45-0.dotdeb.1 

APC
Version: 3.0.15 

Except the APC accelerator, there are no performance optimized settings or proxys.

ApacheBench test result:
- 1000 request
- 5 multiple requests

# ab -n 1000 -c 5 http://mobello2.all2e-dev.de/index.php

This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking mobello2.all2e-dev.de (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests


Server Software:        Apache/2.2.3
Server Hostname:        mobello2.all2e-dev.de
Server Port:            80

Document Path:          /index.php
Document Length:        7287 bytes

Concurrency Level:      5
Time taken for tests:   12.482263 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      7852000 bytes
HTML transferred:       7287000 bytes
Requests per second:    80.11 [#/sec] (mean)
Time per request:       62.411 [ms] (mean)
Time per request:       12.482 [ms] (mean, across all concurrent requests)
Transfer rate:          614.23 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.0      0      29
Processing:    44   61  14.2     58     123
Waiting:       42   59  14.2     56     121
Total:         44   61  14.5     59     123

Percentage of the requests served within a certain time (ms)
  50%     59
  66%     66
  75%     71
  80%     74
  90%     82
  95%     89
  98%     99
  99%    107
 100%    123 (longest request)

As you can see the environment is able to handle about 80 requests per second which is really impressive if you compare them with between 20 and 40 on eZ 3.x.

Within the next days we will migrate a larger site on that environment and do the same tests.

Mit freundlichen Grüßen
Best regards

Norman Leutner

____________________________________________________________
eZ Publish Platinum Partner - http://www.all2e.com
http://ez.no/partners/worldwide_partners/all2e_gmbh

Nabil Alimi

Monday 22 October 2007 2:11:36 pm

Thanks for sharing ! This is really interessting. That proves that migration is worth it.

My blog : http://www.starnab.com/ezpublish / http://www.starnab.com/ / http://www.assiki-consulting.com
eZ Publish Freelance developper. Feel free to contact me +33 674 367 057
nabil at assiki d0t fr

Norman Leutner

Monday 22 October 2007 2:18:36 pm

Yes, it really is...

We'll also do some tests with the Varnish reverse proxy on that installation.

Mit freundlichen Grüßen
Best regards

Norman Leutner

____________________________________________________________
eZ Publish Platinum Partner - http://www.all2e.com
http://ez.no/partners/worldwide_partners/all2e_gmbh

André R.

Tuesday 23 October 2007 2:15:43 am

Nice show down! This is similar to with what we are experiencing internally also (2-4x improvement).

Norman: Don't forget to post some info on what version of varnish you use and your configuration ;)

eZ Online Editor 5: http://projects.ez.no/ezoe || eZJSCore (Ajax): http://projects.ez.no/ezjscore || eZ Publish EE http://ez.no/eZPublish/eZ-Publish-Enterprise-Subscription
@: http://twitter.com/andrerom

Andreas Kaiser

Tuesday 23 October 2007 3:33:27 am

Great news!

Hope the migration to ezp4 will not be a headache ;)

Grüsse,
Andreas

eZ Partner in Madrid (Spain)
Web: http://www.atela.net/

Sébastien Antoniotti

Tuesday 23 October 2007 4:42:27 am

Cool !

Thanks for sharing, I wait impatiently a beta/stable version to upgrade my 3.8.6 !!

eZ Publish Freelance
web : http://www.webaxis.fr

André R.

Tuesday 23 October 2007 5:28:49 am

Andreas Kaiser: well that depends on that amount of php code in custom modules and datatypes you have:)

A unofficial list of php api changes:
Constants have been changed to class constant to avoid use of define().
Static methods have been marked as static.
The eZXml library is not used any more, use php's xml types.
Objects is not returned by reference*.

*example:
$ini =& eZIni::instance();
should be:
$ini = eZIni::instance();

and
function &fnReturnObject()
should be
function fnReturnObject()

eZ Online Editor 5: http://projects.ez.no/ezoe || eZJSCore (Ajax): http://projects.ez.no/ezjscore || eZ Publish EE http://ez.no/eZPublish/eZ-Publish-Enterprise-Subscription
@: http://twitter.com/andrerom

Norman Leutner

Tuesday 23 October 2007 5:40:40 am

@André
I will test the varnish-1.1.1 release on that server.
We will also compare the varnish results with the apache2 reverse proxy.

Mit freundlichen Grüßen
Best regards

Norman Leutner

____________________________________________________________
eZ Publish Platinum Partner - http://www.all2e.com
http://ez.no/partners/worldwide_partners/all2e_gmbh

Kristof Coomans

Tuesday 23 October 2007 8:15:42 am

Constants have been changed to class constant to avoid use of define().

Small remark: avoiding define() was not really our main concern for changing to class constants. The PHP5 autoload mechanism was.

independent eZ Publish developer and service provider | http://blog.coomanskristof.be | http://ezpedia.org

Norman Leutner

Wednesday 24 October 2007 10:49:24 am

Here the results with the Varnish 1.1.1 reverse proxy:

ds87-230-58-81:~# ab -n 1000 -c 5 http://mobello2-varnish.all2e-dev.de/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking mobello2-varnish.all2e-dev.de (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests


Server Software:        Apache
Server Hostname:        mobello2-varnish.all2e-dev.de
Server Port:            80

Document Path:          /
Document Length:        7287 bytes

Concurrency Level:      5
Time taken for tests:   0.86800 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      7753746 bytes
HTML transferred:       7294287 bytes
Requests per second:    11520.74 [#/sec] (mean)
Time per request:       0.434 [ms] (mean)
Time per request:       0.087 [ms] (mean, across all concurrent requests)
Transfer rate:          87235.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0    0   0.0      0       0
Waiting:        0    0   0.0      0       0
Total:          0    0   0.0      0       0

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      0 (longest request)

As you can see this is more than the network interface can handle:
Transfer rate: 87235.02 [Kbytes/sec] received
With a load on server side of below 0.3

Mit freundlichen Grüßen
Best regards

Norman Leutner

____________________________________________________________
eZ Publish Platinum Partner - http://www.all2e.com
http://ez.no/partners/worldwide_partners/all2e_gmbh

Michael Lee

Monday 17 December 2007 6:36:11 am

Hi Norman,

We noticed the VCL sample in VCL man page does work with eZ Publish. And we found an article about this issue at
http://www.ipersec.com/index.php/2007/11/15/varnish-simple-and-fast-http-acceleration.
The author provided a sample vcl configuration file which can be used for eZ Publish. However, would you like to share you vcl file? Or could you please review the article and see if it missed anything important?

Many thanks in advance,

Michael

Michael Lee | Managing Director | ZerusTech Ltd | www.zerustech.com

Skype: zerustech

Michael Lee

Monday 17 December 2007 6:42:15 am

Hi Norman,

We noticed the VCL sample in VCL man page does work with eZ Publish. And we found an article about this issue at
http://www.ipersec.com/index.php/2007/11/15/varnish-simple-and-fast-http-acceleration.
The author provided a sample vcl configuration file as the following, which can be used for eZ Publish.


backend default {
     set backend.host = "127.0.0.1"; // use your own backend ip address
     set backend.port = “8080″; // use your own backend port
}

sub vcl_recv {

      if (req.request != "GET" && req.request != "HEAD") {
          pipe;
      }

      if (req.http.Expect) {
          pipe;
      }

      if (req.http.Authenticate) {
          pass;
      }

      if (req.http.Cache-Control ~ "no-cache") {
          pass;
      }

      lookup;

}

sub vcl_pipe {
      pipe;
}

sub vcl_pass {
      pass;
}

sub vcl_hash {
      set req.hash += req.url;
      set req.hash += req.http.host;
      hash;
}

sub vcl_hit {

      if (!obj.cacheable) {
          pass;
      }

     deliver;
}

sub vcl_miss {
      fetch;
}

sub vcl_fetch {

      if (!obj.valid) {
          error;
      }

      if (!obj.cacheable) {
         pass;
      }

      if (obj.http.Set-Cookie) {
         pass;
      }
     
      if (obj.http.Pragma ~ "no-cache" || 
         obj.http.Cache-Control ~ "no-cache" || 
         obj.http.Cache-Control ~ "private") {
              pass;
       }

       insert;
}

sub vcl_deliver {
      deliver;
}

sub vcl_timeout {
      discard;
}

sub vcl_discard {
      discard;
}

However, would you like to share you vcl file? Or could you please review the vcl file and see if it missed anything important?

Many thanks in advance,

Michael

Michael Lee | Managing Director | ZerusTech Ltd | www.zerustech.com

Skype: zerustech

Norman Leutner

Monday 17 December 2007 8:11:19 am

Hi Michael,

in fact Varnish won't work with eZ if cookies are present, so you'll need to tell varnish to ignore them.

backend all2e {
  set backend.host = "cache.all2e.com";
  set backend.port = "80";
} 

sub vcl_recv {
  if (req.http.host ~ "^www.all2e.com$") {
    set req.backend = all2e;
  }
  elseif (req.http.host ~ "^all2e.com$") {
    set req.backend = all2e;
  }    
 else {
    error 404 "Unknown  virtual host";
  }
  
  # force lookup even when cookies are present
  if (req.request == "GET" && req.http.cookie) {
    lookup;
  }
  
}

Keep in mind that this will only work for static sites, where users can't login.

Getting Varnish working with the current session handling within eZ isn't possible because eZ publish delivers cookies even when the user is a anonymous one.

We are currently planning to <i>hack</i> the eZ Session handling together with xrow in january. If you are also interested we should consider about 'Open funding' this.

I would not recommend to use Varnish without a new Session handling.

Mit freundlichen Grüßen
Best regards

Norman Leutner

____________________________________________________________
eZ Publish Platinum Partner - http://www.all2e.com
http://ez.no/partners/worldwide_partners/all2e_gmbh

Michael Lee

Monday 17 December 2007 9:53:46 pm

Hi Norman,

Thanks a lot for your quick reply. I agree with you that varnish does not cache documents when cookies are present by default, so we forced varnish to insert and lookup even when cookies are present as well.

Besides, I think we found a way to cache dynamic documents as the followings.

Step 1.
We forced varnish to "lookup" even when cookies are present just like what you did with minor differences in code base.


//just for testing purpose, we didn't specify multiple backends
sub vcl_recv {

    //general setttings
    if (req.request != "GET" && req.request != "HEAD") {
        pipe;
    }

    if (req.http.Expect) {
        pipe;
    }

    if (req.http.Authenticate) {
        pass;
    }

    //this part ignores no-cache documents
    if (req.http.Cache-Control ~ "no-cache") {

        pass;

    }

    //other documents will be looked up from varnish even when cookies are present
    lookup;

}

Step 2. Insert documents into cache even when cookies are present

sub vcl_fetch {

    if (!obj.valid) {

        error;

    }

    if (!obj.cacheable) {

        pass;

    }

    //set-cookie response should be passed
    //for example, eZ Publish session handler may 
    //set session cookie the first time when you visit the site
    if(obj.http.Set-Cookie){

       pass;

    }

    //no-cache documents should be ignored and passed
    if(obj.http.Pragma ~ "no-cache" ||
       obj.http.Cache-Control ~ "no-cache" ||
       obj.http.Cache-Control ~ "private"){

       pass;

    }

    //other documents should be inserted into 
    //varnish cache, even when cookies are present
    insert;

}

Step 3. The most important step: change the vcl_hash subroutine.

sub vcl_hash {

    //hash the object with url+host

    set req.hash += req.url;

    set req.hash += req.http.host;

    /*
     uncomment the following code snippet if you also want to hash the 
     object with cookie. 
     If you don't uncomment this line, different users 
     may share same object in cache.
     Otherwise, varnish will insert/fetch a cache object for
     each user
    */
    //set req.hash += req.http.cookie;

    hash;

}

Step 4.
configure "customheader" in site.ini.append.php. Just for testing purpose, we made the "/" cacheable like this:

[HTTPHeaderSettings]
CustomHeader=enabled
Cache-Control[]
Cache-Control[/]=
Pragma[]
Pragma[/]=
Expires[]
Expires[/]=1200

Step 5.
Optional - if you need login function, you can implement a login panel with Ajax. So each time the document is loaded, an ajax function will be invoked to either display login panel or welcome message. You can do this with xajax extension.

We have implemented the solution above. And in order to test the performance improvement, we disabled all template cache and view cache on eZ Publish.

Here is the test cases:

Test case 1:

1. telnet to varnish management port

2. url.purge .*

3. access http://<varnish server>:<varnish port>/ from machine A

4. It takes around 20 seconds to load the page

5. re-load current page and this time it takes less than 1 second

6. try to access http://<varnish server>:<varnish port>/ from machine B

7. access http://<varnish server>:<varnish port>/ from machine B

8. It takes less than 1 second to load the page. ( which means machine B is using the same cache with machine A)

Test Case 2:

1. Change the vcl_hash() subroutine and set hash to "url+host+cookie"

2. Kill and restart varnish

3. Repeat the steps in Test Case 1

This time, machine B can't re-use the cache with machine A. But on both machine, after re-loading current page, the document will be cached by varnish.

Basically, that's all we have investigated so far. Could you please review it and get back to me with your comments?

Kind regards,

Michael

Michael Lee | Managing Director | ZerusTech Ltd | www.zerustech.com

Skype: zerustech

Michael Lee

Wednesday 19 December 2007 10:27:19 pm

Hi Norman,

I forgot to mention one thing.

You must set "keepAlive" to "off" in apache, otherwise, varnish might not work correctly.

I have create a ticket at http://varnish.projects.linpro.no/ticket/191

Michael Lee | Managing Director | ZerusTech Ltd | www.zerustech.com

Skype: zerustech

Michael Lee

Thursday 20 December 2007 6:01:02 pm

Varnish 1.1.2 has been released yesterday.

http://varnish.projects.linpro.no/wiki/Releases/1.1.2

Michael Lee | Managing Director | ZerusTech Ltd | www.zerustech.com

Skype: zerustech

Norman Leutner

Tuesday 18 March 2008 4:58:54 am

I've submitted an open funding sugessstion:

http://ez.no/developer/open_funding/suggestions_for_new_functionality/http_acceleration_with_varnish

Mit freundlichen Grüßen
Best regards

Norman Leutner

____________________________________________________________
eZ Publish Platinum Partner - http://www.all2e.com
http://ez.no/partners/worldwide_partners/all2e_gmbh

Jonny Bergkvist

Monday 02 March 2009 3:06:32 am

Hello!

I have made documentation of a working setup with Varnish and eZ publish.

It's published here:
http://ez.no/developer/contribs/documentation/varnish_and_ez_publish_setup_guide

Hope it can be useful!

Pablo Pernot

Tuesday 31 March 2009 7:45:43 am

Hi,

just installed your config. Everything works fine but the backend login. I can't login anymore.
Obvisouly I lose my session/cookie information. Do you think it could come from varnish ? If yes any idea ?

thanks

Pablo Pernot
http://www.smartview.fr
http://www.areyouagile.com

André R.

Tuesday 31 March 2009 7:59:00 am

Do you use the ezvlogin extension and / or eZ Publish 4.1 so the is_logged_in cookie gets set?

eZ Online Editor 5: http://projects.ez.no/ezoe || eZJSCore (Ajax): http://projects.ez.no/ezjscore || eZ Publish EE http://ez.no/eZPublish/eZ-Publish-Enterprise-Subscription
@: http://twitter.com/andrerom

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.