Forums / Suggestions / Sharding ez publish database

Sharding ez publish database

Author Message

Remigijus Kiminas

Tuesday 02 June 2009 10:01:38 pm

Hellow,

I'm wondering is it possible to implement sharding in ez publish database model. I mean splitting main tables into smaller one, like youtube, facebook and many others does. Idea is simple instead of storing all record's in one monolitic database split records acroos smaller tables, databases.

Why this is needed ?
It would give unlimited scalability. Like currently i realy don't know how could single mysql server handle database with millions of records of content object attributes...

How can this be archieved ?
Some ideas there. Actualy in one of extension i implemented range sharding it's quite easy.
http://blog.maxindelicato.com/2008/12/scalability-strategies-primer-database-sharding.html
If it would be implemented, i thik ez publish would become just perfect :)

Any ideas ?

---------------------------------------------
Remigijus Kiminas

Christian Rößler

Wednesday 03 June 2009 12:37:02 am

Hy,

it would be a simple thing to try out partitioning.
http://dev.mysql.com/doc/refman/5.1/en/partitioning-overview.html

instead of storing millions of ezcontentobject_attributes in one physical table, with partitioning you are able to partition the table into 'virtually' multiple ones, each one holding a subset of all the data.
Nothing has to be changed on the ezpublish-side, as the manipulated/partitioned table looks like any other table, but the (ie. mysql) DBMS takes care of managing the data... pretty complex thing, try to read in. But shurely will make things a bit faster :)

Partitioning: store ezco_attrib from id a to c in part A, from id d to e in part B ... and so on.
It's like partitioning a harddisk...

----------

The second thing you could try is your mentioned sharding. Sharding has to be implemented in the model part of mvc. So ezpublish needs to me modified. This is a more complex part and nearly not possible, as it breaks a lot of code/logic...

----------

A third solution would be to use clustering-feature or master-slave feature.
Master-slave feature is simple to activate as it seems to be active-code in ezpublish. one db-server is used for read-operations, the other one for write-operations which get replicated on the 'read-only'-server.

But you seemed to be interested in sharding - so solution A is an option (partititoning) and the sharding feature itself is nice but very complicated to implement on such a complex system as ezpublish. Also remember that ezP already exists. Sharding is more easy to implement when beginning a new project. Thus you don't have to take care for any upgrades/downgrade issues...

One thing that came in my mind right now: memcache. that is such a thing that would significantly improve performance, but also needs alteration of ezPublish-models (persistent db layer)...

just my 2 cents.

if you intend to write such modifications, let me know. I'm interested in it (not needing it, but extremely interested how you'll solve it)

Christian

Hannover, Germany
eZ-Certified http://auth.ez.no/certification/verify/395613

Gaetano Giunta

Wednesday 03 June 2009 12:48:47 am

Maybe not as cheap as mysql to install, configure or maintain, but Oracle has had table-partitioning and server-clustering (rac) for ages.
They also claim that their handling of blobs is excellent, which should make it a good platform for eZ Publish "cluster mode".
It might be worth a try, if you're going to have a huge eZ Publish installation and money is not a problem.

I thing in general it is a good idea to let the db do the scaling instead of pushing more complexity into the web layer...

Principal Consultant International Business
Member of the Community Project Board

eZ debug

Timing: Jan 15 2025 03:08:49
Script start
Timing: Jan 15 2025 03:08:49
Module start 'content'
Timing: Jan 15 2025 03:08:50
Module end 'content'
Timing: Jan 15 2025 03:08:51
Script end

Main resources:

Total runtime1.1052 sec
Peak memory usage6,144.0000 KB
Database Queries189

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0087 687.9063214.7969
Module start 'content' 0.00870.9795 902.70311,854.4844
Module end 'content' 0.98820.1170 2,757.1875636.9844
Script end 1.1052  3,394.1719 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00700.6376210.0003
Check MTime0.00150.1376210.0001
Mysql Total
Database connection0.00320.287710.0032
Mysqli_queries1.011891.54641890.0054
Looping result0.00170.15541860.0000
Template Total1.063696.220.5318
Template load0.00390.350320.0019
Template processing1.059895.885620.5299
Template load and register function0.00150.132410.0015
states
state_id_array0.00210.193210.0021
state_identifier_array0.00470.421320.0023
Override
Cache load0.00350.3157360.0001
Sytem overhead
Fetch class attribute can translate value0.00140.122740.0003
Fetch class attribute name0.00220.203140.0006
XML
Image XML parsing0.00090.083040.0002
class_abstraction
Instantiating content class attribute0.00000.001040.0000
General
dbfile0.00080.0752210.0000
String conversion0.00000.000630.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
3content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
9content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
7content/datatype/view/ezxmltags/line.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/line.tplEdit templateOverride template
1content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 22
 Number of unique templates used: 6

Time used to render debug report: 0.0002 secs