Forums / Install & configuration / Purge Data from ezdfsfile table and related files

Purge Data from ezdfsfile table and related files

Author Message

Jim Thaxton

Friday 29 January 2010 2:33:38 pm

We have an implementation of eZ Publish utilizing the ezdfs file handling system. Looking at the ezdfs table, it is clear that our system is not removing data from the table nor deleting files form the NFS mount upon delete from the CMS.

Is there a script that should be run in cron to handle this or do we need to build our own mechanism to handle this functionality? Thanks!

Web Developer
Coupon Cabin
Chicago, IL

Robin Muilwijk

Saturday 30 January 2010 12:12:41 am

Hi Jim,

The only doc I was able to find is posted here http://share.ez.no/forums/install-configuration/ezdfsfilehandler#comment55719 but it does not contain anything about cronjobs.

I'll ask around in the share.ez.no team if someone knows more about this and report back.

Regards Robin

Board member, eZ Publish Community Project Board - Member of the share.ez.no team - Key values: Openness and Innovation.

LinkedIn: http://nl.linkedin.com/in/robinmuilwijk // Twitter: http://twitter.com/i_robin // Skype: robin.muilwijk

Bertrand Dunogier

Saturday 30 January 2010 2:52:20 am

You need to invoke ezcache.php with the --purge option to "physically" clean expired cluster items.

There really is a functional gap here. This will indeed clear items, but you'll have to specify a criteria for this, like a cache tag, a cache id, or an expiry date. For instance, purge all items that are more than 1 week old. I have triggered an internal investigation about that, and hope to get a solution soon, but in the meantime, you could try stuff like:

# purge cache blocks older than 1 week
php bin/php/ezcache.php --purge --expiry="1 week" --clear-id=template-block

P.S. I have added a detailled issue for this, http://issues.ez.no/16097. Don't hesitate to comment if required !

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

Robin Muilwijk

Saturday 30 January 2010 10:24:14 am

Thanks for picking this up Bertrand.

-- Robin

Board member, eZ Publish Community Project Board - Member of the share.ez.no team - Key values: Openness and Innovation.

LinkedIn: http://nl.linkedin.com/in/robinmuilwijk // Twitter: http://twitter.com/i_robin // Skype: robin.muilwijk

Bertrand Dunogier

Saturday 30 January 2010 10:47:16 am

"

Thanks for picking this up Bertrand.

"

You're welcome.

P.S. Keep in mind that this will NOT be part of eZ publish 4.3.0. If you were in Geneva, you now know more about the release cycle, and the 4.3 branch will enter its ice age next monday (kind of).

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

Jim Thaxton

Monday 01 February 2010 12:12:03 pm

Thanks all! Glad to know I did not miss the obvious.

Web Developer
Coupon Cabin
Chicago, IL

Massimo Sanna

Monday 12 April 2010 9:00:30 am

Hi there,

just followed these instructions to successfully bring up a new installation running on eZDFS.

Only thing that doesn't seem to work are the CSS...

Basically, on this page for teamroom (?!) it adds that in clustered environments you need to add a rewrite rule for /var/cache/public in order for it to be served by index_cluster.php (maybe that suggestion should be added to the ezdfs instructions) http://ez.no/doc/extensions/ez_teamroom/setup_and_user_guide/requirements/ez_js_core/installation

Although, even if I added that rewrite rule, the css wasn't coming through. I checked in the NFS share and it was there, but it wasn't getting created in the local /var/cache/public so it didn't display at all.

Have you got any clues guys?

Thanks a lot,

Max

Bertrand Dunogier

Tuesday 13 April 2010 3:18:01 am

Hi Massimo.

Could you please:

  • Show me your rewrite rules
  • Try opening the CSS file in its own window, and tell me what exacly happens (404, module not found, etc)

It doesn't really matter if the file in var/cache/public, since it should be streamed out of NFS by index_cluster.php. It should also be cached locally, but even if we've skipped this, it won't prevent it from working.

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

Massimo Sanna

Thursday 15 April 2010 6:53:10 am

Hi Bertrand, thanks a lot for your help! (I didn't receive the topic notification even if I activated the tracking... maybe it went into spam).

So, here's the bits you asked for:

RewriteEngine On
Rewriterule ^/var/([^/]+/)?storage/images-versioned/.* /index_cluster.php [L]
Rewriterule ^/var/([^/]+/)?storage/images/.* /index_cluster.php [L]
RewriteRule ^/var/([^/]+/)?cache/public/.* /index_cluster.php [L]
RewriteRule content/treemenu/? /index_treemenu.php [L]
Rewriterule ^/var/storage/.* - [L]
Rewriterule ^/var/[^/]+/storage/.* - [L]
RewriteRule ^/var/cache/texttoimage/.* - [L]
RewriteRule ^/var/[^/]+/cache/texttoimage/.* - [L]
Rewriterule ^/design/[^/]+/(stylesheets|images|javascript)/.* - [L]
Rewriterule ^/share/icons/.* - [L]
Rewriterule ^/extension/[^/]+/design/[^/]+/(stylesheets|images|lib|flash|javascripts?)/.* - [L]
Rewriterule ^/packages/styles/.+/(stylesheets|images|javascript)/[^/]+/.* - [L]
RewriteRule ^/packages/styles/.+/thumbnail/.* - [L]
RewriteRule ^/favicon\.ico - [L]
RewriteRule ^/robots\.txt - [L]
RewriteRule .* /index.php

Now that I paid more attention, if I try to access the css directly, it downloads it in my download folder, instead of displaying it. Maybe that's the reason why it doesn't render in the page?

Thanks a lot,

Max

Nicolas Pastorino

Friday 16 April 2010 1:19:12 am

"

Now that I paid more attention, if I try to access the css directly, it downloads it in my download folder, instead of displaying it.

"

Content type issue ? Something getting in the way of Apache's standard behaviour in this regard ?

Cheers,

--
Nicolas Pastorino
Director Community - eZ
Member of the Community Project Board

eZ Publish Community on twitter: http://twitter.com/ezcommunity

t : http://twitter.com/jeanvoye
G+ : http://plus.tl/jeanvoye

Bertrand Dunogier

Friday 16 April 2010 3:22:49 am

I can smell a new bug here...

Can you run this query:

SELECT * FROM ezdfsfile WHERE name LIKE 'var/path/to/your/file.css';

and show me the result ?

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

Massimo Sanna

Friday 16 April 2010 6:05:50 am

Hi Bertrand!

Here's the output of the query:

mysql> select * from ezdfsfile where name like 'var/ezflow_site/cache/public/stylesheets/20ffd5f9c8f19244360d3ac9e67ae601_all.css';+-----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+----------------------------------+----------+---------------+--------+------------+---------+--------+| name | name_trunk | name_hash | datatype | scope | size | mtime | expired | status |+-----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+----------------------------------+----------+---------------+--------+------------+---------+--------+| var/ezflow_site/cache/public/stylesheets/20ffd5f9c8f19244360d3ac9e67ae601_all.css | var/ezflow_site/cache/public/stylesheets/20ffd5f9c8f19244360d3ac9e67ae601_all.css | 1236916bc6800445ea6bb73c3e500ef8 | misc | UNKNOWN_SCOPE | 103863 | 1271338992 | 0 | 0 |+-----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+----------------------------------+----------+---------------+--------+------------+---------+--------+1 row in set (0.00 sec)
mysql> select * from ezdfsfile where name like 'var/ezflow_site/cache/public/stylesheets/20ffd5f9c8f19244360d3ac9e67ae601_all.css';+-----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+----------------------------------+----------+---------------+--------+------------+---------+--------+| name | name_trunk | name_hash | datatype | scope | size | mtime | expired | status |+-----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+----------------------------------+----------+---------------+--------+------------+---------+--------+| var/ezflow_site/cache/public/stylesheets/20ffd5f9c8f19244360d3ac9e67ae601_all.css | var/ezflow_site/cache/public/stylesheets/20ffd5f9c8f19244360d3ac9e67ae601_all.css | 1236916bc6800445ea6bb73c3e500ef8 | misc | UNKNOWN_SCOPE | 103863 | 1271338992 | 0 | 0 |+-----------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+----------------------------------+----------+---------------+--------+------------+---------+--------+1 row in set (0.00 sec)

I confirm I can download that file on my desktop if I enter the URL in the browser.

Sorry for the mess while pasting it, but if I try to paste it into a literal everything ends up on one line (another bug of the ezoe) and it's even less clear than the above mess...

Max

Bertrand Dunogier

Friday 16 April 2010 7:34:09 am

Massimo,

I have indeed reproduced this issue. There are two sides of it.

For one, the path of these items (both CSS and JS) isn't known to the cluster rewrite rules. This explains why you might get a 404 when trying to fetch these from the browser (at least we experienced it, and it makes sense). This requires a change to the rules, by adding this one, which you already have, except for the (stylesheets|javascript) added security:

RewriteRule ^/var/([^/]+/)?cache/public/(stylesheets|javascript).* /index_cluster.php [L]

Then the other issue is, as I feared, the content type of these entries in the table. Since nothing is specified by ezjscore, it gets inserted with a 'misc' content type, and this is what the browser receives. Here, firefox accepted that, while chrome and safari didn't. I don't even want to know what IE does.

We're gonna have to fix this soon. In the meantime, it is very dirty, but you can fix it by setting the datatype field for these entries to text/css or text/javascript. These can take some time as the wildcard really is an ugly solution, but it will do for a workaround ;)

-- This is for ezdb, but the same queries will work with eZDB using ezdbfile instead
UPDATE ezdfsfile SET datatype = 'text/css' WHERE name LIKE '%/cache/public/stylesheets/%';
UPDATE ezdfsfile SET datatype = 'text/javascript' WHERE name LIKE '%/cache/public/javascript/%';

Could you post a public issue for that on the issue tracker ?

P.S. I really have too many accounts, this is getting tricky...

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

Massimo Sanna

Friday 16 April 2010 8:50:15 am

Hi Bertrand, thanks for helping me out with this issue.

I suggest for the time being to just disable the packer in ezjscore.ini, it's the cleanest solution and if you empty the caches it keeps on working (whereas with the sql workaround, I believe it will start behaving weirdly again, by setting the datatype to misc, isn't it?).

I posted a public issue here, in case you want to watch it:

http://issues.ez.no/IssueView.php?Id=16669&activeItem=1

Have a nice weekend!
Max

Bertrand Dunogier

Saturday 17 April 2010 4:27:29 am

Great, Massimo !

I think I know how to fix this issue in ezjscore, it's not much. I'll post it here and on the issue tracker when it's ready.

Edit: I have attached a patch for this bug to the bug you have submitted: issues.ez.no/IssueView.php?Id=16669&ProjectId=3#Comment265677

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier