Removing objects takes loooong time, can't seem to disable solr indexing for it.

Tuesday 12 January 2010 9:46:58 am

I'm using EZPublish 4.2.0 and EZFind 2.1.

We are currently in development, and have some product classes with thousands of objects (one with > 10,000). When we need to remove them for a reimport, it can take hours.

Importing was slow, too, but since I set DelayedIndexing=enabled, that is now very speedy. That change did not affect object removal speed.

I tried setting OptimizeOnCommit=disabled in ezfind.ini, and this had no effect on object removal speed, either. (Also tried clearing all caches, no luck).

Viewing processes, Java is topping out the CPU during object removal, so solr is clearly still being engaged.

Is there a way to disable the re-indexing, or is it slow like this for everyone, and this is something I should just get used to?

x

Saturday 16 January 2010 7:28:42 am

Thanks for the reply!

The re-indexing is very fast, actually, when done after-the-fact. For example, DelayedIndexing is enabled, and that prevents indexing during the import, and I do a re-indexing pass after that which is very speedy (just a few minutes). It's only the removal that is still slow (2-3 seconds per object removed!), and Java resources spike during the object removal, so I'm convinced that Solr is doing work on the backend, even though DelayedIndexing is enabled and OptimizeOnCommit is disabled.

So, delaying re-indexing is not a problem...

x

Saturday 16 January 2010 12:47:14 pm

Removing those objects through the API calls a delete and commit with every object to the backend index. This can be indeed be quite slow when there are many objects.

Can you file an issue for this? In the current trunk, there are optimisations for high write traffic sites, I'll add a config option to eZ Find that disables commits on delete and add a small script that you can use after the delete operations are finished to issue a commit.

(warning: before the commit, deleted objects will still show up in the search and trigger fatal errors as the eZP side calls will fail obviously)

@Robin: disabling ezfind alltogether during this delete will indeed speed up the whole operation as well

An alternative would be to patch Solr, so it also accepts a "commit within" parameter like it does for updates/additions ... I'll have a look at that too, the Java code to accomplish this is quite simple at first sight.

hth

Paul

eZ Publish, eZ Find, Solr expert consulting and training
http://twitter.com/paulborgermans

Monday 18 January 2010 6:18:56 am

Thank you for your responses. I will file the issue today, after I'm out of meetings.

x