Import XML Data Topic

Wednesday 07 December 2005 1:09:59 pm

I created this thread to discuss further evolutions (and issues) of the ImportXMLData contrib. This should help leaving the contrib comment area to message directly related to the use of the contrib and not implementation detail.

Best regards

Olivier

Wednesday 07 December 2005 2:17:32 pm

Xavier,

I already dropped you an email saying I was waiting for your changes but unfortunately you could not answer. I'll retry...

Olivier

Thursday 08 December 2005 12:16:08 am

Sorry, but

$fieldValue = utf8_decode($xPathEngine->wholeText("$path_item/$fieldName"."[1]"));

does not solve problem. I'm still getting ??? instead of UTF8 symbols.
I do not unerstand, why utf8_decode is used. As i understood from documentation http://lt.php.net/manual/en/function.utf8-decode.php , this function converts string to ISO-8859-1. After such convertion UTF8 symbols are lost...

Problem is that xml_parser reads file as iso-8859-1 and ignores encoding specified in xml file.

This problem should be fixed in xpathengine. I putted to XPatth.class.php line 1680:

      $parser = xml_parser_create('UTF-8');

This is only way i found, to get it working with utf-8 ...

{set-block scope=root variable=cache_ttl}0{/set-block}

Friday 09 December 2005 7:41:08 am

Import function has limitations: i cannot import more than 340 records by one turn... :(

{set-block scope=root variable=cache_ttl}0{/set-block}

Wednesday 14 December 2005 12:28:02 pm

Surely
I guess we need to add a file called import-cli.php that would parse the argument and call the

function &importXMLData( $xmldata, $datatype, $remove, $movetotrash) 

in importXMLDatafunctioncollection.

Of course context should be set appropriately if not I think the call:

$class =& eZContentClass::fetchByIdentifier( $identifiantClasse );

as many other kernel related eZ API calls.

I think we should have a look at how runcronjob.php is written.

Another option would be to wait for the Xavier hacks to this extension because I know he is using a script approach to run this extension.

Thursday 15 December 2005 12:36:54 am

Sorry Olivier,

I now I'm late, but I can't find the time to clean up the mess of custom things I've added. I'll try to do that this week-end, the delay is just ridiculous.

Thanks for your patience

X+

http://www.sydesy.com

Thursday 15 December 2005 5:55:23 am

There is it:

<?php
//
// Created on: <2005-12-15 14:52:57 vytis>
//
// This file may be distributed and/or modified under the terms of the
// "GNU General Public License" version 2 as published by the Free
// Software Foundation and appearing in the file LICENSE included in
// the packaging of this file.
//
// This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING
// THE WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR
// PURPOSE.
//
// The "GNU General Public License" (GPL) is available at
// http://www.gnu.org/copyleft/gpl.html.
//
// Contact licence@ez.no if any conditions of this licencing isn't clear to
// you.
//

include_once( 'lib/ezutils/classes/ezcli.php' );
include_once( 'kernel/classes/ezscript.php' );

$cli =& eZCLI::instance();
$script =& eZScript::instance( array( 'description' => ( "eZ publish Script Executor\n\n" .
                                                         "Allows execution of simple PHP scripts which uses eZ publish functionality,\n" .
                                                         "when the script is called all necessary initialization is done\n" .
                                                         "\n" .
                                                         "ezexec.php myscript.php" ),
                                      'use-session' => true,
                                      'use-modules' => true,
                                      'use-extensions' => true ) );

$script->startup();

$options = $script->getOptions( "",
                                "[scriptfile]",
                                array() );

if ( count( $options['arguments'] ) < 5 )
{
    $script->shutdown( 1, "Usage of import script:\n SiteAccess, \n XML file, \n datatype, \n remove? (0 - no, 1 - yes) , \n move to trash? (0 - no, 1 - yes),\n user's ID");
    die();
}

$script->setUseSiteAccess($options['arguments'][0]);

$options = $script->getOptions( "",
                                "[scriptfile]",
                                array() );
$script->initialize();


include_once ('extension/importXMLData/modules/importXMLData/importXMLDatafunctioncollection.php');
include_once ('kernel/classes/ezcontentclass.php');
 
$xmldata = file_get_contents($options['arguments'][1]);
importXMLDataFunctionCollection::importXMLData($xmldata, $options['arguments'][2], $options['arguments'][3], $options['arguments'][4], $options['arguments'][5]);

$script->shutdown();
?>

This need some modifications of import script:
1. I didn't login system. Instead of this, i put users ID as parameter in importXMLDatafunctioncollection.php:

	function &importXMLData( $xmldata, $datatype, $remove, $movetotrash, $userID)

So, you should delete from importXMLDatafunctioncollection.php:

		  $user =& eZUser::currentUser();
		  // set user ID 
		  $userID =& $user->attribute( 'contentobject_id' );

2. Additionaly, i put some debug print to see progres of import:
Iin the beginning of function:

 
$cli =& eZCLI::instance();

Then:

 
		$cli->output( "Preparing list for import" );
		$paths_item = $xPathEngine->match("//$listTag/$itemTag");
		$cli->output("List size: ".count($paths_item));

Then i changed:

 
	$ii=0;  
	foreach ($paths_item as $path_item) 
	{
		$ii++;
		if( bcmod($ii, 100) == 0)
		{
			$cli->output("\n imported $ii of ".count($paths_item) );
		}		
		foreach($fieldNameList as $fieldName) 
		{
 

Good luck.
Next, i'm going to make shell script to import multiple xml documents. I think this is usefull, when you need to import several thousands of records, because xPathEngine uses to much memory.

{set-block scope=root variable=cache_ttl}0{/set-block}

Friday 03 February 2006 6:02:47 am

Hey.

I've tried to change the importer so that it's possible to import utf-8 files, but it doesn't work...

$parser = xml_parser_create('UTF-8');

doesnt't work

My problem is that I can't import chars like "ä", "ö", "ü"

Any ideas?? Thanks a lot...

Linux is like a wigwam; no windows, now gates, and apache inside!

Monday 06 February 2006 12:02:01 am

Hm, thanks for your reply, but it still doesn't work...

btw: I'm using eZ version 3.6.0

Linux is like a wigwam; no windows, now gates, and apache inside!

Monday 06 February 2006 1:05:17 am

Hm, ok, I can't write the symbols down here, they are changed into html letters... I made a screenshot:

http://www.philip-kahlen.de/import_error.gif

Linux is like a wigwam; no windows, now gates, and apache inside!