Update2: Removed the prototype from the subroutine.

I’ve always had a problem with recursion. Not with the general theory that a function will call itself, etc – no, that’s easy. The hard part was when I had to deal with complex data structures in Perl (an array- or hashref containing a hash of arrays of hashes, a gazillion levels deep). Well, I guess anyone would have a hard time with that kind of data.

Anyway, in this post I don’t intend to get all complicated explaining all the kinds of recursions out there. If you want that, check this article at wikipedia. What I do want to do is help all of those who are in the situation I was in, by explaining in the simplest way possible how to deal with this scenario.

Let’s start with a need. I have a complex data structure that needs its spaces trimmed on both sides. But since I’m lazy, I’d like my subroutine to modify the data directly, and not return the modified value (pass by reference, not pass by value).

Here’s our data structure:

    my $data = [
                       {
                          key1 => '   trim me!   ',
                          key2 => '   trim me too!    ',
                       },
                       [
                          'some element to trim   ',
                          '    another one    ',
                       ],
                       '    a simple string needing trimming    ',
                  ];

$data explained: an array containing 3 elements: element 0 is a hashref of keys "key1" and "key2", element 1 is an arrayref of 2 elements. Element 3 is a simple string. All values have some extra spaces that need trimming (or so they say). We could use whatever number of levels and data types we want (except for anonymous subroutines, I guess – let’s not get too complicated).

Now, to trim all that, I want to be able to simply call trim() à la PHP.

     trim($data); # note the lack of the lvalue (lvalue = rvalue)

I also want it to accept simple arrays and hashes, and the references thereof: trim(@array); trim(@array); trim(%hash); trim(%hash); trim($string). After all, I never know what kind of data my colleagues will be working with. Better have it deal with everything.

The logic to do that is this: our subroutine will have to do the trimming (s///g) on scalars only. For that, it has to check if the data it received is a hash, array, etc, and if it is, iterate through each element and trim the value… but only if the element is not itself a hash, array, etc. Found it confusing? No problem, it really is.

In Perl, if I tried to remove the white space from element 0 of my $data variable, it wouldn’t work. The reason being is that if I printed $data->[0] onto the screen, I’d get a funny looking output, something like HASH(0x1004f5f0). That’s Perl’s way of saying that you have a HASH structure stored in memory position 0x1004f5f0. You can try to trim the spaces off of that string, but it won’t do you any good. The elements of your hash will still be untouched. That’s why you need to de-reference your data structures and dive into them.

To de-reference a structure is simple, just add a % in front of the variable if it’s a hashref, or an @ if it’s an array. But how do you know which is which? Use ref().

       print ref($data->[0]) . "n"; # HASH
       print ref($data->[1]) . "n"; # ARRAY
       print ref($data->[3]) . "n"; # empty string, which is false

ref() tells you what kind of data you are dealing with. It returns CODE if you have a closure or anonymous subroutine, but we’re not going there today.

So, now that we know how to identify the type of element we’re going to be working with, we can build our subroutine…

sub trim() {
	for my $param (@_) {
		if (ref($param) eq 'ARRAY') {
			for my $element (@{$param}) {
				trim($element);
			}
		}
		elsif (ref($param) eq 'HASH') {
			for my $val (values %{$param}) {
				trim($val);
			}
		}
		elsif (ref($param) eq 'CODE') {
			return;
		}
		else {
			$param =~ s/(^s+|s+$)//g;
		}
	}
}

trim() explained:

We’re working with passing elements by reference instead of by value. This means that the elements themselves will be modified – no need to return any data. The first thing we do is to iterate through all parameters passed to trim(). In a subroutine, parameters (in our case, variables) are populated into the special @_ array, allowing us to call trim($var1, $var2, $var3) if we want.

We iterate through all elements of @_ and verify if they are an Array. If they are, we iterate through each of their elements once, and call trim() again against them. That will handle as many nested arrays we want (or that your computer can handle). Now we have to make it deal with hashes. Same technique – use ref() to see if it’s a hash. If it is, then iterate through each of its key/pair elements. There are several ways to do that. I personally prefer calling keys to get the keys and use them to fetch the values of the hash. The value of the hash is passed to trim() for more validation. We also check to see if we received a sub { } (anonymous subroutine). In that case, we do nothing, just return.

Finally, after handling Arrays, Hashes and Anonymous subroutines, we can set up the actual trimming of the strings. We take the $_[$i] which is the parameter passed and remove the leading and trailing spaces with one neat substitution: ^s+ stands for leading spaces, s+$ stands for trailing spaces, and it’s all joined by the ( | ) (this or that). We only call it once because we’re using the global (g) modifier of the substitution s///g.

And that’s all there is to it!

A note on prototypes: This post generated a healthy discussion on prototypes. I had originally added the dollar prototype to the trim() subroutine, but that was forcing it to accept only scalars (strings and references), and not working with normal hashes and arrays. Thanks to everyone who participated in the discussion.

Here is a list of what I used:

Machine: Sun Solaris SPARC on build and target machines
Oracle: Oracle 10g client (build machine only)
Perl: v. 5.10.0 (build machine only)
Perl modules: DBI (1.607), DBD::Oracle (1.22), PAR (0.983), pp (0.982), PerlIO (1.04) (build machine only)

How I did it:

Being the lazy bastard that I am, I decided to write a Perl module that would do my DBI connections for me once I supplied it the SID, username, and password. This module isn’t in CPAN, but you can probably find similar ones in there. Who knows maybe one day I’ll make it available to the public, but for now I will only explain how it works and show the tidbits I used to get it connecting without the Oracle client installed. Let’s call my module MyDB.pm (at the time of this writing, there is no MyDB.pm module in CPAN).

MyDB.pm: exports one subroutine: connect().  connect() takes an array reference of 3 values: SID, username, password. It then uses the given SID to check against %configured_dbs and fetch hostname/IP,  port, and alternative SID. It then connects to the DBI. Pretty straight-forward.

Oracle Client: You’ll need the Oracle Client installed in your build machine. Or not. As long as you have access to the following files (the example here is for 10.1 version – your filenames may vary): libclntsh.so.10.1, and libnnz10.so. These are the magic libs that we will carry along with us.

PAR/pp: The Perl Archive Tool is a VERY handy module. It basically allows you to pack modules/files into a regular zip (renamed to .par) and even to create self extracting executables. It comes with tools such as parl to load those packages and deploy the scripts inside, or you can run it from inside a separate script to use your modules without having to install them in a specific path. Oh, and with the PAR self extracting executable, you can even bundle Perl core files and run it in a machine that doesn’t have Perl installed! Sweet, eh? pp is part of the PAR family – it’s thescript that does the creation of the par packages. One downside of PAR is that it creates potentially HUGE packages. A simple hello world can be over 2Mb large. But that’s not a concern in this tutorial, since we will be getting rid of having to keep Oracle up to date in every single machine, changing environment variables and the such.

When running a PAR package, it calculates the MD5SUM of the file and creates a temporary directory under /var/tmp/par-userId/cache-md5sum (replace userId with the ID of the person running the file, and md5sum with the actual md5sum) . It then unpacks the contents of the PAR into that directory and sets some environment variables – some of which will tell Perl to check in there for modules.  However, it seems to not automatically extract the libs, so we have to modify MyDB.pm to have it behave appropriately.

Making MyDB.pm work when called in a PAR context

Create a BEGIN block checking for $ENV{PAR_TEMP}. This is the value of the PAR temp directory where the files should be extracted. Since PAR creates zipped files, it offers a method to return an Archive::Zip  handler:

my $zip_handler = PAR::par_handle(‘MyDB.par’);

Now you can use Archive::Zip methods to manipulate your .par file. I used the following code for my BEGIN block:

BEGIN {
	if ($ENV{PAR_TEMP}) {

		# Get $par_cache (easier to type than $ENV{PAR_TEMP})
		my $par_cache = $ENV{PAR_TEMP};

		# Get Zip handler
		my $par_file = PAR::handle("MyDB.par");

		# Set files to look for
		my %membersToFind = (’libclntsh.so.10.1’ =>1,
							 ’libnnz10.so’ => 1 );

		# Unpack files
		for my $m ($par_file->members()) {

			# Get path/name of the zipped file
			my $fileName = $m->fileName();

			# Prepare to check if the libs
			# have already been extracted
			use Find::File;

			# Iterate through %membersToFind keys
			for my $k (keys %membersToFind) {

				# If zipped file is one of the keys...
				if ($fileName =~ /$k/) {

					# do nothing if the file has already been extracted
					next if find(sub { /$k/ or return 0 },$par_cache); 

					# ...then remove from hash (we’ll check the hash later)...
					delete $membersToFind{$k};

					# ... and unpack into $par_cache directory
					$par_file->extractMember($fileName,"$par_cache") ||
					warn ("Failed to extract $fileName n");

				}
			}
		}

		# Check to see if any of the files wasn’t unpacked because it wasn’t found
		if (scalar(keys %membersToFind) > 0) {

			warn("The following file(s) could not be found in PAR file: " .  join(’,’,@{ keys %membersToFind }) ."n");

		}
	}
}

So now MyDB.pm will know what to do if $ENV{PAR_TEMP} is set.  The next step is to generate the PAR file.

Creating the PAR package

You’ll need a dummy perl script to pass to pp:

	$ echo "use MyDB;" > dummy.pl

and then create the PAR file. Make sure MyDB.pm and your oracle libs are in places where Perl can find it. I placed MyDB.pm in the same directory as dummy.pl, and set LD_LIBRARY_PATH to my $ORACLE_HOME/lib32 (if you’re using Instant Client, set it to $ORACLE_HOME/instantclient or wherever your instant client keeps the libs). The pp command is this (on a shell prompt):

$ pp -z 9 -v 3 -p -d -o MyDB.par
	-X DBD::Proxy
	-X DBD::DBM
	-X DBD::File
	-X DBD::NullP
	-X DBD::Gofer
	-X DBD::Sponge
	-X DBD::ExampleP
	-X DBD::mysql
	-X DBD::Gofer::Policy::rush
	-X DBD::Gofer::Policy::classic
	-X DBD::Gofer::Policy::Base
	-X DBD::Gofer::Policy::pedantic
	-X DBD::Gofer::Transport::pipeone
	-X DBD::Gofer::Transport::Base
	-X DBD::Gofer::Transport::stream
	-X DBD::Gofer::Transport::null
	-X DBD::mysql::GetInfo
	-X DBD::Chart
	-X DBD::Chart::Plot
	-M MyDB
	-M PerlIO
	-l libclntsh.so.10.1
	-l libnnz10.so
        -l libociei.so   # for Oracle instant client users only! Thanks to Jonathon Robinson for the heads up. (see comments for details)
	dummy.pl

pp command explained

pp: the PAR packer script
-z 9: maximum compression
-v 3: maximum verbosity
-p: create par file
-d: dependent. This means it will not add the Perl interpreter to the bundle. This reduces its size, but makes it rely on either perl being installed in the target machine, or your script (the one that’s going to use MyDB) being PAR’d without the -d option.
-0 MyDB.par: the filename to be used on output
-X module: removes un-necessary modules. Read more about it here.
-M module: the modules you want to add (PAR sometimes misses a module during its dependency scan).  PerlIO is required.
-l lib: adds the additional libs
dummy.pl: last item of the command, tells pp what the main script is.
There – after lots of output on the screen, you should have a pretty large MyDB.par (at least 7Mb). Now we have to alter our main script (the one that uses MyDB in the first place) to have it handle the PAR file. Add the following lines right after your use strict; line (you DO use strict, RIGHT?!).

          use PAR './MyDB.par';
          use MyDB;
          # the rest is business as usual

Once altered, you can choose to pp your script to have it run without a perl interpreter on another machine. This might take several attempts, since you will have to test the executable a few times to make sure you got all your modules bundled. The basic pp command though is this:

       $ pp -z 9 -v 3 -o yourscript.exe yourscript.pl    # .exe not required

Here the absense of -d makes it assume -B which bundles Perl inside the executable. As mentioned before, you might need to add some -M module parameters before yourscript.pl to catch any missing modules.

Running the scripts

To run the scripts, just send them to the target machine and execute yourscript.exe. It will fetch MyDB.par in the same directory, extract it to $ENV{PAR_TEMP} the first time (so it might be a little slow at first) and carry on with its logic.

Caveat

I strongly advise that you delete your par_cache directory between rebuilds (while testing). The reason for this is that the cache will be created by the execution of yourscript.exe and NOT by MyDB.par. When this happens and you alter MyDB.par, the changes will not be reflected when you run yourscript.exe.

If you liked this post, please leave a comment. If you didn’t, leave a comment too! I’d like to know what people think of my blog and ways to improve it.

Book Suggestions: