Yourkit & Tomcat webapp on Fedora

Yourkit is a great tool for profiling Java applications.  I'm using it to do some profiling on our webapp that is deployed in a Tomcat on a remote server (virtual machine on Rackspace).  I've found that Yourkit is really simple and intuitive to use and set up, for the most part.

The only issue I've had was getting it set up on the remote server to hook my local GUI to.  Theoretically, it is a very simple process.  Download yourkit to the remote server, start up tomcat, and type the following at the command prompt:

/bin/yjp.sh -integrate

I've downloaded & installed the linux version into my /opt/ folder on my remote server.  My remote server is Fedora 14 64bit.  When I run the above command, I get the following prompts:

And here is where I run into a problem. Where is the tomcat startup.sh?? I do a search on the remote machine for the file and come up empty handed. I sift around online and come up empty handed.  Finally after trial and error, I find that the file that is used on Fedora for starting up tomcat is called:

/usr/sbin/tomcat6

I would have thought this would be the end of it.  Unfortunately, it looks the -integrate command munges the startup.sh to add an additional JAVA_OPTS to create the hooks for Yourkit.  And of course /usr/sbin/tomcat6 is not formatted how Yourkit is expecting the startup.sh to be formatted.  So no go.

The next option is to manually add the JAVA_OPTS to the file.  What I've done is add the following:

-agentpath:/opt/yjp-9.5.6/bin/linux-x86-64/libyjpagent.so=port=10001

after the {JAVA_OPTS} in the start block of the /usr/sbin/tomcat6 script (around line 30):

if [ "$1" = "start" ]; then
  ${JAVACMD} $JAVA_OPTS -agentpath:/opt/yjp-9.5.6/bin/linux-x86-64/libyjpagent.so=port=10001 $CATALINA_OPTS \
    -classpath "$CLASSPATH" \
    -Dcatalina.base="$CATALINA_BASE" \
    -Dcatalina.home="$CATALINA_HOME" \
    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
    -Djava.io.tmpdir="$CATALINA_TMPDIR" \
    -Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
    -Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
    org.apache.catalina.startup.Bootstrap start \
    >> ${CATALINA_BASE}/logs/catalina.out 2>&1 &
    if [ ! -z "$CATALINA_PID" ]; then
      echo $! > $CATALINA_PID
    fi

I have added an additional startup option to this, specifying the port to use.  Pay close attention when inserting this option and be sure that is a space between $JAVA_OPTS and [agentpath] as well as a space between [agentpath] and $CATALINA_OPTS.  Because I will not always want the agent attached (overhead and performance reasons), I created two copies of the tomcat6, a tomcat6_yourkit & tomcat6_orig (original tomcat6 file) that I copy to tomcat6 as needed to swap the configurations.

Finally, the last thing that needs to be done is to open up the port in the iptables file on the remote server to allow the connection to be made.

Summary:

1. Download the correct distribution for the remote server's platform to /opt/.
   
   (In my case I need the Linux distribution)

   [root@bakeYourkit opt]# wget http://www.yourkit.com/download/yjp-9.5.6-linux.tar.bz2

2. Extract the file.
   
   (In my case it is a tar.bz2, which I always forget how to do).
3. Make a copy of /usr/sbin/tomcat6 as /usr/sbin/tomcat6_orig
   

   [root@bakeYourkit ~]# cp /usr/sbin/tomcat6 /usr/sbin/tomcat6_orig

4. Edit /usr/sbin/tomcat6, save, and make a copy as /usr/sbin/tomcat6_yourkit
   

   [root@bakeYourkit ~]# vim /usr/sbin/tomcat6
   [root@bakeYourkit ~]# cp /usr/sbin/tomcat6 /usr/sbin/tomcat6_yourkit
   

5. Edit the iptables to listen to the port specified in the startup option above (in my example here, port number 10001).  See this post for information about editing the iptables file.
6. Restart tomcat to pick up the options.
7. On your local machine, start up yourkit and connect to the remote server:
   

Ta-da!  That's all.  Now if you want to switch off the agent, copy the original /usr/sbin/tomcat6_orig over the /usr/sbin/tomcat6 file.  There is definitely a nicer way to this, possibly by adding some environment variable and setting it to.  But this works for me.

Perl script to "copy" a Rackspace image

I recently found myself putting together a perl script that performs an rsync of one server in the Rackspace cloud to another server on another account.  Why?  See my next post about restrictions found with Rackspace for details.  Long story short, I needed a way to copy an image from one account to another and Rackspace recommends using rsync.

Since I will be needing to do this at least once a month for my testing, I thought it best to write up a script using perl that does the rsync.  The first thing was to figure out to successfully do this rsync before putting things together in a script.  I followed the instructions and when I rebooted my new server with the files rsync'ed over, it failed to accept ssh anymore.  On a whim, I decided to go through the basic exclude file from the instructions to see if those directories/files really lived where the file says they would be.

Ah-ha! That was my problem. I'm not sure which distro was used to create the exclude file, but some of the recommended files/directories to be excluded were located in different places.

#Suggested by Rackspace
/etc/hostname

#Actual location on Fedora 14 64bit
/bin/hostname
 

#Suggested by Rackspace
/etc/modules 

#Actual location on Fedora 14 64bit
/etc/sysconfig/modules

What I did was to do a

find / -name [sometext]

Where [sometext] I started from the end of the path and worked my way in to find the location. It was definitely trial and error.  Here is what I have for my final exclude file for Fedora 14 64bit:

#exclude.txt
/proc
/sys
/tmp
/dev
/root/copyImage.log
/root/.ssh/
/var/lock
/etc/fstab
/etc/mtab
/etc/resolv.conf
/usr/share/dbus-1/interfaces
/etc/networks
/etc/sysconfig/network
/etc/sysconfig/network-scripts
/etc/sysconfig/iptables-config
/lib/modules/
/usr/share/selinux/devel/include/kernel
/bin/hostname
/usr/lib64/gettext/hostname
/etc/hosts
/etc/modprobe*
/etc/selinux/targeted/modules
/etc/selinux/targeted/modules/active/modules
/etc/sysconfig/modules
/etc/selinux/targeted/modules
/etc/selinux/targeted/modules/active/modules
/usr/lib64/gio/modules
/lib/udev/devices/net
/usr/lib/ruby/1.8/net
/etc/init/

So now that I have a working exclude file, the next thing was to code up a perl script to do this for me.  In order to automate this through the script I needed to be able to:

• Create an ssh tunnel from the "sending" server to the "receiving" server and be able to generate an ssh key on the "receiving" server.
• Add the "sending" server's public key to the "receiving" server's authorized_key file
• In my case, as part of the image, there is an authorized_key file with some public keys in it, which I want available to all servers generated with the image. I added the "sending" server's public key to this file and rsync'ed this file over first
• Rsync the "sending" server data to the "receiving" server.

Two of the largest hurdle is:

1. Open the ssh connection to the "receiving" server
2. Know when the "receiving" server is asking for the password and supply it

There are many many perl modules available that do just this in various ways. I didn't think to make a note of which ones I did try, unfortunately.  I ended up using Net::OpenSSH, it had the best documentation and did everything I needed.  I used this in conjunction with Expect module.


Neither of these modules are included with the Perl package that comes with Fedora 14 64 bit.  In trying to add these modules, I learned some more things.  For example, I learned about CPAN.  This is a little piece of software that manages installation and dependency resolution for perl modules.  Other benefits I found are:

• Access to modules that may not be in repos yet, i.e. Net::OpenSSH is not available via yum on Fedora 14
• Type in instmodsh at the command prompt and you can see a list of all installed modules and get information about the modules
• If module is installed via yum install , will not show up in instmodsh

Here is a link describing how to install CPAN.  After you follow the instructions at that link, you should also check to see if gcc is installed.  In my case, it was also not included in my default Fedora 14 64bit installation.  It is needed to compile some of the modules (in this case, one of the dependencies for Expect).  The error I got before I installed it is:

ERROR: cannot run the configured compiler 'gcc'
(see conf/compilerok.log). Suggestions:
1) The complier 'gcc' is not in your PATH. Add it
   to the PATH and try again. OR
2) The compiler isn't installed on your system. Install it. OR
3) You only have a different compiler installed (e.g. 'gcc').
   Either fix the compiler config in the perl Config.pm
   or install a perl that was built with the right compiler
   (you could build perl yourself with the available compiler).

Note: this is a system-administration issue, please ask your local
admin for help. Thank you.

Warning: No success on command[/usr/bin/perl Makefile.PL]
'YAML' not installed, will not store persistent state
  TODDR/IO-Tty-1.10.tar.gz
  /usr/bin/perl Makefile.PL -- NOT OK
Could not read metadata file. Falling back to other methods to determine prerequisites
Failed during this command:
 TODDR/IO-Tty-1.10.tar.gz                     : writemakefile NO '/usr/bin/perl Makefile.PL' returned status 6400

Finally, my last hurdle for putting together this script is trouble using the open function in perl.

#DOES *NOT* WORK
open(CAT_FILE,"cat /root/test.txt >>/root/test2.txt ") || die "Failed: $!\n";

#DOES WORK
open(CAT_FILE,"| cat /root/test.txt >>/root/test2.txt ") || die "Failed: $!\n";

Can you spot the difference?
So apparently that "|" is required ... oops...guess the documentation was right. *insert chagrin here*.

So without further ado, please find below my full script to do the rsync step of copying an image to another server on Rackspace: