March 2010 Archives

While upgrading one of my Zeus ZXTM traffic managers from v5.1r2 to v6.0r4 it crashed on startup. This pretty surprising because the upgrade process appeared to have proceeded without a hitch. Here’s the error I saw in the logs and when I attempted to fire up the zxtm program using SSH:

$ ./start-zeus
Initializing Zeus Application Framework. (C) 1995 - 2010 Zeus Technology Limited
Zeus Administration Server already running: 235
Zeus Traffic Manager - (C) 1995 - 2010 Zeus Technology Limited
Version 6.0r4, Build date: Feb 10 2010 08:32:37
Process permissions set to zeus:zeus
 INFO   Zeus Traffic Manager starting
 INFO   Version 6.0r4, Build date: Feb 10 2010 08:32:37
 FATAL Parent 1234 hit FATAL at Cannot fork:Not enough space
[0x6ac417] function __1cOcommkeyChanged6FpknNConfigSection_rknKStringBase_pknKConfigFile_p6_nIRetValue__ + 0x417
[0x8d4376] function __1cFFATAL6Fpkc1i_v_ + 0x66
[0x6ad2b3] function __1cUreally_nice_shutdown6F_v_ + 0x883
[0x6b3a3b] function __1cKParentBoot6Fpkc_v_ + 0xa8b
[0x5e83c1] function main + 0x571
[0x5c532c] function _start + 0x6c
[0x0] function ?? + 0xffffffffffa3ad40
$

Not enough space? Something was seriously amiss. The solution turned out to be pretty simple, but was not something I could find in the manual.

In v6.0, a new configuration parameter was added: sharedpoolsize. This was not set since I was upgrading from v5.1. The ZXTM made its best guess, it guessed wrong on my virtualized environment and picked a value that exceeded the memory available to my zone. The fix was simple: set sharedpoolsize in $ZEUSHOME/zxtm-6.0r4/conf/settings.cfg to a size small enough to fit into my available memory. Since this configuration did not exist, I mad to add it at the bottom of the file.

Almost a year ago I purchased a wildcard SSL certificate from GoDaddy. When it was first issued I simply loaded it into into my Zeus ZXTM load balancer with the import button.

Everything seemed fine for quite awhile. I visited my web site in Firefox and in all of the flavors of IE. It seemed to work great. That is, until, someone called me to let me know that Safari was not accepting the certificate! I thought I was in trouble until I googled around and found many blog entries about the root cause. It turns out that my server, or in this case my Zeus ZXTM load balancer, was not configured to display the whole certificate chain back to the root authority.

This makes me wonder how this worked at all in every other web browser. Perhaps this is such a common problem that the other browsers hack around it?

Anyway, the fix was easy enough but the terminology was different. Rather than an SSLCertificateChainFile, my ZXTM called it an ‘Intermediate Certificate’. One click of a button, browsing to gd_bundle.crt (provided with my original certificate), and it was loaded up and the issue was fixed.

I attempted to set up an SSH tunnel today to access a web application that was only bound to a private IP. I tunneled to my opensolaris box, which worked fine, and then attempted to establish a connection with telnet. Telnet reported that the connection was refused and I saw this on my command line for the tunnel:

$ ssh johndoe@foo.bar.com -L 8080:foo.bar.com:8080 -N
channel 2: open failed: administratively prohibited: open failed

I made a pretty silly mistake, but I figure it may save someone’s time to post it here. Of course the connection was refused. My ssh tunnel attempts to establish the connection to the public domain name, which uses the public IP (the same one I’m SSHing into). The web server I’m trying to reach is not bound to that adapter!

Here’s a correct command:

$ ssh johndoe@foo.bar.com -L 8080:localhost:8080 -N