Friday 19 September 2008

Using Splunk's iplocation search command behind a proxy

Splunk has an iplocation search command that will add City and Country fields to your search results. It does this by looking up the IP addresses it finds using the hostip.info API. Unfortunately if your Splunk server doesn't have direct Internet access then this script will fail.

The script itself is a very simple Python script that use the module urllib.urlopen to make the API call. To get it to use your proxy server is easy.

Make a backup of the original script:

$ cd $SPLUNK_HOME/etc/searchscripts
$ cp iplocation.py iplocation.py.bak


Edit iplocation.py and add the following line below the LOCATION_URL definition:

PROXIES = {'http':'http://proxy.example.com:8080'}

Then find the line that reads:

location = urllib.urlopen( LOCATION_URL + ip )

and change it to:

location = urllib.urlopen( LOCATION_URL + ip, proxies=PROXIES )

Then perform your search and pipe it to iplocation. Make sure to limit your search as the script will do a HTTP request for every IP address it finds.

Tuesday 17 June 2008

SUNWapch2u, SUNWPython, and mod_python mismatched expat issue

It is fantastic that Sun have shipped a recent version of Python with Solaris 10 8/07. Unfortunately it seems the SUNWapch2u package builders aren't talking to the SUNWPython package builders which has resulted in the well documented “Expat Causing Apache Crash” issue when you try to build mod_python linked against SUNWapch2u and SUNWPython.

The mismatched expat versions used by each are:
  • SUNWapch2u: expat 1.95.2
  • SUNWPython: expat 1.95.8
  • SUNWlexpt: expat 1.95.7 (neither apache or python use this but it is there just to add to the confusion)
In order to build a working mod_python you will have to compile your own copy of expat 1.95.8 as well as your own apache2 ensuring that you pass the “--with-expat=...” option to apache2's configure script.

I've opened a case with Sun to see if they'll fix this. Watch this blog for updates.

Update 2007.11.13: Changed SUNWapache2 to correct package name SUNWapch2u.

Update 2008.06.17:

With the release of the following Solaris 10 patches the issue describe above has been resolved:
  • 120543-11 SunOS 5.10: Apache 2 Patch fixes Bug ID 6630259 "If Python and Apache 2 are used together with libexpat, httpd crashes".
  • 137147-04 SunOS 5.10: libexpat patch updates libexpat to version 2.0.x.
  • 121606-03 GNOME 2.6.0: Python patch fixes Bug ID 6630230 "Link Python dynamically to /usr/sfw/lib/libexpat.so".
This combination of patches allows you to build your own mod_python linked against Sun's apache2 and python 2.4. If you can wait for Bug ID 6630237 "Supply mod_python with Apache 2" to be delivered then you won't even have to do that!

Wednesday 11 June 2008

IPv4 Exhaustion Counter

I've added the IPv4 Exhaustion Counter to my side bar. It uses data provided by Geoff Huston in his IPv4 Address Report. His models currently predict that IANA will run out of addresses on 24th January 2011 with the RIR's running out by 11th December 2011.

Wednesday 16 April 2008

Solaris 10 5/08 is out!

The release is also known as Solaris 10 Update 5 and you can download the installation DVD from here. I'm still trying to track down the release notes. I'll post an update when I find them.

Update: Here is where you can find out what's new.

Tuesday 1 April 2008

One rule to bind them - part III

Apologies for the length of time between this article and the last in the series. Home and work life has been hectic and hasn't left much time for writing.

In the previous article in the series I wrote a how-to on DHCP booting a Solaris installation and passing custom DHCP options. In this article I'll conclude the series and explain what you can do with this information to make your installation hands-free.

What does a begin script do?

A begin script allows you to execute a series of commands before the installation commences, such as backing up files before an upgrade or creating a derived jumpstart profile. I'll be discussing the latter.

In my jumpstart environment a build list is generated which provides the minimum information about each Solaris build I maintain. It looks like this:
DNS:5.10:RNET:Domain Name Server
FIREWALL:5.9:CORE:Firewall-1
CORE:5.10:CORE:Hardened CORE
CORE:5.9:CORE:Hardened CORE
DEVELOPER:5.10:DEVELOPER:Hardened DEVELOPER
DEVELOPER:5.9:DEVELOPER:Hardened DEVELOPER
ENTIRE:5.10:ENTIRE:Hardened ENTIRE
ENTIRE:5.9:ENTIRE:Hardened ENTIRE
MINIMIZED:5.10:CORE:Hardened Minimized
MINIMIZED:5.9:CORE:Hardened Minimized
The first column corresponds to the JSBuild DHCP option that is sent to the client. The second column is the Solaris version. The third column is the Solaris metacluster: Core, User, Developer, Entire, Entire OEM. The fourth column is a short description of the build.

As the name of the article implies there is only one rule therefore only one begin script. This script does all the heavy lifting required to generate the derived jumpstart profile on a per build basis.

Determining what to build

The first thing the script does is extract the JSBuild DHCP option which tells it which build it is installing. If the JSBuild DHCP option empty (due to incorrect DHCP server configuration or use of old style bootparams installation) then the begin script falls back to the build list above and generates an interactive menu for the user to choose a build from.

Deriving the profile

Now the system knows what it is, it needs to know how to install it. This is what the profile does. The begin script tests whether it has to perform an initial installation or flash archive installation by NFS mounting a directory from the jumpstart server and looking for a flash archive. I have an automated process that builds and patches flash archives for each Solaris metacluster (more about this in a future article). If no flash archive is present then the script falls back to an initial install.

The initial install logic looks for a metacluster include file which defines all of the packages to install. The include file looks like:
#!/bin/sh
#
# Minimal Solaris 10 build
# Start with the minimal required number of packages
echo "cluster SUNWCreq" >> ${SI_PROFILE}

# For performance monitoring
echo "cluster SUNWCacc add" >> ${SI_PROFILE}

# To support Compression (gzip)
echo "cluster SUNWCfwcmp add" >> ${SI_PROFILE}

# To support the Network Time Protocol
echo "cluster SUNWCntp add" >> ${SI_PROFILE}
...
For Solaris 10 initial installations the script also outputs a profile patch keyword like:
patch patch_order nfs://jumpstart/path/to/10_Recommended
Which applies the recommended patch bundle during jumpstart. For Solaris releases prior to 10 this is done using the JASS install-recommended-patches.fin finish script.

Disk layout and mirroring

The next step is to see if the build has a custom partition configuration. Here is an example partition configuration file:
s0:6144:/:logging:
s1:1024:swap::
s3:4096:/var:logging,nosuid:
s6:free:/srv:zpool:srv
zfs1:free:/opt::srv/opt
zfs2:free:/var/opt::srv/var-opt
This file defines the disk slice, size, file system mount, and options. You'll notice that it caters for creating ZFS file systems as well even though the current 8/07 release of Solaris 10 doesn't. If the custom configuration doesn't exist then it uses a default one.

The begin script (well actually another nawk script) parses the partition configuration and outputs filesys profile keywords for each slice. It also outputs SVM mirroring information for Solaris 10 if a second disk is present with the same geometry as the root disk, again, for older Solaris releases, this is done by a script on first boot.

Preserving information between begin and finish

Once the disk layout has been derived then the begin script has done its job and can exit and hand the rest of the installation over to the installer and finish script(s).

What if you want to use some of the information you just derived in your finish scripts?

You can't just export an environment variable and have it persist from the begin script to the finish script. You have to store the information somewhere that the finish script can read it from. The simplest way to do this is write them out in VAR=value format to a file under /tmp and then source this file at the beginning of your finish script. The information I keep between stages is the build, metacluster, root disk device, mirror disk device, patch cluster release date and whether the system was built from a flash archive or not.

Are we there yet?

Although I have managed to modularize most aspects of a build you can still see the complexity for a build developer in maintaining all of these files. In a future article I'll discuss a new layer on top of JASS that I have developed that consolidates all of this information into a single, easy to maintain configuration file.

Running a DTrace script as an SMF service

Sam Faulkner has put together an SMF manifest that will run a DTrace script and send the output to syslog. Very useful if you wish to DTrace something over a period of time while also ensuring that it is always running.

Thursday 13 March 2008

SMF Tutorial

Joerg Moellenkamp has put together a great five part tutorial on SMF where he explains the concepts, then dives into developing an SMF service for OpenVPN.

Monday 18 February 2008

Solution: Solaris 10 fails to install on EFI labeled disks

I recently came across the following error when jumpstarting a system that has 4 disks, 2 of which were previously in a zpool and were EFI labeled:
Checking rules.ok file...
awk: division by zero
record number 17
awk: division by zero
record number 15
expr: syntax error
awk: division by zero
record number 17
The error seemed harmless enough in that it didn't affect the installation. Even still I tracked it down to the /usr/sbin/install.d/chkprobe script in the Solaris 10 mini-root. I opened a case with Sun and they informed me it was a known issue (BugID 6457349: chkprobe cannot handle disks with EFI labels). Sun provided me with a work-around patch to chkprobe which produced the following output:
Checking rules.ok file...
c0t8d0 doesn't have a VTOC label
c0t9d0 doesn't have a VTOC label
This was fine in a system that had at least one VTOC labeled disk as the jumpstart installation could still proceed. When the all of the disks are EFI labeled then the installation fails with the message:
ERROR: One or more disks are found, but one of the following problems exists:
- Hardware failure
- The disk(s) available on this system cannot be used to install Solaris Software. They do not have a valid label. If you want to use the disk(s) for the install, use format(1M) to label the disk and restart the installation.
Solaris installation program exited.
To solve this you need to run the format -e command and re-label the disks. Note the "-e" (expert mode) option to format is required otherwise you won't be given the choice of label types.
# format -e
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c0t0d0 <FUJITSU-MAP3367N SUN36G-0401-33.92GB>
/pci@1c,600000/scsi@2/sd@0,0
1. c0t1d0 <FUJITSU-MAP3367N SUN36G-0401-33.92GB>
/pci@1c,600000/scsi@2/sd@1,0
Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted]


FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
inquiry - show vendor, product and revision
scsi - independent SCSI mode selects
cache - enable, disable or query SCSI disk cache
volname - set 8-character volume name
! - execute , then return
quit
format> label
[0] SMI Label
[1] EFI Label
Specify Label type[1]: 0
Warning: This disk has an EFI label. Changing to SMI label will erase all
current partitions.
Continue? y
Auto configuration via format.dat[no]? y
format> quit
The fix Sun provided should make it into a future Solaris 10 update and I suspect once ZFS boot is released this problem will be resolved for good.