Wednesday 25 November 2009

Package Installation Do's and Don'ts

An important note from Gerry Haskins: Do not apply packages from one Update onto a system installed with a different Update.

Cherry picking packages from a newer Solaris 10 update and installing them on a system running an older update will result in an unsupported configuration and likely lead to system corruption.

Note also that adding a package from the same update means that you will have to re-apply all the patches already on the system to bring the new package to the same level as the rest of the system.

Tuesday 17 November 2009

Behaviour Driven Infrastructure

I've been following the development of puppet for many years and this gem of a thread caught my attention recently. Martin Englund asks the Puppet Users mailing list:
how do you validate that puppet has done what it is supposed to, and even troublesome, how you validate that it has done what you intended it to do?
This is something I've struggled with over the years with my JASS/SST-based jumpstart build system. I've gone so far as to automate the build testing process using buildbot and pass or fail a build by using regexps to search for errors in the installation process. But my testing ends when the console login prompt appears. Validating whether a system build functions as intended is beyond what I currently test now and even beyond the capabilities of puppet.

This is where Martin's Behaviour Driven Infrastructure (BDI) approach comes into play. Martin is using Cucumber, a Behaviour Driven Development testing tool, to describe a system's behaviour using natural language that is readable and easy to understand by non-technical users (i.e. your IT helpdesk or even your business stakeholders).

Where this really gets interesting is combining puppet, cucumber and a monitoring system such as Nagios to do Test Driven Infrastructure. For example, you can use cucumber-nagios to integrate your cucumber tests with Nagios, then write a test for a new feature you want the system to have, e.g. "Should be able to send email". Initially this would result in Nagios marking the system as having a fault because you haven't yet implemented the feature that passes the test. You then proceed to implement the feature using puppet such that the test passes.

Over time I would imagine the amount of test coverage to grow such that system behaviours like DNS resolution, LDAP authentication, host based firewall policies etc would be tested. Any change to the system that broke one of these tests could be quickly pinpointed and fixed.

A perfect example in the Solaris world is the application of the latest recommended patches to a box. At a minimum it tends to break the sendmail and snmpd configs on my systems and I know to manually backup and restore these files before and after applying the patches. With BDI combined with a monitoring system you could be alerted to these breakages or any others that you weren't previously aware of and rapidly respond to them.

Brilliant!

Thursday 8 October 2009

Speaking of Solaris 10 Update 8...

You can now read about What's New.

Quite a few ZFS changes including Flash Archive support integrated into installer, cache devices, and a bunch of new properties that breakdown space usage by child dataset, snapshot, etc.

Solaris 10 Kernel PatchID Sequence - Patch Corner

Watch this page Solaris 10 Kernel PatchID Sequence - Patch Corner. It is regularly updated with the Solaris 10 kernel patch IDs as well as the sustaining patch IDs. Solaris 10 Updates 8 and 9 have just been added.

Sunday 16 August 2009

Improvements to Solaris 10 Recommended and Sun Alert Patch Clusters released

Sun have made quite a significant revamp to the Solaris 10 Recommended and Sun Alert patch clusters.

  • Filtering out "false negatives" from the patch utility return codes.
  • The new 'installcluster' script will exit as soon as it encounters an unexpected failure.
  • The new 'installcluster' script includes context intelligence for patching operations.
  • The new 'installcluster' script provides better integration with Solaris Live Upgrade.
  • The new 'installcluster' script performs space checking prior to installing each patch, and will halt if it believes there is insufficient space to complete the installation successfully.
  • The messages and log files produced by the 'installcluster' script are clear and well structured. For example, a "failed" log is created if a patch fails to apply.
  • The 'patch_order' places patches in an optimal order for installation to avoid known issues.
  • The patches have been moved to a 'patches' sub-directory, to de-clutter the top level directory of the unzipped cluster.
  • Please see the cluster README file for further information. Customers should read the cluster README file and look at the Special Install Instructions in the patches within the cluster prior to installation.

They have also improved the QA process so that patch clusters get tested prior to release. Amazing!

Wednesday 12 August 2009

Requiring at least one inline FormSet

Last month I posted an article about my Improved Django FormWizard, well this month I've release a simple subclass of Django's BaseInlineFormSet that demonstrates how you can require a user to enter at least one entry in an inline formset.

After updating to wadofstuff.django.forms 1.1.0 you can use the RequireOneFormSet class as the formset argument to inlineformset_factory().

When the formset is validated and it does not contain one or more entries, then a ValidationError is raised which gets put into formset.non_form_errors. You will need to check this in your templates if you wish to display the error message to your users.

Saturday 25 July 2009

Inlines support for Django generic views

Django's excellent Generic Views provide developers with most of what they need to get a site up and running (if they aren't using the Admin of course). The flexibility of these views is such that for most sites you don't need much else. Extending these views is also well documented and probably covers off 95% of the situations where the plain generic views fall short.

In a recent project I found another area where the generic views fall short: inline formsets.

I have added the ability to use inline formsets in generic views. The new view functions are drop in replacements for Django's with the addition of a new inlines argument.

Installation

From Source

Download wadofstuff-django-views.

To install it, run the following command inside the unpacked source directory:

python setup.py install

From pypi

If you have the Python easy_install utility available, you can
also type the following to download and install in one step:

easy_install wadofstuff-django-views

Or if you're using pip:

pip install wadofstuff-django-views

Or if you'd prefer you can simply place the included wadofstuff
directory somewhere on your Python path, or symlink to it from
somewhere on your Python path; this is useful if you're working from a
Subversion checkout.

Note that this application requires Python 2.4 or later. You can obtain
Python from http://www.python.org/.

Usage

wadofstuff.django.views.create_object(..., inlines=None)
wadofstuff.django.views.update_object(..., inlines=None)

These functions are identical to the Django ones except for the addition of the
inlines argument. This argument consists of a list of dictionaries that will
be passed as arguments after the parent_model argument to
inlineformset_factory(parent_model, ...).

For example, arguments to a generic view might typically look like:
crud_dict = {
'model':Author
'inlines':[{
'model':Book,
'extra':2,
'form':BookForm,
},{
'model':Article,
}],
# ... other generic view arguments
}
would translate to calls to inlineformset_factory() like:

inlineformset_factory(Author, model=Book, extra=2, form=BookForm)

and

inlineformset_factory(Author, model=Article)

The view function will create a formset for each inline model and add them to the template context. In the example above the context variables would be named book_formset and article_formset.

Update: A quick change to allow the views to be imported from wadofstuff.django.views. Bumped version to 1.0.1.

Tuesday 21 July 2009

Atom Feed for SVN Commit Log

Recently the developers of Extjs added a page that opened up their subversion commit log so users could see what was being added/fixed. Unfortunately they decided to only publish it using the output of svn log -v --xml rendered in an Ext.grid.GridPanel.

It is a nice example of what you can do with their framework, but this format is not so friendly to use or keep tabs on so I've created a mashup that converts it to Atom Syndication Format so interested users can subscribe to it in their favourite feed reader.

The URL to subscribe to is http://feeds.feedburner.com/ExtjsSvnCommitLog. I'm hoping the developers publish their own feed soon. If they do then I'll stop mine.

Here is the script I wrote to convert it to Atom. Witha few tweaks it should be usable for any SVN commit log.

My first attempt at this was to use Yahoo! Pipes to try and munge it but I couldn't get it to work. I'd be interested to see if Pipes was capable of doing this.

Since writing it I also found this XSLT script that does something similar.


Update: I've cleaned the script up and made if reusable for any SVN log source. See the documentation for details on how to use it.

Improved Django FormWizard

A few months back I had a project that I thought needed a wizard-style interface for one of its forms. For a while now Django has included the FormWizard class in django.contrib.formtools.wizard so I decided to use that. However, I immediately hit a couple of issues with it.

FormWizard requires you to output the previous_fields context variable in each of the form's step templates. Django's FormWizard previous_fields is just a string of raw HTML. If you are using some other kind of markup to do your forms e.g. XForms, or even javascript widgets from dojo or extjs, then this is no good.

I quickly came up with a fix for this and opened #10557 which has now been slated to be included in Django 1.2.....

Wait a minute! Django 1.1 isn't even out yet!

The second issue was that FormWizard didn't handle FormSets.

Who knows when 1.2 will be out so I've bundled up my code and released it as wadofstuff.django.forms 1.0.0.

From the README:

Functions

security_hash(request, form, exclude=None, *args)

Calculates a security hash for the given Form/FormSet instance.

This creates a list of the form field names/values in a deterministic
order, pickles the result with the SECRET_KEY setting, then takes an md5
hash of that.

Allows a list of form fields to be excluded from the hash calculation. This
is useful form fields that may have their values set programmatically.

Classes

BoundFormWizard

A subclass of Django's FormWizard that adds the following functionality:
  • Renders previous_fields as a list of bound form fields in the template context rather than as raw html.
  • Can handle FormSets.
The usage of this class is identical to that documented here with the exception that when rendering previous_fields you should change your wizard step templates from:

{{ previous_fields|safe }}

to:

{% for f in previous_fields %}{{ f.as_hidden }}{% endfor %}

The latest stable release for the forms module can be obtained by:


Note: The BoundFormWizard class can only handle regular FormSets and not ModelFormSets. I have ModelFormWizard class that I'll be adding to a future release that can handle simple Models.

Monday 13 July 2009

Flash Back

Late last year Solaris 10 Update 6 was released and included the long waited for ZFS root capability. Unfortunately for many customers using Jumpstart to build their systems it also meant they had to stop using Flash Archives as Sun explicitly stated that they were incompatible. Solaris 10 Update 7 didn't fix this either and zfs-discuss forum discussions seemed to indicate it would not be available until Solaris 10 Update 8 and may be available as a patch for earlier releases.

Well the day has come and the miniroot patches are now available to download from SunSolve:

Sparc
x86
My own build environment skipped S10U6 and went to U7 recently and had to put up with the loss of flash archive builds with the knowledge it would be reinstated soon. I'll be testing these patches in the next week and will report back here with the results.

Update 2009.08.20: Sun have posted some preliminary documentation on how to use ZFS and Flash.

Saturday 30 May 2009

New Solaris PatchFinder tool

Anyone who has ever used the existing Solaris patch search tool will know how limited it is. Well Sun have released a much improved search page called Software Update Finder. Gerry Haskins has detailed blog about its new capabilities which include:
  • Filtering results by OS release and architecture.
  • Limiting search results to Security and/or Recommended patches.
  • Searching for patches that fix a particular CR or BugId.
See Gerry's article for more features. This is just the initial release and there are plans to enhance it further in the future. Let Sun know what you want the tool to do.

Tuesday 26 May 2009

Python ipaddr performance

Last weekend while I was cleaning up my IP address summarization script (I added a setup.py, created a Cheese Shop entry and a downloadable archive) I had a look at the state of IP address manipulation in Python and found a new module called ipaddr. What sparked my interest in this module was that it had already been integrated into upcoming Python 2.7 and 3.1 as a standard library.

It seemed to contain all the functionality of IPy but with much cleaner code. It also has a function called collapse_address_list() that is similar to my summarize function but it can handle a list of non-contiguous networks and IP addresses and collapse them down.

When I wrote my summarize() function I spent quite a while optimizing the algorithm so that it can handle large address ranges in a reasonable amount of time. For example, on my development box it can summarize the two worst case ranges: IPv4 0.0.0.0 to 255.255.255.254 in 0.1s and IPv6 :: to ffff:ffff:ffff:ffff:ffff:ffff:ffff:fffe in 0.2s.

I had to see how well ipaddr performed so I wrote a little benchmarking script that summarized a /24. The results were very poor with it taking 64.27 seconds to perform 1000 runs. I then ported my summarize() code to use ipaddr and modified collapse_address_list() to use it. The result was a 50 times improvement in performance (1000 runs took 1.21 seconds).

ipaddr is developed by Google and the coders use Reitveld for reviewing patches. So I've submitted my patches there. Feel free to review and comment on the code.

Here's hoping that they are accepted and benefit everyone by making it into the next release(s) of Python!

Sunday 24 May 2009

Django Serializer Updates

I've had a couple of emails and forum posts where users were having difficulty with my serialization module. The problems mostly centered around installing it correctly.

I did a little work tonight to clean up the installation side of things. As a result you can now find the module in the Cheese Shop.

The latest stable release for the serialization module can also be obtained by:
The source tree has been moved around a little to accommodate these changes and can be found here.

There were no new features or bug fixes made but I have a few interesting tickets from users that I'll be reviewing over the coming weeks.

Wednesday 29 April 2009

Solaris 10 5/09 (Update 7) is out...

...well almost. The DVD ISO isn't available yet for download but you can check out What's New and the Release Notes on docs.sun.com while you wait.

From my reading so far there doesn't seem to be many exciting new features but the Solaris 10 5/09 Patch List at least lists some patches that I'll no longer have the pain of applying to the new miniroot boot archives.

Monday 2 March 2009

Full Django Serializers - Part II

In the first part of this article I covered the excludes and extras options to the Wad of Stuff serializer. In this article I introduce the relations option.

Relations

The Wad of Stuff serializers allow you to follow and serialize related fields of a model to any depth you wish. This is why it is considered a "full serializer" as opposed to Django's built-in serializers that only return the related fields primary key value.

When using the relations argument to the serializer you may specify either a list of fields to be serialized or a dictionary of key/value pairs. The dictionary keys are the field names of the related fields to be serialized and the values are the arguments to pass to the serializer when processing that related field.

This first example shows the simplest way of serializing a related field. The Group model has a ManyToManyField to the Permission model. In this case there is only one permission but if there were more then each would be serialized.
>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations=('permissions',))
[
{
"pk": 2,
"model": "auth.group",
"fields": {
"name": "session",
"permissions": [
{
"pk": 19,
"model": "auth.permission",
"fields": {
"codename": "add_session",
"name": "Can add session",
"content_type": 7
}
}
]
}
}
]
The simple case may be all you need but if you want more control over exactly which fields or extras are included, excluded, and the depth of relations to follow then you need to pass a dictionary in the relations option. This dictionary is a series of nested dictionaries that are unrolled and passed as arguments when serializing each related field.
>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'fields':('codename',)}})
[
{
"pk": 2,
"model": "auth.group",
"fields": {
"name": "session",
"permissions": [
{
"pk": 19,
"model": "auth.permission",
"fields": {
"codename": "add_session"
}
}
]
}
}
]
The relations option in this example roughly translates to a call to serialize('json', permissions_queryset, fields=('codename',)) when the permissions field is serialized.

Serializing deeper relations

The power of the relations option becomes obvious when you see it in action serializing related fields that are 2 or more levels deep. Below the content_type ForeignKey field on the Permission model is also serialized.
>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'relations':('content_type',)}})
[
{
"pk": 2,
"model": "auth.group",
"fields": {
"name": "session",
"permissions": [
{
"pk": 19,
"model": "auth.permission",
"fields": {
"codename": "add_session",
"name": "Can add session",
"content_type": {
"pk": 7,
"model": "contenttypes.contenttype",
"fields": {
"model": "session",
"name": "session",
"app_label": "sessions"
}
}
}
}
]
}
}
]

Combining options

You may also combine the other options when serializing related fields. In the example below I am excluding the content_type.app_label field from being serialized.
>>> print serializers.serialize('json', Group.objects.all(), indent=4, relations={'permissions':{'relations':{'content_type':{'excludes':('app_label',)}}}})
[
{
"pk": 2,
"model": "auth.group",
"fields": {
"name": "session",
"permissions": [
{
"pk": 19,
"model": "auth.permission",
"fields": {
"codename": "add_session",
"name": "Can add session",
"content_type": {
"pk": 7,
"model": "contenttypes.contenttype",
"fields": {
"model": "session",
"name": "session"
}
}
}
}
]
}
}
]
That wraps up this series of articles. Head over to Wad of Stuff at Google Code to grab the source and don't hesitate to open a ticket if you find any bugs or have any enhancement requests.

Friday 27 February 2009

Feed URL change

Quite a while back the feed URL for this blog changed back to the default Blogger one. I've been using FeedBurner from day one but Blogger initially didn't have the ability to redirect to a different URL for your feed.

I've noticed that few subscribers are still directly using the old FeedBurner URL. When you find the time please update your reader to use the canonical URL:

http://wadofstuff.blogspot.com/feeds/posts/default

I don't expect the old one to go away any time soon but better to be safe than sorry.

Django Full Serializers - Part I

Introduction

The wadofstuff.django.serializers python module extends Django's built-in serializers, adding 3 new capabilities inspired by the Ruby on Rails JSON serializer. These parameters allow the developer more control over how their models are serialized. The additional capabilities are:
  • excludes - a list of fields to be excluded from serialization. The excludes list takes precedence over the fields argument.
  • extras - a list of non-model field properties or callables to be serialized.
  • relations - a list or dictionary of model related fields to be followed and serialized.
De/Serialization Formats

At the moment the module only supports serializing to JSON and Python. It will also only deserialize data that is in the original Django format. i.e. it won't deserialize the results of using the excludes, extras, or relations options.

Source

The source for the serialization module can be obtained here.

Examples

Project Settings

You must add the following to your project's settings.py to be able to use the JSON serializer.
SERIALIZATION_MODULES = {
'json': 'wadofstuff.django.serializers.json'
}
Backwards Compatibility

The Wad of Stuff serializers are 100% compatible with the Django serializers when serializing a model.
>>> from django.contrib.auth.models import Group
>>> from django.core import serializers
>>> print serializers.serialize('json', Group.objects.all(), indent=4)
[
{
"pk": 2,
"model": "auth.group",
"fields": {
"name": "session",
"permissions": [
19
]
}
}
]
Excludes
>>> print serializers.serialize('json', Group.objects.all(), indent=4, excludes=('permissions',))
[
{
"pk": 2,
"model": "auth.group",
"fields": {
"name": "session"
}
}
]
Extras

The extras option allows the developer to serialize properties of a model that are not fields. These properties may be almost any standard python attribute or method. The only limitation being that you may only serialize methods that do not require any arguments.

For demonstration purposes in this example I monkey patch the Group model to have a get_absolute_url() method.
>>> def get_absolute_url(self):
... return u'/group/%s' % self.name
...
>>> Group.get_absolute_url = get_absolute_url
>>> print serializers.serialize('json', Group.objects.all(), indent=4, extras=('__unicode__','get_absolute_url'))
[
{
"pk": 2,
"model": "auth.group",
"extras": {
"get_absolute_url": "/group/session",
"__unicode__": "session"
},
"fields": {
"name": "session",
"permissions": [
19
]
}
}
]
Stay tuned for the second part of this article where I demonstrate the ability to serialize related fields such as ForeignKeys and ManyToManys.

Tuesday 10 February 2009

Django: DRY Custom Model Forms and Fields

With the release of Django 1.0, the ability to add a list of validators to a model field was removed. This was replaced with the clean*() methods in the new forms classes. Unfortunately when you write your own form subclasses you may end up repeating a lot of the parameters in your form field definitions that you had already declared in your model fields such as max_length, required, help_text, etc. So in the spirit of DRY I worked out a way to avoid this.

A few months back I was updating an old Django site to use new the latest 1.0 release and needed to re-implement the functionality of the following, slightly contrived, pre-1.0 model:
from django.db import models
from django.core.validators import MatchesRegularExpression, isNotOnlyDigits

class Acronym(models.Model):
acronym = models.CharField(maxlength=3, unique=True, db_index=True,
validator_list=[isNotOnlyDigits,
MatchesRegularExpression(r'^[A-Z0-9]+$',
error_message='This field may only contain uppercase letters and' \
' numbers.')],
help_text='A three letter acronym.')
definition = models.CharField(maxlength=64)
The new model looked like this after I fixed up all of the parameters:
from django.db import models

class Acronym(models.Model):
acronym = models.CharField(max_length=3, unique=True, db_index=True,
help_text='A three letter acronym.')
definition = models.CharField(max_length=64)
The problem above is that the extra validation is now not being done and needs to be reimplemented either in a new form or form field class.

My first attempt probably looked like this:
import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class NaiveAcronymForm(forms.ModelForm):
acronym = forms.RegexField(TLA_RE, max_length=3, required=True,
label='Code', help_text='A three letter acronym.')
class Meta:
model = Acronym
The amount of repetition is already becoming obvious and the code above only provides half the solution. We also need to implement the isNotOnlyDigits validation.
import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class NaiveAcronymForm(forms.ModelForm):
acronym = forms.RegexField(TLA_RE, max_length=3, required=True,
label='Code', help_text='A three letter acronym.')

def clean_acronym(self):
if self.cleaned_data['acronym'].isdigit():
raise forms.ValidationError(u"This value can't be comprised soley of digits."

class Meta:
model = Acronym
I got to this point in my own code and thought that there must be a way to leverage all the work already being done by Django to generate a form field from a model field. When I make a change to my model fields I want those changes to automatically flow on to my forms.

The solution I came up with was this:
import re
from django import forms
from models import Acronym

TLA_RE = re.compile(r'^[A-Z0-9]+$')

class AcronymField(forms.RegexField):
"""Form field for three letter acronym."""
default_error_messages = {
'invalid': u'This field may only contain uppercase letters and ' \
'numbers.',
'notonlydigits': u'''This value can't be comprised solely of digits.'''
}

def __init__(self, *args, **kwargs):
"""Initialize the field with the acronym regex."""
super(AcronymField, self).__init__(TLA_RE, min_length=3, *args,
**kwargs)

def clean(self, value):
"""Ensure acronym matches regex and is not only digits."""
value = super(AcronymField, self).clean(value)
if value.isdigit():
raise forms.ValidationError(self.error_messages['notonlydigits'])
return value

class AcronymModelForm(forms.ModelForm):
"""Form for Acronym model."""
def __init__(self, *args, **kwargs):
"""Programmatically declare fields."""
super(AcronymModelForm, self).__init__(*args, **kwargs)
field = self.Meta.model._meta.get_field('acronym')
self.fields['acronym'] = field.formfield(form_class=AcronymField)

class Meta:
model = Acronym
How does this work? First of all, AcronymField is subclassed from forms.RegexField and the __init__() method overriden. It then calls the parent class's __init__() with my extra parameters, the regexp and min_length, as well as passing the original positional and keyword arguments. This ensures that any extra parameters from the model such as required, label, max_length, and help_text are preserved. I also override the clean() method and again call the superclass's clean() method before checking that the value is not only digits.

Next I subclass AcronymForm from forms.ModelForm and override the __init__() method again. Here the superclass's __init__() method is called then the acronym form field is replaced with one that uses the new AcronymField class. It is within the call to field.formfield(...) (see django.db.models.fields.Field class for the gory details) that all the values for required, label, and help_text are taken from the model field and the form field instance created.

While this results in more lines of code for a model form definition it ultimately results in less presentation and validation errors due to model fields and form fields getting out of sync. This approach could probably be generalized even further to a subclass of ModelForm but I've only had to use it for a handful of models so I leave that as an exercise for the reader.

Update: fixed formatting mistake so you can more clearly see the before and after models.