So, I decided to make the dashboard authenticate against Active Directory.  It took a surprising amount of time.

The first trick was to find the authoritative config for the ruby-cas YAML with a hash of authenticators (the file /etc/puppetlabs/console-auth/cas_client_config.yml is fairly straightforward):

# /etc/puppetlabs/rubycas-server/config.yml
<snip>
authenticator:
  - class: CASServer::Authenticators::SQLEncrypted
    database:
      reconnect: true
      adapter: mysql
      database: console_auth
      username: console_auth
      password: ‘xxxxxxxx’
    user_table: users
    username_column: username
  - class: CASServer::Authenticators::ActiveDirectoryLDAP
    ldap:
        host: 192.168.0.6
        port: 389
        base: dc=puppet-ad,dc=spence,dc=org,dc=uk,dc=local
        filter: (&(objectClass=person)(memberof=CN=puppetconsoleaccess,OU=Groups,DC=puppet-ad,DC=spence,DC=org,DC=uk,DC=local))
        auth_user: cn=ldapbind,cn=users,dc=puppet-ad,dc=spence,dc=org,dc=uk,dc=local
        auth_password: xxxxxxxx
</snip>

The other trick is to make sure the filter actually works.  I think somehow I managed to hose the default filter (objectClass=person), which caused all sorts of aggro.  The debugging info from the rubycas is awful - I resorted to running a tcpdump -i eth0 tcp port 389 -X to see the messages I was getting from the Active Directory, and the authentication method doesn’t cause any access denied errors in the event log at all on a Windows 2003 DC (which is irritating).

The rubycas doesn’t really help you when the YAML is broken either - using irb helped me here to validate that at least the YAML is sane:

irb(main):001:0> require ‘yaml’
=> true
irb(main):002:0> YAML.load_file ‘/etc/puppetlabs/rubycas-server/config.yml’
=> {“maximum_session_lifetime”=>172800, “<snip>

One thing I would have found useful is the option to configure access levels via active directory rather than having to add an AD user, then configure access level in dashboard.

Don’t forget to edit the console-auth/cas_client_config.yml file:


authorization:
  local:
    default_role: read-only
    description: Local
  activedirectoryldap:
    default_role: read-only
    description: Active Directory


So, to login just use the bare username in AD:


Which translates to:

Some time ago, I wrote a backend for Hiera to provide another option for encrypting/decrypting data at rest.  It exists in the same kind of problem space as hiera-gpg, but with the additional downside that you don’t quite have the flexibility that you do with GPG keyrings.  It does have the advantage that it just uses Puppet SSL certs, so there aren’t any external requirements for e.g. GPG.  I always planned on having a specific SSL keypair to do the work so as not to compromise host keys.

That code is here: https://github.com/fiddyspence/hiera-puppetcert

It’s quite well integrated - there is a Puppet face to encrypt the YAML data that hiera will query, and the backend will decrypt that data so you get the privacy of your data in your VCS/backup systems etc.  What it lacked was flexibility in terms of the implementation.

What I really wanted to be able to do was to reuse the encryption routines in a portable way to do encryption/decryption using a library but across different bits of Puppet, but this implementation sucked for that.   The aim was to be able to reasonably sensibly implement fact encryption, so sensitive data can get passed around and only be decrypted in memory but held reasonably safely at rest.

I realised last week, after implementing some mcollective magic to do config file hacking (http://ibroketheinternet.co.uk/blog/2012/12/01/mcollective-config-file-hackage/) that I could reimplement the functions into a Puppet::Util class(thanks to cprice404 for the inspiration), and use them wherever I wanted to.  The advantage of this is that you get to pluginsync the Util class, and Puppet will be able to load it.



[root@puppet certencryption]# tree
.
├── lib
│   └── puppet
│       └── util
│           └── certencryption.rb



Admittedly, the implementation of the actual encryption is also a bit hacky, but that’s relatively easy to change to something better (on the roadmap).

So, I give you https://github.com/fiddyspence/puppet-certencryption or Puppet::Util::Certencryption as it’s better known.

Watch this space for the fact encryption code - should be along shortly.

Technical news this week:  I was inspired by a fellow engineer to create a solution to edit config files out of band to the configuration management platform of choice (Puppet) using the orchestration tool of choice (mcollective).

The problem was updating the configuration file for Puppet when Puppet is set to not make any changes during a puppet run - i.e. there a ‘noop = true’ line in the puppet.conf.  Using Puppet to fix this problem is a non-starter.

Thusly did I create 2 mcollective agents - a specific solution to the noop problem (that reports on, enables or disables noop in the puppet.conf) which is called puppetnoop, and another general purpose agent (puppetconf) that can change any setting in any section of the config file.  The implementation is built upon cprice404’s ini_file Puppet Util class which actually updates the configuration file.  Note, that the agent could be extremely easily extended to update programmatically any file that conforms to the ini file format by adding another argument to the agent that indicates which file to edit (by default either agent will check for Puppet[‘config’] which should be the puppet.conf your particular version of Puppet uses.

The only requirement for the agent to work is that the RUBYLIB of the mcollective agent needs to include the cprice404 ini_file utility libraries, which if you’re using Puppet should be auto distributed (on the Ubuntu agent I tested on I had to do some hacking to make sure this was the case).

The code can be found at puppetnoop on github

I do quite a significant amount of travel for work.  This means lots of time spent on planes, in airports and hanging around in hotels.  This has given me a newfound depth of hatred for stuff I never knew annoyed me.   I have decided to write about some of these things.

I loathe the….

• … hopelessly bewildered

Sometimes funny ideas just occur to me.  One such hit me yesterday.  I was wandering around London taking break from a customer visit, and I started thinking about wardriving and finding open wifi.  I have kind of wardriven myself - travelling around South-East Asia on a shoestring looking for bandwidth makes you do that sort of stuff.  Sometimes it transpires that folks wardriving look for open wifi just to see what’s there.

It occurred to me that one could do the reverse.  You see, I am fundamentally lazy (it’s why I think I make a reasonable sysadmin - my goal is to always make myself replaceable so I can do other, more interesting things).  Thus today, I am going to take the credit for inventing Warcouching.

The principle here is that I am interested in seeing what devices wander past my flat that promiscuously associate themselves with a wholly private wireless VLAN using a commonly known SSID, watching the DHCP logs and having a look at what connects.

So, a teensy bit of ruby watches the messages file on the dhcp server triggers an nmap scan of the IP address of the device that connected and captures the log:


<snip>
      if runcommand
        begin
          system(“/bin/nmap -O -vvv #{$log[loopstart.to_int].split(’ ‘)[7]} >> #{$mylog}”)
        rescue => e
          puts “#{e.message}”
        end
        runcommand = false
      end
</snip>      


and thusly do we see in the logfile:


Starting Nmap 6.00 ( http://nmap.org ) at 2012-11-13 13:28 GMT
Initiating ARP Ping Scan at 13:28
Scanning 192.168.0.54 [1 port]
Completed ARP Ping Scan at 13:28, 0.15s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 13:28
Completed Parallel DNS resolution of 1 host. at 13:28, 6.50s elapsed
DNS resolution of 1 IPs took 6.50s. Mode: Async [#: 2, OK: 1, NX: 0, DR: 0, SF: 0, TR: 3, CN: 0]
Initiating SYN Stealth Scan at 13:28

Scanning dhcp-54.spence.org.uk.local (192.168.0.54) [1000 ports]
…


Warcouching - watching passing strangers automagically.  It’ll be fashionable, I tell you.

A short thought today.

My bank invites you on their homepage to send them a tweet in the event that you’ve got any problems.  I assume this is to help in in the eventuality that you can’t pick up the phone, or use their interwebnet banking service.

Their website today, and a couple of times over the last couple of days has been and is now a bit broken - it displays HTML source rather than actually rendering the site.

My experiment with social media triggered the ‘must tweet them’ library in my brain, and was about to but then thought the following things:

1/ I want to tell them their site is broken,
2/ I can’t be arsed to phone them, but I can’t use their website to tell them,
3/ I could tweet them, but then I’d be telling the world that I’m a <bankname> customer,
4/ I should expect abuse for telling the world where I keep all my money if I did this,
5/ I should probably go get a cup of coffee and wait for someone else to fix it.

I checked the twitter briefly, and I now know that if I want to steal money from 3 randoms on the interwebs at least where to start trying to tickle their current account.

Banks - please don’t encourage people to advertise where they keep their money publicly on the interwebs - it’s probably not wise.

One of the big topics that keeps coming up again and again with the configuration management space is how to deal with configurations that span multiple nodes. Typically these are OS instances, but the requirement is almost always to update inter-dependent configurations (even when on the same host).

A typical example of this would be a service requiring an application instance such as a Tomcat container, a web front end to the application, maybe a database server and possibly a load balancer in front of it all.

This requirement manifests itself in needing to do orchestrated configurations of one or more nodes which may then need to be installed or updated in a particular sequence in order to correctly configure the service that they should offer.  In our example, probably get the databases going, then the Tomcat, then the web front end(s) and based on all that configuration configure the load balancer.  If a config changes for one part of the stack, then you will probably want changes to other parts to flow through your infrastructure in the right order and probably soon, too!

Consider the following Puppet code for a vhost in our application:

file { ‘/etc/httpd/conf.d/vhost_for_my_custom_domain.conf’:
  ensure  => present,
  content => template(‘apachemodule/vhost.conf.erb’)
}

When that file changes on my webserver, I want the config on my load balancer to update.  I can probably use an exported resource within the Apache class that resource lives in to pass the config to apply on my load balancer node later.  However when my original node changes I don’t want to sit around and wait for the load balancer to check in and update it’s configuration - that might be 30 minutes time - I want my configuration to update now dammit!

One way of doing this would be to wrap the change up in some external orchestration - my tool of choice would be to do some ordered mcollective magic - and run the nodes in order somehow (manually or scriptomagically).

 What I really want to achieve is a way of Puppet doing this for me transparently so I don’t need to worry about external orchestration. So using some resource tagging sauce I can now drive mcollective at the end of a Puppet run from the Puppet master.  The runs are filtered appropriately using the data the resources are tagged with so I don’t need to worry too much about having every single one of my nodes check in at once (thundering herd ahoy!).

The code is here: Puppet mcollective notify report processor

To use it, I need a working mcollective install - my target nodes need to have the mcollective server and Puppet on them. I also need to add some data tags to the resources to trigger the run from the report processor:

file { ‘/etc/httpd/conf.d/vhost_for_my_custom_domain.conf’:
  ensure  => present,
  content => template(‘apachemodule/vhost.conf.erb’),
  tag     => [‘mconotify–class–loadbalancer’],
}

If this resource changes (successfully, and not in noop mode), the report processor will trigger an mcollective run for all nodes in my configuration that have the Puppet class ‘loadbalancer’ applied to them. Magic!

I need to configure the processor with some mcollective configuration so it can authenticate and send RPC messages appropriately but that only requires a sensibly formatted configuration file.  In this way we cut down the volume of external dependencies to get our orchestration going.

I think this method is pretty cool - it uses pure Puppet, although I’d like to not have to use tags to provide the data to the report processor (that would mean a meta-parameter in it’s own right - which isn’t a massive deal but it would require a change to Puppet core).  Throw away your shell scripts!


If I look at the debug output for a Puppet run containing tagged resources, I can see what the processor is doing:

Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: matched mconotify–node–debian–1 to a node

Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: Filters: node 2 class 1

Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: Doing an mco run for ibroketheinternet.co.uk,debian

Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: /ibroketheinternet.co.uk|debian/

Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: Doing an mco run for loadbalancer

I was having a rummage around some old code that I wrote in a previous life the other day, and encountered my old chum ‘wincheck’.  Most, if not all of you will not have heard of wincheck, but I like to think it was pretty groovy in two quite separate ways.

Way the first - it checked common definable misconfigurations and/or configuration items to validate whether they were right or not.  For example, it was marginally pluggable in that it referenced files containing data that referenced specific configuration instances (such as a Windows service, or a package) to validate whether it was installed/not installed or running/not running and more importantly whether that was the right thing to be doing.

Looking at the code, it did some other cool stuff too.  Mostly it was WMI driven, so you could run your check against the local machine or some other arbitrary host on the network, assuming you had rights to read the WMI instances on the node.  The report was human readable, it output some HTML in a handy browser tab so you could consume it - with a small amount of tweaking it could have collated those reports or run unattended and collected all sorts of useful information.

The way the second it was pretty groovy is a sad indictment of why people give up on tools or their development.  Wincheck was pretty rudimentary, though it could have (with more development effort) been really useful.  No doubt it might have evolved into a tool that as well as auditing and reporting could have remediated.  It might have ended up being rewritten in something better and more efficient than vbscript.

Sadly, wincheck serves, for me, as a reminder that even though something has already been invented, you don’t win prizes if people don’t either deign to acknowledge a tool’s existence, or aren’t aware of it.  In either of those two cases, when the organisation you are working for runs a clever ideas competition, the prize (it happened to be a laptop, if I recall correctly) for best cleverest idea (which happened to invent a configuration auditing tool) doesn’t go to one of the direct reports of the person giving the award, working on a project sponsored by that person.  Yeah - that still pisses me off.

Thus, for lack of interest, wincheck has been dormant in a tar file for 8 years, with the last commit dated 21st June 2004.  I might upload it to the github just for fun.