One of the big topics that keeps coming up again and again with the configuration management space is how to deal with configurations that span multiple nodes. Typically these are OS instances, but the requirement is almost always to update inter-dependent configurations (even when on the same host).
A typical example of this would be a service requiring an application instance such as a Tomcat container, a web front end to the application, maybe a database server and possibly a load balancer in front of it all.
This requirement manifests itself in needing to do orchestrated configurations of one or more nodes which may then need to be installed or updated in a particular sequence in order to correctly configure the service that they should offer. In our example, probably get the databases going, then the Tomcat, then the web front end(s) and based on all that configuration configure the load balancer. If a config changes for one part of the stack, then you will probably want changes to other parts to flow through your infrastructure in the right order and probably soon, too!
Consider the following Puppet code for a vhost in our application:
file { ‘/etc/httpd/conf.d/vhost_for_my_custom_domain.conf’:
ensure => present,
content => template(‘apachemodule/vhost.conf.erb’)
}
When that file changes on my webserver, I want the config on my load balancer to update. I can probably use an exported resource within the Apache class that resource lives in to pass the config to apply on my load balancer node later. However when my original node changes I don’t want to sit around and wait for the load balancer to check in and update it’s configuration - that might be 30 minutes time - I want my configuration to update now dammit!
One way of doing this would be to wrap the change up in some external orchestration - my tool of choice would be to do some ordered mcollective magic - and run the nodes in order somehow (manually or scriptomagically).
What I really want to achieve is a way of Puppet doing this for me transparently so I don’t need to worry about external orchestration. So using some resource tagging sauce I can now drive mcollective at the end of a Puppet run from the Puppet master. The runs are filtered appropriately using the data the resources are tagged with so I don’t need to worry too much about having every single one of my nodes check in at once (thundering herd ahoy!).
The code is here: Puppet mcollective notify report processor
To use it, I need a working mcollective install - my target nodes need to have the mcollective server and Puppet on them. I also need to add some data tags to the resources to trigger the run from the report processor:
file { ‘/etc/httpd/conf.d/vhost_for_my_custom_domain.conf’:
ensure => present,
content => template(‘apachemodule/vhost.conf.erb’),
tag => [‘mconotify–class–loadbalancer’],
}
If this resource changes (successfully, and not in noop mode), the report processor will trigger an mcollective run for all nodes in my configuration that have the Puppet class ‘loadbalancer’ applied to them. Magic!
I need to configure the processor with some mcollective configuration so it can authenticate and send RPC messages appropriately but that only requires a sensibly formatted configuration file. In this way we cut down the volume of external dependencies to get our orchestration going.
I think this method is pretty cool - it uses pure Puppet, although I’d like to not have to use tags to provide the data to the report processor (that would mean a meta-parameter in it’s own right - which isn’t a massive deal but it would require a change to Puppet core). Throw away your shell scripts!
If I look at the debug output for a Puppet run containing tagged resources, I can see what the processor is doing:
A typical example of this would be a service requiring an application instance such as a Tomcat container, a web front end to the application, maybe a database server and possibly a load balancer in front of it all.
This requirement manifests itself in needing to do orchestrated configurations of one or more nodes which may then need to be installed or updated in a particular sequence in order to correctly configure the service that they should offer. In our example, probably get the databases going, then the Tomcat, then the web front end(s) and based on all that configuration configure the load balancer. If a config changes for one part of the stack, then you will probably want changes to other parts to flow through your infrastructure in the right order and probably soon, too!
Consider the following Puppet code for a vhost in our application:
file { ‘/etc/httpd/conf.d/vhost_for_my_custom_domain.conf’:
ensure => present,
content => template(‘apachemodule/vhost.conf.erb’)
}
When that file changes on my webserver, I want the config on my load balancer to update. I can probably use an exported resource within the Apache class that resource lives in to pass the config to apply on my load balancer node later. However when my original node changes I don’t want to sit around and wait for the load balancer to check in and update it’s configuration - that might be 30 minutes time - I want my configuration to update now dammit!
One way of doing this would be to wrap the change up in some external orchestration - my tool of choice would be to do some ordered mcollective magic - and run the nodes in order somehow (manually or scriptomagically).
What I really want to achieve is a way of Puppet doing this for me transparently so I don’t need to worry about external orchestration. So using some resource tagging sauce I can now drive mcollective at the end of a Puppet run from the Puppet master. The runs are filtered appropriately using the data the resources are tagged with so I don’t need to worry too much about having every single one of my nodes check in at once (thundering herd ahoy!).
The code is here: Puppet mcollective notify report processor
To use it, I need a working mcollective install - my target nodes need to have the mcollective server and Puppet on them. I also need to add some data tags to the resources to trigger the run from the report processor:
file { ‘/etc/httpd/conf.d/vhost_for_my_custom_domain.conf’:
ensure => present,
content => template(‘apachemodule/vhost.conf.erb’),
tag => [‘mconotify–class–loadbalancer’],
}
If this resource changes (successfully, and not in noop mode), the report processor will trigger an mcollective run for all nodes in my configuration that have the Puppet class ‘loadbalancer’ applied to them. Magic!
I need to configure the processor with some mcollective configuration so it can authenticate and send RPC messages appropriately but that only requires a sensibly formatted configuration file. In this way we cut down the volume of external dependencies to get our orchestration going.
I think this method is pretty cool - it uses pure Puppet, although I’d like to not have to use tags to provide the data to the report processor (that would mean a meta-parameter in it’s own right - which isn’t a massive deal but it would require a change to Puppet core). Throw away your shell scripts!
Update 20121024: I uploaded the module to the Puppet forge here - http://forge.puppetlabs.com/fiddyspence/mconotify
If I look at the debug output for a Puppet run containing tagged resources, I can see what the processor is doing:
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: CONFIG:{:delimiter=>”–”, :debug=>true, :mcoconfig=>”/tmp/client.cfg”, :mcotimeout=>5}
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: Added mconotify tag mconotify–class–loadbalancermconotify–node–ibroketheinternet.co.ukmconotify–node–debian–1
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: End of tag matching
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: matched mconotify–class–loadbalancer to a class
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: matched mconotify–node–ibroketheinternet.co.uk to a node
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: matched mconotify–node–debian–1 to a node
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: Filters: node 2 class 1
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: Doing an mco run for ibroketheinternet.co.uk,debian
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: /ibroketheinternet.co.uk|debian/
Oct 3 16:41:09 puppet puppet-master[21564]: MCONOTIFY puppet.spence.org.uk.local: Doing an mco run for loadbalancer