fiddyspence's blog

Puppet Auto Scaling

Wouldn’t it be a thing if Puppet just knew how many things of a type you wanted and when you built a server it just did the right thing. People in the community have been abusing Hiera as a classification tool for ages (and my feelings on this are still that if you’re using facts as a classification engine, you’re doing it wrong). Certificate CN regex matching is a thing, but it means you’ve got to worry about making the CN right.

Is there another way?

External Fact Pluginsync

Facter 1.7 introduced the ability to have external facts, but no real way to distribute them so that pluginsync operates and they get loaded on the same run. Suggestions for distributing them have included packages, and using file resources. Neither of which solve this problem. After a conversation on #puppet recently, we came up with this partial solution:

In a module, with the normal structure. using the lib directory add another directory for the external facts:

├── lib
|    ├── externalfacts
|    |  └── foofact.py
|    └── facter
│       └── rubyfact.rb
├── manifests
│   └── init.pp
├── Modulefile
├── spec
│   └── spec_helper.rb
└── tests
    └── init.pp

Thus, the external facts are pluginsynced, but not to the default facter directory, instead they are sent to an externalfacts directory under the plugindest directory.

Currently there’s no way to provide custom config to facter other than on the command line, so without modifying either Puppet or Facter (I found a ticket which would make sense: give Facter a config file), the easiest way of making the external fact work would be just to symlink some directories on setup. Having done that, Facter will automatically load the pluginsynced external facts during the remainder of the Puppet run. This still requires some setup before it works seamlessly, but as an interim solution should be OK.

Thusly, /var/opt/lib/pe-puppet/lib/facter for PE, or /var/lib/pe-puppet/lib/facter for Puppet Open Source could be symlinked to either /etc/facter/facts.d or /etc/puppetlabs/facter/facts.d and the facts will get loaded during the same run that they are synced.

[root@puppet3 facter]# pwd
[root@puppet3 facter]# ls -l
total 0
lrwxrwxrwx 1 root root 26 Oct 28 09:50 facts.d -> /var/lib/puppet/lib/externalfacts

Abstract All the Things

One of the features of configuration management platforms, and this is expecially true of Puppet with its resource model, is that it presents an abstraction of specification from implementations. This I can write some Puppet DSL to install a package, that assuming many other things are equal, will work on multiple OS platforms to implement the configuration you expect (installing some software).

There are some higher levels of abstractions available too - putting configurations into Classes, wrapping Classes in Classes to implement things that look like whole application stacks.

Zabbix Is a Thing

Monitoring is a fairly important component of a production stack. I’ve heard really good arguments for it being part of the dev stack too - using production monitoring to test application response during continuous build/deployment supplements unit or integration tests really nicely. Puppet has OK support for Nagios, I thought it was time we had some better support for something else - a quick check showed nothing really compelling for Zabbix. Choice made.

Encrypted Facts and Inventory Service

Puppet extensions like hiera-gpg provide some entry points to enable a Puppet administrator to protect static data that feed into generating configurations. There are however occasions where one might want existing potentially sensitive data to be moved between managed nodes, for example via storeconfigs. Another example of this might be facts that are passed into the inventory service for storage that needs to be consumed within a class but needs to be obscured against casual viewing. I encountered this last requirement first when dealing with things like pre-shared keys (ceph volume keys, for example) that are easiest to generate on the node and then shared to other nodes via a fact - however I didn’t particularly want that data to be easily viewable.

Racoon, Ipv6 and My Mac


So, I was doing some hacking about today, and I noticed all of a sudden I had a shiny new utun0 interface on my Mac Mini. Needless to say, I was panicked.

 utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
 inet6 fe80::d89b:9a8a:xxxx:xxxx%utun0 prefixlen 64 scopeid 0xa
 inet6 fd1b:6be5:9aec:1374:d89b:xxxx:xxxx:xxxx prefixlen 64 

What the bloody bloody is this, I hear you ask?!

It turns out that BackToMyMac makes this happen using racoon

0 211 1 0 1:53pm ?? 0:00.07 /usr/sbin/racoon -D

Based on the configuration at /private/var/run/racoon and /private/etc/racoon

I turned it off, but only after noting that I liked that it’s one of the few in the wild implementations of anything using solely IPv6 I’ve seen….

Rest Api Security for Fun and Profit

Puppet fileserving supports a REST api for file serving.  Individual mount points can be created which give access to specific nodes.  Alternatively you might use the special modules mount to serve data out of the files directory from <insert name of module here>.

There are security implications of fileserving from modules using the default REST API security configurations.  The issue is that the security landscape is a bit on the flat side - any authenticated node can gain access to any file resource from any module.  The upside is that it’s only open to authenticated nodes - no anonymous access allowed, but the downside is that any node can request any file.

Thus a file resource for my SSL private key,

file { ‘/etc/httpd/ssl/private_keys/puppet.spence.org.uk.local.pem’:
  ensure => file,
  owner  => ‘apache’,
  group  => ‘apache’,
  mode   => ‘0600’,
  source => ‘puppet:///modules/ssl/puppet.spence.org.uk.local.pem’,

is a little bit on the risky side if you ask me, because any node knowing the path can ask for the key.  Not.  Awesome.

The solution is to use auth.conf  to restrict access to the files path to just the nodes that need that particular set of resources either at the module or the individual file resource level:

path ~ \A/file_(metadata|content)/modules/ssl/puppet.spence.org.uk.local.pem
auth yes
allow puppet1.spence.org.uk.local
allow puppet2.spence.org.uk.local

You don’t necessarily need to restrict right down to the file level, like I’ve shown here.  In fact auth.conf gives enough flexibility to be able to serve *machine* specific files out of a module using regular expressions.


file { ‘/etc/httpd/ssl/private_keys/puppet.spence.org.uk.local.pem’:
  ensure => file,
  owner  => ‘apache’,
  group  => ‘apache’,
  mode   => ‘0600’,
  source => ‘puppet:///modules/ssl/${::clientcert}/puppet.spence.org.uk.local.pem’,

Which is controlled by the following ACL:

path ~ \A/file_(metadata|content)s?/modules/[^/]+/([^/]+)
auth yes
allow $2

This means that you can have directory structures to securely serve node specific files from a single module where each node can only access it’s own files and not any other files from the same module:

├── files
│   ├── puppet1.spence.org.uk.local
│   │   └── puppet1.spence.org.uk.local.pem
│   └── spacewalk.spence.org.uk.local
│       └── spacewalk.spence.org.uk.local.pem

There’s some additional overhead to managing this - it’s a slightly custom auth.conf configuration, and it would need to go towards the top of the file so as to be the most specific match for the file_(content|metadata) endpoint.

Whether this is worthwhile depends entirely on whether the paranoia torrent is running strong or not.  I think it’s entirely necessary for multi-tenant infrastructure especially where one is hosting configurations for multiple business units or different organisations.

I also think it’s well worth doing for configurations that are security sensitive - private key data such as I’ve shown here, maybe passwords - that sort of stuff.

Secure Puppet Code Collaboration

I originally got this posted at PuppetLabs blog, but am reposting here so I have a copy.
Puppet compiles a catalog using Puppet DSL code and data that’s usually at rest on the local filesystem of the puppet master. An agent requests a catalog from a master, and subsequently enforces the catalog locally. It’s common for multiple teams to collaborate on the configuration code and data, with the different teams being responsible for different modules. Often times, this means that the Puppet module path is actually made up of two or more module paths, where each modulepath entry is a different version control system repository managed by a different team.
  modulepath = /data/puppet/core/modules:/data/puppet/webapps/modules
I will have a manifest (maybe an ENC too, but let’s just talk about manifests here):
  manifest = /data/puppet/manifests/site.pp
I am more likely to have my core security modules in the first part of my module path. Potentially I *could* have a class security {..} in both halves of my module path; however, the module found first by the autoloader will take precedence, and the module in the web apps part of the path won’t be evaluated. It’s important to get the configurations that you want in first, especially because there’s no hard and fast rule that will restrict resource definitions for security configurations to that security module. Any resource declaration might be in any module, subject to uniqueness constraints, and so forth.
Thus, my security class manages a resource:
   user { 'root': 
password => 'myspecialpasswordhash',
If another class attempts to manage the same resource, Puppet will return a catalog compilation error, and no configurations will be managed. This is good. My root user won’t get hosed by a rogue configuration. However, there’s a twist to this story. What if my manifest (site.pp) declares another class ‘configurethewebserver‘? That class is defined in the the second half of my module path (which is where the modules that deploy my applications live, as opposed to the core standard operating environment), and gets declared, quite legitimately, in a node definition in the site manifest.
The downside is that it also turns out that someone (a rogue/misguided sysadmin and/or developer who can contribute to my Puppet code) has remembered that you can do class inheritance and override parameters and, in order to achieve their configuration aims, has inserted code something like the following in a Puppet class:
class configurethewebserver inherits security { 
User['root'] {
password => 'wejustpwnedyourwebinfrastructure',
Unfortunately, the next time I do a Puppet run on my production boxes, my root user no longer has the password hash ‘myspecialpasswordhash‘,but the overridden inherited value ‘wejustpwnedyourwebinfrastructure‘ from the rogue class in the second half of the module path. Sigh.
The clear security moral here is that if you give someone the ability to put code on your puppet master, you’ve given them root on all the nodes that you’re managing on that puppet master.Don’t give people root if you don’t have to.
Aside from the security implications of delegating access to specific portions of Puppet module path aside, the issue we are really trying to fix here is making sure that the configurations we think we are defining are actually taking effect and work. So what can we do with processes and workflow to help ensure consistency and make sure we don’t fall victim to accidental or deliberate misconfigurations?
First make sure that all the configurations you regard as being core for security are actually defined in your code and data — that way you can positively validate them. If you don’t define something, you’re not managing it, and you can’t test it. Testing things you haven’t attempted to manage just won’t work — really, you’d just be relying on the defaults. Even if you just want to use the default, enforce it because that’s your desired configuration.
Second, you might want to have a gateway between commits to your version control system and the updates that follow on your puppet masters from a workflow perspective. That gateway is likely to be a person tasked with merging edits into the branch of the repository that the puppet master uses for catalog compiles. That way, you stand a chance that a real human being will review incoming changes. It turns out that humans are really good at heuristics; spotting subtle vulnerabilities tends to be a real person’s job. When it comes down to it, computers essentially say ‘yes’ or ‘no’, and a computer that says ‘maybe’ is probably screwing with you. At the end of the day, you’ve got to fundamentally understand the codebase; if you allow rogue code on your puppet master, you’re in trouble.
Third, you should certainly run automated testing against your module code using a continuous integration framework. Make sure you use appropriate data when you do so! Unit testing toolsets like rspec and rspec-puppet to positively validate the functionality of your code and modules is valuable.
Fourth, consider testing full catalog compiles prior to deploying them. This would involve compiling a catalog for each node in your infrastructure and having integration tests that validate each security related resource configuration based on the your data.
Finally, as well as unit testing code, cheap virtualization technologies make it possible to fully validate the entire application stack in an automated fashion and subsequently test it. If you use continuous integration tooling to automatically spin up your environment, you can test and validate your entire build before you deploy to production since the tool will use your Puppet code and data for configuration. By using the same monitoring tools you use in production against this ephemeral application stack, you can prove a full integration test. It’s important to be confident that when you move to production, you will get the configurations you want.

The Fact Is….


This post was originally published at https://puppetlabs.com/blog/the-fact-is/ - I am republishing it here.

One of the major interfaces to extend functionality with Puppet is the use of Facter and custom facts to provide catalog compile time data to the Puppet master to customise the configurations for a specific node (using PuppetDB, this data becomes available even when you’re doing things elsewhere too).  A fact, in case you were wondering, is a key/value data pair that represents some aspect of node state, such as it’s IP address, uptime, operatingsystem or whether it’s a virtual machine.

As an example within Puppet, one might use this data in the context of a catalog compile to make a decision about the catalog that we sent back to the node.  A fact that tells us what Operating system a node has could cause some conditional logic in a class to tell the node the right set of packages to configure ntp on the system (because the package names differ between Linuxes, let alone what you do for a Windows).  Alternatively one might use the ipaddress fact to customise the contents the appropriate listen directive in Apache or SSH.

Custom facts might be written using either by extending Puppet using Ruby, or even more trivially by using the facter.d library shipped with the Puppet Labs stdlib module or even by simply setting an environment variable available during a Puppet agent run.  In this way, you can provide the catalog compilation infrastructure rich state information about your nodes.

We can see that it might become common practice in all one’s modules to alter behaviour in Puppet classes on the basis of node state data - in fact it’s one of the things that makes it so useable and flexible.  However, there are some security implications here.

To take an example, one of the more common facts that gets used is the ‘hostname’ fact.  Typically, unless for some reason you choose otherwise, this will match the certname of the node (i.e. the CN on the certificate generated out of the Puppet CA when you provisioned the node) though Puppet won’t use the same mechanism to arrive at that determination.

The value of $::hostname, depending on your platform, will probably be arrived at by running a command like hostname from the context of the Facter libraries used by Puppet.

A better solution is to extract the validated CN of the certificate used to authenticate a node on the Puppet master and use that data instead.  Previous versions of this post suggested using $clientcert - turns out that the shared popular misconception that there was any value in that bit of data was utterly mistaken.

$actualclientcert = certcheck()

module Puppet::Parser::Functions
newfunction(:certcheck, :type => :rvalue, :doc => <<-EOS
                Returns the actual certname
           ) do |arguments|
  return host

From a security perspective there is a vast difference between the assertion backed by PKI that a node’s hostname is the fact versus the certname.  This is especially true given the variety of ways we might arrive at the value that has been sent to the Puppet master.  Assuming we trust the PKI, the state of being that the node’s hostname matches the certificate (if that’s the way we run our PKI) is significantly more trustworthy that the assertion that the hostname fact is the correct one.  This is the major reason Puppet uses SSL and PKI for node identification.

Admittedly, in order to get to the point where you’ve sent that data to a Puppet master, one needs sufficient rights over the private key material on a node to get as far as getting Puppet to think about sending you stuff.  The implication for secure module writing is that if you want to securely distribute the right configuration settings and associated data back down to nodes you must use trustworthy data when compiling a catalog.

For settings with a lesser security impact, OS package names for example, trusting facts is cheaper and probably secure enough (depending on your environment) that you might as well just use them.  What you probably don’t want to do however is have your module that distributes, for example, private SSL key data to nodes that asset via facts that they should have them.

For modules with security critical information in them, the determination as to how that class instantiates on a node should be made solely using data on the Puppet master - that might be data in the puppet code that comprises your class, in an ENC or in your hiera backend - it should in no way be influenced by an untrusted assertion from a node.  Even then, the root of the decision tree within your code ought to be your PKI - thus based on the SSL certificate the node used to authenticate.

If you completely trust data supplied via facts, then including data, settings or anything else in a catalog  where that information might be used to perform a privilege escalation where a node has spoofed fact data, you’ve been hacked.  The true danger here is that the escalation you achieve is to get root on something else.  If in your organisation different sets of users have root on different sets of boxes, your module design philosophy could lead to leakage across node sets.

As an aside, if users have root on multiple boxes, what’s to stop them moving SSL certs around the place and putting the right configurations on the wrong boxes.  Don’t give people root if you don’t have to by the way.  Sudo everything is probably a bad thing too.  Sudo don’t do that.

The alternative to using facts, as I mention above, is to use trustable data.  That’s a combination of $actualclientcert and the data you have control over on your Puppet master.  It may increase the overhead of the amount of static data you need to manage, but this is going to be a trade off between security, how much you can be bothered, and what the outcome of it all going wrong is.  Incidentally, this significantly promotes the idea of encoding useful data in hostnames and therefore Puppet certificate names (see my post on the simple things - this is an extension of stuff I hadn’t fully thought through there).

I personally have no problem with using facts to make decisions with low impact outcomes security wise.  I can’t immediately see is any way of making facts trustworthy given the current architecture though - Puppet runs as root, if you’re root you can run Puppet and have access to the private key and you can spoof facts.

Ultimately however, this comes down to is the good old fashioned security principles of not trusting user input, and sanitising the hell out of it before you do anything with it.

You Are Feeling Very Sleepy…

I recently wrote a Puppet report processor (mconotify) that processes Puppet reports and sends mcollective messages to trigger Puppet runs.  It was an attempt to build infrastructure where one part of an infrastructure can securely touch another part of the same infrastructure.

I’ve also been thinking about how to make catalog application also take into account parts of an infrastructure when applying state to a node.  Being able to test another configuration at apply time in a way that pauses a Puppet run while you wait for something else to happen sounded like a good idea to me.

One solution I came up with was to use the Puppet resource abstraction layer to do a sleep for an arbitrary length of time - if you have to wait 5 minutes, then

  sleep { 300: }

A more sophisticated approach would be to sleep for 5 minutes, but stop sleeping if your condition is satisfied:

  sleep { ‘until i need to wake up’:
    bedtime       => 300,
    wakeupfor     => ‘nc thedatabaseserver 3306 -w 1’,
    dozetime      => 10,
    failontimeout => true,

In this example, we’ll sleep for a maximum of 5 minutes, but wake up every 10 seconds to netcat to the mysql port on a database server and if that exits successfully exit the sleep and carry on.  If the command doesn’t return 0 by the end of 5 minutes, we fail the resource (and anything in my catalog dependent on it will therefore not apply).  If you wanted to continue anyway, set failontimeout to false, and the thing won’t fail.  Simples.

You could use exec resource types, but it starts to look complicated in terms of the commands you have to run - wrapping it up in a type felt like the right solution here.

I think this is pretty neat.

The code is available on the Puppet Forge, as usual: http://forge.puppetlabs.com/fiddyspence/sleep