Mcollective: ActiveMQ TLS with no Puppet CA and other caveats

As these days we have been working on our new Mcollective setup for the new Cloudways Click&Go product, I wanted to share here some of our experiences.

 

I Am Not Using a Pupper CA. What Should I Do?

For a number of reasons (that would make a good post, noted!) we are using a masterless puppet approach for our puppet infrastructure.  

This works very well, but when deploying mcollective and ActiveMQ  TLS, you may run into some issues as documentation is mainly geared to setups within a traditional puppet master approach.

This is apparent in several stages of the process but more obvious when setting up TLS for ActiveMQ.

So we will focus here on Option C: Do Whatever You Want

 

Option C: Do Whatever You Want

So, how do we go about doing whatever we want?. Simple. We will basically setup our own CA in our Linux box and generate the necessary certificates. I will be focusing here on Debian, as it is our distro of choice. You need to export these instructions to your own distro.

We will be using the CA.pl (/usr/lib/ssl/misc/CA.pl) script that comes with openssl package.  

First of all we edit the CA.pl file and set the duration of the certificates (set whatever you want): 

$DAYS="-days 1825";    # 5 years for server certificates
$CADAYS="-days 3650";    # 10 years for CA itself

Now we edit /etc/ssl/openssl.cnf and do similar changes:

default_days    = 1825            # same as DAYS above

You can set too this section under  req_extensions to avoid having to complete it for each certificate (change with your own details!):

[ req_distinguished_name ]
countryName            = Country Name (2 letter code)
countryName_default        = MT
countryName_min            = 2
countryName_max            = 2

stateOrProvinceName        = State or Province Name (full name)
stateOrProvinceName_default    = Malta

localityName            = Locality Name (eg, city)
localityName_default        = Mosta

0.organizationName        = Organization Name (eg, company)
0.organizationName_default    = Cloudways Ltd

So we are now ready to generate the new CA (provide a good PEM pass-phrase, you can leave default CA file name):

root@cw-mcollective-cg:~# /usr/lib/ssl/misc/CA.pl -newca

Now we need to generate a certificate for our server to be used by ActiveMQ TLS (we run this in /etc/ssl/ssl where our certs will be stored): 

root@cw-mcollective-cg:/etc/ssl/ca#  /usr/lib/ssl/misc/CA.pl -newreq

Most important here is  Common Name. i.e. for our server:

Common Name (eg, YOUR name) []:cw-mcollective-cg.cloudways.com

Now we sign the request with our CA certificate:

root@cw-mcollective-cg:/etc/ssl/ca# /usr/lib/ssl/misc/CA.pl -sign

And there is one last thing we need to do before the certificate is usable for our ActiveMQ TLS setup, remove the PEM pass-phrase that we set for our server certificate when creating it above (it can't be password protected to use it with ActiveMQ):

root@cw-mcollective-cg:/etc/ssl/ca# openssl rsa -in cw-mcollective-cg.cloudways.com-private.pem -out cw-mcollective-cg.cloudways.com-private-nopass.pem

We need now to create additional certificates for our mcollective clients  and mcollective servers following the same approach listed above (-newreq, -sign).

Setting Up Keystores For ActiveMQ

Now that we have all certificates that we need, the last step to get our ActiveMQ TLS infrastructure working is to manually create the keystores. For this we follow mcollective documentation:

http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq_keystores.html#manually-creating-keystores 

One important caveat here. In Step 2: Keystore when we perform the steps to import the server certificate into the keystore, the documentation mentions:

These commands use both an “export/source” password and a “destination” password. The export/source password is never used again after this series of commands.

Even if the “export/source” password is temp, IT MUST BE THE SAME AS THE DESTINATION PASSWORD. If you set them differently (i.e. temppass for export/source and verydifficultpassword for destination), you will get this in ActiveMQ log when trying to start it:

java.security.UnrecoverableKeyException: Cannot recover key

Once all this is done, we just need to move both stores into the appropriate folder  (/var/lib/activemq/main in Debian) and setup ActiveMQ appropriately as explained in Step 4 (adding following ssl context block to activemq.xml):

         <sslContext>
                <sslContext
                keyStore="${activemq.base}/keystore.jks" keyStorePassword="ultrasecret"
                trustStore="${activemq.base}/truststore.jks" trustStorePassword="ultrasecret"
                />
        </sslContext>

Hope it helps someone to save some time!. Let me know if you have any question.