Ansible Role Creation With Molecule

Molecule is a way to quickly create and test Ansible roles. It acts as a wrapper around various platforms (GCE, VirtualBox, Docker, LXC, etc) and provides easy commands for linting, running, and testing roles. There's a bit of a learning curve in figuring out what its doing, and what it wants, but that time is well made up with the productivity increase in using it effectively.

Installation

Molecule and Ansible can be installed via pip. I typically run on a Fedora system, and have run into issues with libselinux when using a virtual environment. A quick search provides some work arounds, but really it's easiest to just use the --user flag to install molecule with the user scheme.

pip install --upgrade --user ansible
pip install --ugprade --user molecule

If you don't already have ansible/molecule installed, that'll give you some significant output. Pip is good about drawing attention to errors (though the resolution may not always be terribly clear), but the last couple lines of output will provide libraries and their versions that were installed.

Getting started

I have a ~/Projects directory that fills up with half finished projects on my personal computer. Really this works to consolidate things rather than filling up ~/. Wherever you keep your projects, to get started just create a playbooks directory.

~/$ mkdir ~/Projects/example_playbooks
~/$ cd ~/Projects/example_playbooks

Since I'm not including the installation output, below provides the software versions of this example. Both Ansible and Molecule move quick and do have some significant (but not necessarily breaking) changes between point releases, these instructions might not work verbatim if the version numbers vary significantly.

~/Projects/example_playbooks$ ansible --version
ansible 2.6.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/dan/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/dan/.local/lib/python2.7/site-packages/ansible
  executable location = /home/dan/.local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
~/Projects/example_playbooks$ molecule --version
molecule, version 2.17.0
~/Projects/example_playbooks$ tree
.
0 directories, 0 files
~/Projects/example_playbooks$

Create a role

Molecule has pretty excellent help output with molecule --help. In this example, we're going to create a role with molecule and use the vagrant provider. molecule defaults to Docker for provisioning, but I prefer to use vagrant with VirtualBox (see section below for why I prefer that).

Creating a role, and specifying the name and driver will create a role directory structure.

~/Projects/example_playbooks$ molecule init role --role-name nginx_install --driver-name vagrant
--> Initializing new role nginx_install...
Initialized role in /home/dan/Projects/example_playbooks/nginx_install successfully.
~/Projects/example_playbooks$ tree
.
└── nginx_install
    ├── defaults
    │   └── main.yml
    ├── handlers
    │   └── main.yml
    ├── meta
    │   └── main.yml
    ├── molecule
    │   └── default
    │       ├── INSTALL.rst
    │       ├── molecule.yml
    │       ├── playbook.yml
    │       ├── prepare.yml
    │       └── tests
    │           ├── test_default.py
    │           └── test_default.pyc
    ├── README.md
    ├── tasks
    │   └── main.yml
    └── vars
        └── main.yml

9 directories, 12 files

As you can see, that command creates quite a few directories. Most of these are standard/best-practices for Ansible.

defaults - default values to variables for the role handlers - specific handlers to notify based on actions in Ansible meta - Ansible-Galaxy info for the role if you are uploading this to Ansible-Galaxy molecule - molecule specific information (configuration, instance information, playbooks to run with molecule, etc) README.md - Information about the role. Well documented, excellent feature (I'm a big fan of documentation, should be obvious if you're reading this) tasks - tsaks for the role vars - other variables for the role

Why I prefer Vagrant and VirtualBox over Docker

Docker is great, don't get me wrong. I'm a huge proponent of Linux. Docker on Mac, vs Docker on Linux, vs Docker on Windows are all different things. VirtualBox is far more cross platform than Docker. Something that can be done on Fedora is far different than the latest iteration of OSX, and even more different than Windows. Also, using a VPN client that dictates IP routes causes serious issues in networking between Docker containers. Finally, Systemd with Docker requires specific images and root access specifying mounting a cgroup volume. Since the majority of the work I do is not using Docker for orchestration and instead relies on services running on systemd, a VM is a better solution for my use-case (and closer to "production") than a container. Yes, a container is far lighter than a VM, but not an issue with decently modern hardware (for my use case). Ultimately, while I might be able to make something work for Docker locally on my system, odds are it's not going to work for anyone not running the same OS/Distro.

Modifications from default Molecule

There are a few defaults I always change when using molecule as it uses Cookie-Cutter to create a default configuration. The first, molecule defaults to Ubuntu, but almost all of the systems I interact with are RHEL based. Also I prefer to specify the memory and CPUs rather than relying on the box defaults. Finally, we're using nginx in this example we may as well set up port forwarding to hit the webserver locally.

These changes are made through modification of the molecule/default/molecule.yml file to look like something below. The molecule.yml is the configuration used by molecule for instances, tests, provisioning, etc.

Heads up, a raw copy/pasta of below will result in an error. Read on to see why

~/Projects/example_playbooks/nginx_install$ cat molecule/default/molecule.yml 
---
dependency:
  name: galaxy
driver:
  name: vagrant
  provider:
    name: virtualbox
lint:
  name: yamllint
platforms:
  - name: nginx_install
    box: centos/7
    instance_raw_config_args:
      - "vm.network 'forwarded_port', guest: 80, host: 9000"
    memory: 512
    cpus: 1
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

Once we've got the `molecule` configuration to our liking, time to start working on the role itself. Ansible role tasks are in `tasks/main.yml` for the role. This example is pretty simple, so all we're doing is installing a repository to install `nginx`, installing `nginx`, and starting/enabling `nginx`. The only Ansible modules we need for this is `yum` for package installation, and `systemd` to start and enable the service.

~/Projects/example_playbooks/nginx_install$ cat tasks/main.yml

tasks file for nginx_install

  • name: Install epel-release for nginx yum: name: epel-release state: present become: "yes"

  • name: install nginx yum: name: nginx state: present become: "yes"

  • name: ensure nginx running and enabled systemd: name: nginx state: started enabled: "yes" become: "yes"

Molecule does some great things. It handles the Orchestration of the virtual environment to test, lints Ansible syntax, runs a test suite, and even lints that test suite, as well as destroying the orchestrated environment at the end.

Writing tests for the role

We can manually test the role with some SSHing and curl, but testinfra is included as the default test step of molecule. Testinfra uses pytest and makes it easy to test the system after the role is run to ensure the role has the results that we expected.

This role is pretty simple, so our tests are pretty simple. Since we're just installing and starting nginx, there's not a whole lot more we're looking for in our test. Of course molecule provides a good default, and testinfra documentation even uses nginx in their quickstart.

Tests - quantity or quality?

The tests below are three tests that are all pretty simple. The overall count of tests really doesn't matter. Below we've got three tests, but we could easily have one, or five. This may vary based on the Test Developer, but I chose the three below because it follows a pretty logical order.

  • Make sure nginx is running and enabled

This is easiest looking at it backwards. If we had one test to see if nginx is running, if that fails do we have any idea why? Was it installed? Was the configuration wrong? Was it not started? My approach is to first make sure it is installed, if not, the rest of our tests fail and we can see pretty easily why. So we see it's installed, so next we check if the configuration exists (in a more elaborate example, we'd probably check to make sure there is some expected text in the configuration file). Finally, we make sure nginx is running and enabled. The tests follow a logical flow of prerequisites to get to our ultimate state, and knock out some troubleshooting steps along the way.

```cat molecule/default/tests/test_default.py import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner( os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')

def test_nginx_installed(host): nginx = host.package('nginx') assert nginx.is_installed

def test_nginx_config_exists(host): nginx_config = host.file('/etc/nginx/nginx.conf') assert nginx_config.exists

def test_nginx_running(host): nginx_service = host.service('nginx') assert nginx_service.is_running assert nginx_service.is_enabled

## Running Molecule

We've got our role written, and our tests. We could just run `molecule test` and work through all the steps. But, I prefer running `create`, `converge`, and `test` all separately, and in that order. This separates the various steps and makes any fails easier to track down.

### Molecule create

The first step of Molecule is the creation of the VirtualMachine. For `Docker` and `vagrant` providers, Molecule includes a default `create` playbook. Running `molecule create` will create the VirtualMachine for our role based on the `molecule.yml` configuration.

```~/Projects/example_playbooks/nginx_install$ molecule create
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── create
    └── prepare

--> Scenario: 'default'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Create molecule instance(s)] *********************************************
    failed: [localhost] (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
    fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

    PLAY RECAP *********************************************************************
    localhost                  : ok=0    changed=0    unreachable=0    failed=1   


ERROR:

This was entirely unplanned, but as I was gathering output for this command I got an error. Ansible has a no_log property for tasks that is intended to prevent the outputting secrets. Obviously in the create part here we received an error with no usable output to determine the cause of the error. We can set the environment variable of MOLECULE_DEBUG to log errors, but the first thing I do (because it's less typing) is add the --debug flag.

~/Projects/example_playbooks/nginx_install$ molecule --debug create
...
        }, 
        "item": {
            "box": "centos/7", 
            "cpus": 1, 
            "instance_raw_config_args": [
                "vm.network 'forwarded_port', guest: 80, host: 9000"
            ], 
            "memory": 512, 
            "name": "nginx_install"
        }, 
        "msg": "ERROR: See log file '/tmp/molecule/nginx_install/default/vagrant-nginx_install.err'"
    }

    PLAY RECAP *********************************************************************
    localhost                  : ok=0    changed=0    unreachable=0    failed=1

Reading into the error tells us it was an "error" in Vagrant, and not necessarily one with molecule itself. We can look at the file provided in the error output for more clues.

~/Projects/example_playbooks/nginx_install$ cat /tmp/molecule/nginx_install/default/vagrant-nginx_install.err
### 2018-09-07 17:32:59 ###
### 2018-09-07 17:32:59 ###
There are errors in the configuration of this machine. Please fix
the following errors and try again:

vm:
* The hostname set for the VM should only contain letters, numbers,
hyphens or dots. It cannot start with a hyphen or dot.

### 2018-09-07 17:33:20 ###
### 2018-09-07 17:33:20 ###
There are errors in the configuration of this machine. Please fix
the following errors and try again:

vm:
* The hostname set for the VM should only contain letters, numbers,
hyphens or dots. It cannot start with a hyphen or dot.

Well, that's easy. Our hostname can't contain _. A quick edit to the molecule.yml should fix this right up.

```~/Projects/example_playbooks/nginx_install$ grep -A1 platform molecule/default/molecule.yml platforms: - name: nginx-install

Try again on the `create`:

~/Projects/example_playbooks/nginx_install$ molecule create --> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml. Validation completed successfully. --> Test matrix

└── default ├── create └── prepare

--> Scenario: 'default' --> Action: 'create'

PLAY [Create] ******************************************************************

TASK [Create molecule instance(s)] *********************************************
changed: [localhost] => (item=None)
changed: [localhost]

TASK [Populate instance config dict] *******************************************
ok: [localhost] => (item=None)
ok: [localhost]

TASK [Convert instance config dict to a list] **********************************
ok: [localhost]

TASK [Dump instance config] ****************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=2    unreachable=0    failed=0

--> Scenario: 'default' --> Action: 'prepare'

PLAY [Prepare] *****************************************************************

TASK [Install python for Ansible] **********************************************
ok: [nginx-install]

PLAY RECAP *********************************************************************
nginx-install              : ok=1    changed=0    unreachable=0    failed=0

### Molecule converge

Molecule `create` only acts as orchestration. The `converge` step is what runs our playbook that calls our role. There's good reason to do these steps separate. First, the `create` step ensures our VirtualMachine is provisioned and started correctly. Once it's up, we've got less troubleshooting when actually running the playbook.

When first learning Ansible or working on a more complicated role, we could just run `converge` after every task added (or every few depending on our confidence) to our role to make sure it does what we intend for it to do. Because we only have three simple tasks, we can run converge to test all tasks at the same time.

```~/Projects/example_playbooks/nginx_install$ molecule converge
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── dependency
    ├── create
    ├── prepare
    └── converge

--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'create'
Skipping, instances already created.
--> Scenario: 'default'
--> Action: 'prepare'
Skipping, instances already prepared.
--> Scenario: 'default'
--> Action: 'converge'

    PLAY [Converge] ****************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [nginx-install]

    TASK [nginx_install : Install epel-release for nginx] **************************
    changed: [nginx-install]

    TASK [nginx_install : install nginx] *******************************************
    changed: [nginx-install]

    TASK [nginx_install : ensure nginx running and enabled] ************************
    changed: [nginx-install]

    PLAY RECAP *********************************************************************
    nginx-install              : ok=4    changed=3    unreachable=0    failed=0

Cool. It worked. We think, anyway. While our playbooks ran, running our tests will really make sure that it all worked.

Molecule test

Next we run test. This goes through all the steps and will tell us if what we think we're doing is actually working based our our testinfra tests. Destroying any existing VirtualMachine, checking syntax on role, creating the VirtualMachine, linting and running our tests, etc. If there are any issues, this should let us know.

```~/Projects/example_playbooks/nginx_install$ molecule test --> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml. Validation completed successfully. --> Test matrix

└── default ├── lint ├── destroy ├── dependency ├── syntax ├── create ├── prepare ├── converge ├── idempotence ├── side_effect ├── verify └── destroy

--> Scenario: 'default' --> Action: 'lint' --> Executing Yamllint on files found in /home/dan/Projects/example_playbooks/nginx_install/... Lint completed successfully. --> Executing Flake8 on files found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/... /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/test_default.py:13:1: E302 expected 2 blank lines, found 1 /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/test_default.py:17:1: E302 expected 2 blank lines, found 1 /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/test_default.py:21:1: W391 blank line at end of file An error occurred during the test sequence action: 'lint'. Cleaning up. --> Scenario: 'default' --> Action: 'destroy'

PLAY [Destroy] *****************************************************************

TASK [Destroy molecule instance(s)] ********************************************
changed: [localhost] => (item=None)
changed: [localhost]

TASK [Populate instance config] ************************************************
ok: [localhost]

TASK [Dump instance config] ****************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=2    unreachable=0    failed=0

Another unintended failure. Lint issues in the python tests. Flake provides excellent output for pep errors, so we know exactly what to fix based on the output.

Addressing those issues and rerunning results in the following:

```~/Projects/example_playbooks/nginx_install$ molecule test
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/dan/Projects/example_playbooks/nginx_install/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/...
Lint completed successfully.
--> Executing Ansible Lint on /home/dan/Projects/example_playbooks/nginx_install/molecule/default/playbook.yml...
Lint completed successfully.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Populate instance config] ************************************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    skipping: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=2    changed=0    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'syntax'

    playbook: /home/dan/Projects/example_playbooks/nginx_install/molecule/default/playbook.yml

--> Scenario: 'default'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Create molecule instance(s)] *********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config dict] *******************************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Convert instance config dict to a list] **********************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=4    changed=2    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'prepare'

    PLAY [Prepare] *****************************************************************

    TASK [Install python for Ansible] **********************************************
    ok: [nginx-install]

    PLAY RECAP *********************************************************************
    nginx-install              : ok=1    changed=0    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'converge'

    PLAY [Converge] ****************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [nginx-install]

    TASK [nginx_install : Install epel-release for nginx] **************************
    changed: [nginx-install]

    TASK [nginx_install : install nginx] *******************************************
    changed: [nginx-install]

    TASK [nginx_install : ensure nginx running and enabled] ************************
    changed: [nginx-install]

    PLAY RECAP *********************************************************************
    nginx-install              : ok=4    changed=3    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'idempotence'
Idempotence completed successfully.
--> Scenario: 'default'
--> Action: 'side_effect'
Skipping, side effect playbook not configured.
--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/...
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
    rootdir: /home/dan/Projects/example_playbooks/nginx_install/molecule/default, inifile:
    plugins: testinfra-1.14.1
collected 3 items                                                              

    tests/test_default.py ...                                                [100%]

    =========================== 3 passed in 5.33 seconds ===========================
Verifier completed successfully.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config] ************************************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=3    changed=2    unreachable=0    failed=0

Great! With all that, now we know that our Ansible and python tests are linted, and our tests run meaning our role does what we intend for it to do.

Molecule verify

I kind of skipped a step here. I've described the steps of:

  • molecule create - create the VirtualMachine to make sure molecule is configured correctly.
  • molecule converge - run multiple times as we add tasks to our role.
  • molecule test - once we're happy, run all the steps of Molecule.

Really though, since molecule test runs through all the steps (creation, linting, testing, deletion...), and earlier I laid out the steps of running converge to manually test each time, this doesn't really fit the workflow I metioned. We can seperate out the molecule steps a little further.

Rather than running molecule test, instead we can run molecule verify seperately:

```~/Projects/example_playbooks/nginx_install$ molecule verify --> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml. Validation completed successfully. --> Test matrix

└── default └── verify

--> Scenario: 'default' --> Action: 'verify' --> Executing Testinfra tests found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/... ============================= test session starts ============================== platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0 rootdir: /home/dan/Projects/example_playbooks/nginx_install/molecule/default, inifile: plugins: testinfra-1.14.1 collected 3 items 

tests/test_default.py ...                                                [100%]

=========================== 3 passed in 5.25 seconds ===========================

Verifier completed successfully. ~/Projects/example_playbooks/nginx_install$ molecule lint --> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml. Validation completed successfully. --> Test matrix

└── default └── lint

--> Scenario: 'default' --> Action: 'lint' --> Executing Yamllint on files found in /home/dan/Projects/example_playbooks/nginx_install/... Lint completed successfully. --> Executing Flake8 on files found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/... Lint completed successfully. --> Executing Ansible Lint on /home/dan/Projects/example_playbooks/nginx_install/molecule/default/playbook.yml... Lint completed successfully. ```

Conclusion

Molecule is a great abstraction for multiple steps to the create/test/clean up steps of testing an Ansible role during development. Not only does it create and provide sane defaults to the directory structure of a role, it makes it easy to create a test a role during development. While there may be a bit of a learning curve, the increased productivity of testing during development makes it an absolutely worthwhile investment.

social