Ansible - using Woodpecker as an alternative to Semaphore
Ansible can be used in many ways: most people likely execute their playbooks on their local machines. Then there is Ansible Automation Platform (AAP) (formerly Ansible Tower) and its small brother AWX. Both are the standard in larger organizations to allow for a more controlled way of running Ansible playbooks and offer RBAC capabilities. These two are also really “heavy” and require a lot of resources to run. Since some time both also require a Kubernetes backend.
These facts are not attractive for homelabs or other small-scale use cases. Luckily, Semaphore exists, which is a project aiming to provide a lightweight alternative to AAP. Maybe there are also even more alternatives, yet I was not able to find any others when researching the topic until today.
I have experience with all of the above tools (AAP/AWX for business, Semaphore for homelab). This points particularly addresses the “homelab side”, i.e., searching for a tool that executes Ansible code remotely while providing access to execution logs and some kind of RBAC.
In my homelab, I had some hickups every few months with Semaphore: Webhooks did not function properly, some runs were stuck forever as pending in the UI and I couldn’t delete them, etc. Also, I was not so happy with the development speed of Semaphore: it is mainly a one-person project for a while now, and while occasionally some PRs get merged, the maintainer is often away for several weeks, and in this time, nothing happens. (Disclaimer: this is of course not meant as a critique, I am very greatful for any open-source work being done, and I am aware that maintaining a project is a lot of work and that the time of anyone is limited. 🤝️)
Secret handling in Ansible
When looking at the technical challenges of running Ansible playbooks remotely, especially when doing so in a public space (e.g. for the infrastructure of FOSS projects), secret handling is a big issue 1 2 3.
Yes, one can set no_log: true but to me, this defeats the whole purpose of running Ansible remotely with log output: I want to have log output and see what is changing/has changed.
On top, I don’t want to fear having overlooked one task where I should have set no_log: true which then exposes a secret.
This is not an issue with any of the tools mentioned above, but rather a general issue with Ansible (and also, the built-in “Ansible vault” does not provide this).
For to me unknown reasons, there is still not yet any solution to the problem of exposing secrets in Ansible’s diff mode, i.e., when telling Ansible to show what (would) change in a certain task.
As there are dozens of integrations for secret injection into Ansible, it might not be so easy to find a common way of declaring which parts of a task content are actually secrets.
Yet to me, this is one of the most essential features of an automation tool: do not expose your secrets.
OK, as long as you’re in a private environment, you might not care so much about this. Yet this fact is a big issue when you are not. And given the wide adoption of ansible in the devops space in the last year, having a central place to execute (and colloborate) on Ansible projects in a public space is becoming more and more popular.
Another important point for such public environments is usually cost. You want a tool that has no license costs and is low on resources. Semaphore checks these boxes (as it is written in Go), but does not solve the secret problem.
Woodpecker for Ansible execution (and secret masking)
And at this point, I want to showcase another alternative that solves the secret problem “by accident,” is low on resources, and has no license costs: Woodpecker CI (WP). Yes, a CI tool - it does not aim to be an Ansible frontend and follow the project/environment/inventory structure of AWX/Semaphore, but is very good at running Ansible playbooks efficiently thanks to its Ansible plugin. Disclaimer: I initiated the WP plugin and ported it over from the (unmaintained) Drone CI Ansible plugin.
A quick tl;dr on Woodpecker: WP is a fork of Drone CI, written in Go. It has minimal resource needs (< 100 MB), is very fast and versatile.
And now let’s talk about how Woodpecker solves the “ansible secret issue”: WP executes the playbooks as they would be logged in the console. However, when a sensitive value is added as a Woodpecker secret and then assigned to an ansible variable, the value is not shown in the output but masked:
🤯️ This is great! So without aiming for it, Woodpecker solves one of the biggest issues of remote Ansible execution. To get a bit more specific, let’s showcase how this works in a technical way:
In a Woodpecker pipeline, one can declare which secrets are injected into a pipeline step:
steps:
- name: docker # random name
image: docker # random image
secrets: [my_ansible_secret] # arbitrary secret name
The secret is then available as an env var in the task environment.
Now to parse this env var correctly into ansible, variables should be declared using the lookup function:
my_ansible_secret: "{{ lookup('env', 'MY_ANSIBLE_SECRET') | default('') }}"
Now, whenever the value of variable my_ansible_secret is used in a task, it will be masked in the log output using *****.
Let’s also briefly address the downsides of using Woodpecker for Ansible execution:
- Plain log output without the possibility to inspect details of a task (as one can do in AWX/Semaphore)
- Woodpecker needs to install all ansible-galaxy dependencies on every pipeline run. This can often take up to 30 seconds. Luckily, one can do this centrally in a task which runs before the playbooks, so multiple subsequent playbooks can run in parallel and make use of the shared dependency cache. Another alternative would be to use caching in the first place, but this is out of scope for this post.
- Secrets must be declared in Woodpecker and injected into each step, otherwise Woodpecker is not aware of them and will not mask their value.
- The above point also means that other external secret plugins, which are often doing some secret scraping before the playbook starts, must export their secrets as environment variables. This way, they can overwrite the “placeholder” secrets declared in Woodpecker.
Summary
Overall, Woodpecker is a great alternative to execute Ansible playbooks remotely. Especially if a CI tool is needed in general, the resource costs of running ansible for both CI and Ansible tasks are close to zero. This is great for homelabs and other resource-sensitive environments.
The approach shown in this post should also be achievable with other CI tools: the WP Ansible plugin only provides a convenient way of running playbooks, but the same can be done by starting out from a bare-bones Alpine image.
PS: If you want to see a public repo where Ansible playbooks are executed using Woodpecker, check out Codeberg-Infrastructure/ansible-configuration.