Satellite and Ansible Tower are each powerful tools, and many customers utilize both of them. It is possible to integrate these tools and in part 1 of this series we covered how to configure Ansible Tower to pull a dynamic inventory of hosts from Satellite. 

One of Satellite’s features is the ability to provision new hosts, while one of Ansible Tower’s main features is the ability to configure hosts. By integrating these tools, Red Hat Enterprise Linux (RHEL) hosts provisioned by Satellite can be configured to automatically make a provisioning callback to Ansible Tower. This provisioning callback functionality allows you to run an Ansible playbook on the new RHEL host so it can be configured using the Ansible Tower infrastructure. The end result is the ability to go into Satellite, provision a new host, and automatically obtain a configured host via Ansible Tower. This can save system administrators time and allow them to meet the needs of their organization. 

One of the prerequisites for setting up provisioning callbacks is having Satellite configured as a dynamic inventory source within Ansible Tower so, if you haven’t already, first follow the steps outlined in the previous blog post. 

We must also have Satellite setup for provisioning, and it should be configured to use host groups when provisioning. If you don’t currently have Satellite setup for provisioning, refer to the Provisioning Guide documentation. 

In this post, we will show how to configure Satellite and Ansible Tower to run a playbook after Satellite provisions hosts. 

Overview

Automation tools, such as Ansible Tower, generally fall into one of two categories of operation: push or pull. In push systems, like Ansible Tower, the connection is initiated by the automation server to the hosts. In pull systems, hosts initiate the connection to the automation server. 

To illustrate why we need provisioning callbacks, consider the following scenario: we have a playbook doing host configuration that runs daily at midnight. We provision several new servers with Satellite at 8:00 AM. Satellite has been configured as a dynamic inventory source for Ansible Tower, so the new hosts will be included in the inventory. However, the next scheduled run of the playbook isn’t until midnight, which is 16 hours later. Obviously, this is not an ideal situation as we would like the playbook to run immediately after the new host is provisioned. 

Provisioning callbacks allow hosts to instruct Ansible Tower to run a playbook immediately. So as soon as a new host is built, it can request Ansible Tower run a playbook. 

The high level overview of the process is as follows:

1. An Ansible Tower machine credential is created for the root user, with the same root password defined in the existing Satellite host group used for provisioning. Ansible Tower will use this credential to authenticate to the newly provisioned host. 
2. The Ansible Tower job template is configured with provisioning callbacks enabled. Callbacks can be initiated by making a request to a URL with the host config key shown on the job template page. 
3. The Satellite host group is configured with several parameters defining how the client can make the Ansible Tower provisioning callback request (Ansible Tower URL, the host config key, and the template ID).
4. A new host is provisioned by Satellite and the provisioning templates cause a /etc/systemd/system/ansible-callback.service file to be created on the new host (for RHEL 7 and RHEL 8 hosts, RHEL 6 uses a script). When the new host boots for the first time, this systemctl service file causes it to make a request to the Ansible Tower provisioning callback URL with the correct host config key and template ID number. 
5. Ansible Tower receives the provisioning callback request and validates the host config key is correct. If so, it initiates a run of the template on the new host and uses the root user credential to authenticate. The template is run on the host, which configures the host.

Configuring Ansible Tower for Provisioning Callback

We’ll start on the Ansible Tower server by creating a playbook that will set the contents of /etc/motd, create a user, and install a package. This is the playbook that will be configured to run on newly provisioned hosts. 

In most situations, we would want to use a source configuration management system to store our playbooks; however, for the sake of simplicity in this post, we will use a playbook stored locally on our Ansible Tower server. We will log in to the Ansible Tower server over SSH and create the /var/lib/awx/projects/provision directory with the following command:

​# mkdir /var/lib/awx/projects/provision

We’ll then create our playbook, named provision.yaml, within this directory with the following content:

​---

- name: Provision new host

  hosts: all

  tasks:

  - name: Set content in /etc/motd 

    copy:

      content: Authorized use only!

      dest: /etc/motd

      mode: 644

      owner: root

      group: root


  - name: Create brian user account

    user:

      name: brian

      uid: 10000

      state: present


  - name: Install tmux package

    yum:

      name: tmux

      state: present

We’ll then login to the Tower web interface and go to Credentials in the menu. Next, we’ll click on the green + (Plus) button to create a new credential, give it a name (in this example we use provisioning_root for the name), and set the Credential Type to Machine. We’ll enter root for the Username and set the Password to the same password Satellite is configured to use for new hosts in the Satellite host group (pull up the host group in the Satellite web interface and reference the Operating System tab). This credential will allow Ansible Tower to authenticate to newly provisioned hosts. 

After this, we’ll go to Projects in the Ansible Tower web Interface and click on the green + (Plus) to create a new project. We’ll name the project provision, change the SCM Type to Manual, and select the provision Playbook Directory:

Then we’ll go to Templates in the Ansible Tower web interface, click on the green + (Plus), and select Job Template. We’ll name the template provision, for Inventory select our Satellite inventory source, select provision for the Project, provision.yaml for the Playbook, and provisioning_root for the Credential. We’ll also select the Allow Provisioning Callbacks checkbox and click the wand to generate a Host Config Key.  The Host Config Key is used to ensure that not just anyone can initiate the provisioning callback. Ansible Tower will only run the provisioning callback if it is made with the correct Host Config Key. 

At this point, we’ll want to make a note of the generated Host Config Key as well as the Template ID, which can be determined by looking at the URL of the Template:  

https://tower.example.com/#/templates/job_template/11

In this example, the template has an ID of 11. 

Configuring Satellite for Provisioning Callback

We’ll now move over to the Satellite web interface. Once logged in, go to the Configure menu and select Host Groups. Then select the existing host group used for provisioning; in this example, it is the RHEL 8 host group. 

From the edit host group screen, select the Parameters tab, and create the following 4 new parameters:

ansible_host_config_key,  which should be set to the Host Config Key from our Ansible Tower Template. 

ansible_job_template_id, which should be set to the template ID that we gathered from the template URL earlier. 

ansible_tower_fqdn, which should be set to the fully qualified domain name of the Ansible Tower server.

ansible_tower_provisioning, which should be set to true.

Provisioning a Host to Validate

The final step will be to provision a new host from Satellite and validate the provisioning callback causes our provision playbook to be run automatically on the new host. 

From the Satellite web interface, we’ll go to the Hosts menu, select Create Host, and make sure to use the host group we just added the parameters to (in this example, the RHEL 8 host group). 

Once the host is provisioned, we’ll validate the template successfully ran on the host. To do so, from the Ansible Tower web interface, go to Templates in the menu. The provision template should be listed, and have a green box listed next to it in the leftmost spot:

Click on the green box to see details of the run, which in this example, was successful. Also note that this job was limited to only run on the newly provisioned host. This is expected, as provisioning callbacks only run the template on the host that made the request. 

We’ll also log in to the new host and validate the state of the system matches what is in the playbook:

[root@provision-test-rhel8 ~]# rpm -qa | grep tmux
tmux-2.7-1.el8.x86_64

[root@provision-test-rhel8 ~]# id brian
uid=10000(brian) gid=10000(brian) groups=10000(brian)

[root@provision-test-rhel8 ~]# cat /etc/motd
Authorized use only!

Troubleshooting

If the Ansible Tower template doesn’t automatically run on a newly provisioned host, several things can be checked to troubleshoot what went wrong. 

The first step is to determine if Ansible Tower received the provisioning callback request. To determine this, go to Templates in the Ansible Tower menu, and see if there is a green or red box for the template indicating a successful or failed run, respectively. 

If you see a red box, click on it to see more details on why it failed (which could be related to a syntax issue in the playbook or other issues). 

If you see a green box, click on it to see additional information. The job might have reported as successful, but not actually run. For example, if the playbook’s hosts line specifies specific hosts and groups, and the newly provisioned host isn’t included, the job will report successful. However, if you look at the details it will show skipping: no hosts matched. 

If you don’t see any boxes next to the template, start by checking the parameters defined in the Satellite host group. Verify that the Ansible Tower URL, the host config key, and the template ID are all correct. 

Next, login to the newly provisioned host. If it is a RHEL 7 or RHEL 8 host, it should have a /etc/systemd/system/ansible-callback.service file that was created when the system was provisioned. On RHEL 6 hosts, it should have a /root/ansible_provisioning_call.sh file. 

If you don’t see one of these files, validate that the ansible_tower_provisioning host group parameter was set to true, and that the provisioning templates on the Satellite server haven’t been modified.  

On the newly provisioned host, if we look at the contents of the /etc/systemd/system/ansible-callback.service file, it will show the URL that the system is calling for the provisioning callback:

[root@provision-test-rhel8 ~]# cat /etc/systemd/system/ansible-callback.service
[Unit]
Description=Provisioning callback to Ansible Tower
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/curl -k -s --data "host_config_key=aa5ebe82-491c-4fbb-bd36-a6657549451e" https://tower.example.com/api/v2/job_templates/11/callback/
ExecStartPost=/usr/bin/systemctl disable ansible-callback

[Install]
WantedBy=multi-user.target

We can manually run the curl command listed, which should initiate the provisioning callback:

[root@provision-test-rhel8 ~]# /usr/bin/curl -k -s --data "host_config_key=aa5ebe82-491c-4fbb-bd36-a6657549451e" https://tower.example.com/api/v2/job_templates/11/callback/

If there the host config key is incorrect, an error message will be shown:

​[root@provision-test-rhel8 ~]# /usr/bin/curl -k -s --data "host_config_key=wrong-key-here" https://tower.example.com/api/v2/job_templates/11/callback/

{"detail":"You do not have permission to perform this action."}

If the template ID is incorrect (it was changed from 11 to 43 in this example), it will report:

[root@provision-test-rhel8 ~]# /usr/bin/curl -k -s --data "host_config_key=wrong-key-here" https://tower.example.com/api/v2/job_templates/43/callback/

{"detail":"Not found."}

It can also be helpful to take the -s (silent mode) off the curl command, which will then show if the host name cannot be resolved (it was changed from ansible.example.com to ansible-tower.example.com, which is not a valid hostname in the environment):

​[root@provision-test-rhel8 ~]# /usr/bin/curl -k  --data "host_config_key=wrong-key-here" https://ansible-tower.example.com/api/v2/job_templates/11/callback/

curl: (6) Could not resolve host: ansible-tower.example.com

It can also be helpful to review the /var/log/tower/tower.log file on the Ansible Tower server, which might indicate the cause of the issue.

When an invalid host config key is used, this was logged:

​2019-11-19 23:19:17,371 WARNING  awx.api.generics status 403 received by user AnonymousUser attempting to access /api/v2/job_templates/11/callback/ from 192.168.0.138

When an invalid template ID was used, this was logged:

2019-11-19 23:19:49,093 WARNING  awx.api.generics status 404 received by user AnonymousUser attempting to access /api/v2/job_templates/43/callback/ from 192.168.0.138

2019-11-19 23:19:49,095 WARNING  django.request Not Found: /api/v2/job_templates/43/callback/

Summary and Closing

In conclusion, integrating Satellite and Ansible Tower together is a powerful way to streamline the new host provisioning and configuration process. This can allow system administrators to save time and respond to organizational needs faster. 

• 
• 
• 
•