Fundamental concepts

The descriptive approach

Declarative vs Imperative

In the past, most automation tools have used an imperative approach, often called a scripts & workflows approach to automation. In the imperative world, the user writes code to script tasks that have to be performed on the target systems and the automation tool distributes, executes and verifies their proper execution. As an example, a simple script could be

install(httpd);
start(httpd);

ComodIT is part of a new generation of tools that tackle automation with a descriptive approach. In the descriptive world, the user writes specification of what is required on his target system, and let the automation engine bring the system is in the correct state. Our previous example could be rewritten in a descriptive style:

package[httpd]: present
service[httpd]: enabled, running

You may think that the difference between the imperative and the descriptive style is purely cosmetic. There is however much more than a style difference. In the case of the script based approach, the content of the script is opaque to the engine. The automation tool has absolutely no knowledge of the fact that a package has to be installed or a service has to be started. It just knows that, at some point in time, it has to execute a sequence of actions on the target host.

On the other hand, the descriptive approach actually describes to the automation tool what the user wants. The tools knows that a package has to be present and a service has to be running. It can therefore take all required actions to ensure that the system is in the correct state (this is usually referred to as convergence in the literature). If someone inadvertently stop the service, the automation engine can detect that the host in in an incorrect state and bring it back to the defined state (this is usually referred to as ensuring compliance).

It may not always be possible to achieve the desired result using only a descriptive approach, so as a last resort, ComodIT leaves the possibility to the user to also provide scripts when absolutely necessary. It is however our goal to keep on improving the description language and its features to reduce the need for scripts to its absolute minimal.

Generalizing by grouping resources in a Template

We have just seen that a declarative approach can be used to describe the expected state of an infrastructure. For each hosts in his infrastructure, a user can now describe the resources required on the host and let the automation engine take care of the deployment. There may be however a lot of duplication since multiple hosts may be running the same applications.

This is why we introduce the concept of application templates. By bundling resources as a an application template, the user can easily reuse the same description of resources across different hosts. It becomes possible to build a catalog of useful applications and quickly provision a new host with some of them. Other tools in the configuration management space refer to this generalization as ‘classes’ or ‘recipes’, we have chosen the term ‘template’

As an example, the following is the description of the required resources for an apache server on a Redhat like system, using the ComodIT JSON based descriptive language. Keep in mind that when you are using the web interface, you won’t see any of this. The wizards are there to help you build this description easily and without having to worry about the underlying grammar.

{
   "name": "Apache2",
   "description": "Simple Apache 2 setup with mod_ssl",
    "packages": [
       {"name": "httpd"},
       {"name": "mod_ssl"}
   ],
   "files": [
       {
           "group": "root",
           "mode": "644",
           "name": "httpd.conf",
           "owner": "root",
           "path": "/etc/httpd/conf/httpd.conf"
        }
   ],
   "services": [
       {
           "name": "httpd",
           "enabled": "true"
       }
   ]
}

This application template describes the fact that two packages have to be installed, one service must be enabled (i.e. start on boot) and a configuration file (httpd.conf) has to be placed at the right location with proper user/group/mode. The content of the file httpd.conf is provided as an attachment to the application template.

As you can see, an additional benefit of this approach is that the user does not need to describe how to install a package or enable a service. It will be the responsibility of the automation engine to know how to bring the system in the required state. In this example, ComodIT will know that in the case of a Centos 6.2 it should use chkconfig to enable the service, but on a Fedora 16 it will use the new sysctl approach.

Generalizing even more using Parameters

We have seen that application templates enable to quickly reuse a description of resources across different hosts. In most cases however, the user will need to change part of this template, especially making changes in the configuration files. This is why we use the name ‘application template’ and also introduce the concept of ‘application parameters’. A parameter is a variable for which a value can be assigned when deploying the application on a host. The value assigned to the parameter can be used within the application template, and especially inside the provided configuration file.

The following is an example of adding a ‘Http Port’ parameter to the Apache 2 application we have seen before. It will require two modifications. The first one is to declare the parameter in the application template, providing a name, description and default value.

{
     "name": "Apache2",
   "description": "Simple Apache 2 setup with mod_ssl",

        …...


    "parameters": [
        {
            "name": "Httpd Port", 
            "key": "httpd_port", 
            "description": "Allows you to bind Apache to specific IP addresses and/or ports", 
            "value": "80"
        }
     ]
}

The second change is to actually use the parameter within one (or more) of the application configuration files. Below is a snippet of the httpd.conf template associated with the application template.

## Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#

Listen ${httpd_port}

When Parameters become complex and arrays

There may be cases where exposing parameters as simple variables with a string value is not sufficient. One may want to provide arrays of complex objects as value for a parameter. For example to configure rules in a firewall, or to easily manage a variable list of vhosts. To support these use cases, we allow parameters to provide a schema describing the format of the expected values. As an example, the vhosts parameter used in a Apache configuration file could look like the one below.

{
   "name": "Vhosts", 
    "description": "A list of vhosts to configure on this web server", 
   "key": "vhosts", 
   "schema": {
        "type": "array",
       "items": {
            "type": "object",
           "properties": {
                "name": {
                   "description": "Name of the vhost", 
                   "pretty": "Name", 
                   "required": true, 
                   "type": "string",
                    "index": true
               }, 
                "hostname": {
                   "description": "Hostname for the vhost", 
                   "pretty": "Hostname", 
                   "required": true, 
                   "type": "string"
               }, 
               "document_root": {
                   "description": "Path to the document root", 
                   "pretty": "Document Root", 
    "default": "/var/www/html/", 
                   "required": true, 
                   "type": "string"
               }, 
               "enable_index": {
    "description": "Enable Index option for this host", 
                   "pretty": "Index+", 
    "type": "boolean", 
    "value": "false", 
                   "advanced": true
               }
           } 
       }
   }
}

It becomes then possible to use this parameter within a template to render a variable list of vhosts based on the value provided by the user when adding the application to a host. The following is an example on how to use this parameter within a file template.

<#list vhosts as vhost>
<VirtualHost ${vhost.hostname}:80>
      ServerAdmin webmaster@${vhost.hostname}
      DocumentRoot ${vhost.document_root}
      ServerName ${vhost.hostname}
</VirtualHost>
</#list>

To assign a value to a parameter when adding an application to a host, the user must specify an application context that contains a set of settings. The following is an example of a user configuring the vhosts when using the Apache 2 template described so far.

{
   "application": "Apache2",
   "settings" : [
       {
           "key": "vhosts",
           "value": [
                {
                             "name": "comodit",
                     "hostname": "comodit.com",
                     "document_root": "/var/www/comodit/"
                },
                {
                             "name": "default",
                     "hostname": "example.com",
                     "document_root": "/var/www/html/"
                }
]
        }
   ]
}

Beyond applications: platforms and distributions

One need more than basic resources and applications templates to describe a complete infrastructure. We must also provide the means to describe the machine themselves (physical or virtual) and their operating system. Only then are we in a position to fully describe an infrastructure and deploy it automatically.

To this extent, ComodIT has the concept of Platform Template and Distribution Template. Just like for Application Templates, these describe the resources required to create the machines and deploy/install the operating system. They also support parameters to enable customization on a per host basis.

Describing a platform

A platform is linked to an underlying driver that provides the logic to manage the platform. It configures the driver with a set of settings and configuration files. These files can themselves be templates using parameters exposed by the platform.

As an example, the following is the description of a KVM hypervisor accessed through the libvirt driver. The only setting required by the driver is the libvirt connect url. Default libvirt templates are used, they can however be overwritten by attaching new templates to the platform.

{
    "name": "KVM 1",
    "description": "Our local KVM hypervisor",
    "driver": "Libvirt",
    "settings": [
        {
            "key": "libvirt.connectUrl",
            "value": "qemu:///kvm.local/system",
        }
    ]
}

Describing a distribution

A distribution describes the resources required to setup a machine with a given operating system. It consists of various settings and additional files. The settings depend on the way the operating system is provisioned (gold image, unattended install, boot from iso, etc.). As an example, below is the description of a Centos 6.2 distribution to be installed through kickstarting. The kickstart template is an additional file attached to the description.

{
    "name": "Centos 6.2", 
    "description": "Centos 6.2 distribution.", 
    "files": [
        {
            "name": "kickstart.ks"
        }
    ], 
    "settings": [
        {
            "key": "initrd_path", 
            "value": "centos/6/x86_64/initrd.img", 
        },
        {
            "key": "vmlinuz_path", 
            "value": "centos/6/x86_64/vmlinuz", 
        }, 
        {
            "key": "kernel_params", 
     "value":"text ks = [.. long url...]/distribution/files/kickstart.ks"
        }
    ], 
}

Summary

In this section, we have seen that a descriptive approach is more suitable than an imperative one for an automation tool. We have then looked at how ComodIT enables to describe the various resources required in an infrastructure, abstract the description of the underlying platform, modularize groups of resources into templates, and make these more flexible using parameters. Finally we have seen how dependencies between different applications can be captured in the description.

These templates can then be assembled as building blocks to form the description of a complete infrastructure, from platform to operating system and applications. Together with their custom configuration. We have designed ComodIT in such a way that having this description is sufficient to (re)-deploy the complete infrastructure.

This complete description of an infrastructure is what we call the ‘system state’. In the next section, we explore in deeper details this concept of system state. What does it mean ? How does it evolve when changes are made to the configuration ? How can we propagate its changes to the underlying infrastructure ?

The system state

What you see is what you (eventually) get

In the previous section, we have seen that ComodIT descriptive approach enable you to precisely describe a complete infrastructure, including the machine requirements, the operating system, the applications with their configurations and their dependencies. This knowledge is sufficient for ComodIT to (re)-deploy your complete infrastructure automatically. The database containing that knowledge is what we call the system state.

In ComodIT, the system state is thus a complete description of the state in which you want the system to be. The goal of ComodIT is to keep the infrastructure in sync with the changes applied to the system state. Thus when you make a configuration change to the system state (e.g. adding an application on a host), ComodIT will perform the actions required to bring the underlying infrastructure back in sync with the defined system state. We call this process convergence.

There may also be cases where the infrastructure diverges from the required state. For example a service is stopped, a configuration file has been manually changed or a package is updated to the wrong version. ComodIT detects these changes and will propose you to recover to the defined state. We call this process compliance.

The process of convergence and compliance are executed at the host level in ComodIT. A host can evolve through the following five states:

  1. defined: the host is defined in the system state but does not yet exist in the infrastructure. From this state one can go into the provisioning state by asking ComodIT to provision the host.

  2. provisioning: ComodIT is in the process of deploying an instance of the host, following the specification in the system state database. When the provisioning is achieved, the host transition to the ready state.

  3. ready: the instance is ready, and complies to the host definition in the system state. If the instance is deleted, the host moves back to the defined state. If a change is made to the host configuration, the state moves to updating while the changes are processed. If a resource diverges from the required state, the host moves to the diverged state.

  4. updating: a change has been made to the host definition (e.g. application has been added, a setting has been changed) and ComodIT is busy executing the required action to bring the instance back in the ready state. When successful, the host transition back into the compliant state. If an error occurs, the host state is moving into diverged to indicate that the reality does not match the defined state.

  5. diverged: the instance has diverged from the host defined in the system state. Either because an error occurred when converging, or something happened on the instance.

Achieving convergence

When the configuration of a host is changed in the system state, we have to ensure that the corresponding instance is updated to bring it in a state matching the one defined. ComodIT achieves this by sending a sequence of messages to an agent running on the machine. The messages instruct the agent to updates one or more resources. For details on how these messages are sent, have a look at the ‘Architecture’ section.

We are thus using a ‘push’ approach to updates. This means that messages are sent as soon as possible to the target machine, without having to wait for the machine to ‘poll’ for updates. In most case, this will lead to near real-time update of the machine state. There may be cases however when the machine is unavailable (offline, high load, etc.). When this happen, the messages are queued and delivered as soon as possible. When all changes are processed, ComodIT is informed by the agent and can revert the state of the host to ‘ready’, if everything went smooth, or to ‘diverged’ if some error occurred.

Just updating the resources to match the new definition may however not be sufficient on a running system. In most cases, updating a service configuration file is not sufficient for the changes to take effect. The service would usually need a restart/reload or the machine may need a full reboot for the changes to take effect. On a running system, we therefore have to go beyond convergence and ensure that appropriate actions are taken for the changes to take effect.

In ComodIT, we achieve this using handlers which are executed on top of the convergence process. Using handlers, an application architect can instruct ComodIT to perform additional tasks following a configuration change. For example, reloading a service following a setting changing a setting in a configuration file may require a reload of a service, while changing another setting in the same file would require a restart.

The following is an example of the handler defined for our Apache 2 application. As you can see, the handler is written in a platform agnostic way (just like for the resources). The hanlder instruct ComodIT to restart the httpd service when some settings are updated, it does not have to tell how the service is restarted.

{
    "name": "Apache2",
   "description": "Simple Apache 2 setup with mod_ssl",

        …...
    "handlers": [
        {
            "do": [


                {
                    "action": "restart", 
                    "resource": "service://httpd"
                }
            ], 
            "on": [
                "httpd_port"
            ]
        }
    ]
}

Keeping it compliant

There may be situations when resources on a host are changed through other means than ComodIT: user actions via ssh, automated updates, an application behavior, etc. What if such a change modifies a resource that was supposed to be in a well defined state managed by ComodIT ? When this happen, we say that the host has diverged and is not anymore in compliance with the definition.

The agent running on the managed instance is responsible for monitoring the resources under its control. When it detects a divergence from the defined state it sends an alert to ComodIT that changes the state of the host to ‘diverged’ and shows to the user which resources have changed.

It is then possible for the user to attempt a recovery, asking ComodIT to reset the resources to their defined state.

Summary

In this section, we have seen the central role played by the system state to guarantee the compliance of the underlying infrastructure. Through its convergence process, ComodIT attempts to keep the resources in sync with their definition. It also ensure compliance of these resources throughout their lifetime.

For this approach to be effective, every changes to a host configuration should go through ComodIT, so that the system state is properly updated. This means that one should let go the SSH way of managing hosts. Using ComodIT for all administration tasks require a change of habits but brings many benefits, beyond the compliance and convergence we have seen so far. The next section explores what ‘managing through the API’ means for the sysadmin and the users.

SSH is deprecated

ComodIT is based on two fundamental concepts. The first one is that, given the right tools, it is possible to describe entirely an infrastructure, from the (virtual) hardware level to the configuration details of applications. This description can then be used to automate tasks, eliminating ad-hoc manual processes. The second is that this description can be centralized in a system state and kept in sync with the underlying infrastructure. A change in the definition is automatically applied to the real thing (convergence) and a change in a real resource is detected and potentially automatically reverted (compliance).

These two concepts are what make ComodIT so powerful. They enable all the great features such as one click provisioning, configuration management, up-to-date documentation, compliance, auditing, etc. For this to truly work however, one must do as much as possible with ComodIT. The only way to maintain all documentation inside the system state is to actually use the system state for managing one’s infrastructure.

There is therefore a third fundamental concept in the ComodIT philosophy that is emerging from the first two: ssh is deprecated. In order to benefit from the descriptive approach and the system state, one must let got the manual management of hosts through ssh. However we know that whatever the benefits, users will go this route only if the tool makes their life easier, solving the pain of daily tasks instead of making it worse. We therefore put a lot of efforts in making sure that it is the case.

Managing through an API

Instead of executing commands on a host, someone using ComodIT ultimately manipulates resources through an API. The API enables to read/add/edit/delete all resources through simple HTTP requests that can be easily executed in pretty much any language.

Of course we don’t expect users and sysadmins to spend their days crafting http requests to manage their infrastructure, this is why we’ve built an intuitive web frontend and a command line client on top of this API. The point to keep in mind however is that all these different user interface are using the same API. In case of problems, or complex use cases, going back to the documentation of the API will help you.

The following is an example of a web request to update the value of the httpd_port parameter of the Apache 2 application deployed on a host. As you can see it is simply an authenticated HTTP Put request with the appropriate JSON description of the new setting. All other actions using the API have a similar flow.

curl --user <username>:<password> 
     -v 
     -X PUT 
     -d '{"key":"httpd_port","value":"8123"}' 
     -H"Content-Type: application/json"  http://example.com/api/organizations/Acme/environments/Prod/hosts/myhost/applications/Apache2/settings/httpd_port

The same use case can easily be executed through the command line. The dynamic auto-completion makes it easy to fill in the name of the organization, environment and host.

#comodit-agent hosts applications settings update Acme Prod myhost Apache2 httpd_port

Whether you are executing the previous example with the web interface, the command line client or the API, the result is the same: the value of the httpd_port setting is changed for the given host inside the system state. ComodIT will then attempt to modify the required resources on the host itself (httpd.conf in this case) and execute any handlers attached to the event (the one to restart the service).

Therefore, more than managing through the API, you are actually managing the host using the settings and handlers defined in the application template.

Users and Architects

For a system administrator, it may seems cumbersome or complicated to use the API and applications recipes to change a setting like we have just seen. After all, opening httpd.conf in vim (or emacs :-) and executing a service restart is quick and painless operation. Why go through the trouble of writing an application recipe and using a client to make simple changes to a host ?

The key point to keep in mind is that the system administrator is not the one who should make this change. When the application templates are properly defined, the user of the service can be the one making his configuration changes himself. ComodIT enables a ‘self-service’ IT helpdesk where users can manage their hosts, applications and configurations alone, thus giving more time to system administrator to focus on writing great application templates that can accommodate their users needs.

We therefore see two distinct roles in using ComodIT:

  1. application architects are responsible for configuring the ‘raw material’ available on ComodIT. They setup the platforms, define operating system distributions and how to install them, and prepare application templates that exposes essential parameters to their users.

  2. users don’t need knowledge on how things are done, they just use ComodIT to deploy new hosts with the application they need. They can configure some aspects of their deployments thanks to the parameters defined by the application architects.

In the future, it will be possible for users to browse the ComodIT application store and acquire (for free or a fee) applications and their templates, making it easy to benefit of well-thought templates.

Summary

To obtain the greatest value out of using ComodIT, one must ensure that all key system administration tasks occurs through ComodIT, keeping the system state up-to-date with the infrastructure requirements. This means stopping the manual ad-hoc way of doing things through ssh and instead embracing the API and Template based approach.

This require a change in the way the infrastructure is managed. Decoupling between an architect who focus on writing great templates exposing key parameters and a user actually leveraging these templates to deploy and operate the needed infrastructure.