Lifecycle

Structure

Module files are located in the /modules directory. The directory can be set via $MODULES_DIR variable. Global hook files are located in the /global-hooks directory (you can set your own directory with the $GLOBAL_HOOKS_DIR variable).

Startup sequence

During startup, Addon-operator finds and initializes all global hooks. For more info, see HOOKS.

After the global hooks initialization, Addon-operator executes all global onStartup hooks.

Then, the global hooks with kubernetes binding are executed with a binding context of type Synchronization and Kubernetes monitors for global hooks are started.

Reload all modules

Next, the ‘reload all modules’ process is started. First, it finds all modules and their hooks.

Then, all global hooks with beforeAll binding are executed.

Next, the ‘module discovery’ process is started, it finds which modules are enabled by executing ‘enabled’ script for modules enabled in values.yaml and in ConfigMap/addon-operator.

Enabled modules are started.

During each module start-up, it executes all onStartup hooks and initializes the installation of a Helm chart. Prior to the installation of a Helm chart, the beforeHelm hook is executed. The afterHelm hook is executed after the installation.

When all modules are started, all global hooks with afterAll binding are executed.

Main loop

After the first run of ‘reload all modules’, the main loop starts. It reacts to schedule and Kubernetes events, and to a values changes: it restarts a particular module if its values are changed and runs ‘reload all modules’ process again if global values are changed.

Named queues

The Addon-operator supports named queues to execute hooks in parallel for schedule and kubernetes/Event bindings.

All other actions are handled in a single “main” queue:

  • global hooks:
    • onStartup
    • kubernetes/Synchronization
    • beforeAll
    • afterAll
  • module hooks:
    • onStartup
    • kubernetes/Synchronization
    • beforeHelm
    • execution of helm commands
    • afterHelm

This document mainly describes modules. To get more information on hooks, see HOOKS document. To get a full view of how hooks, modules, values, binding contexts, and queues are interlinked, see LIFECYCLE-STEPS document.

Module lifecycle

The onStartup hooks of enabled module is executed at the startup of the Addon-operator or later on module enablement.

Next, the module’s chart is installed with helm upgrade --install. Before launching Helm, beforeHelm hooks are executed, after the launch, afterHelm hooks are executed.

After the launch the module would start responding to two types of events:

  • schedule — events that are generated by the crontab scheduler built in the addon-operator;
  • kubernetes — events within the cluster that API server announces to the Addon-operator.

When the module is deactivated, the Addon-operator launches command helm delete --purge and after the release deletion, the afterDeleteHelm hooks are executed.

All necessary hooks will be restarted if there are errors during the module activation or deactivation. For example, if an error occurred in the hook with afterHelm binding during the first module execution, then after a 5 seconds delay the onStartup and beforeHelm hooks are executed, the Helm chart is installed and then afterHelm hooks are executed.

Modules discovery

The Addon-operator makes a list of all enabled modules for their execution and a list of disabled modules for the deletion of their Helm releases. This process is called ‘modules discovery’ and is started in the following cases:

  • during the start of addon-operator
  • when an event to restart all modules occurs (see VALUES).

Modules are disabled by default. The module can be enabled by a key with the module name suffixed by Enabled. This key should contain a boolean value and can be specified in these sources:

  • $MODULES_DIR/values.yaml
  • values.yaml files in modules directories
  • ConfigMap/addon-operator

Boolean values from values.yaml files and ConfigMap/addon-operator are combined and if the result is equal to false or is empty, then the module is disabled.

If the value is true, an additional check is performed – the enabled script is executed (see below). If the script is present in the module and it returns false, then the module is considered disabled. If the script is not present or returns true, then the module is enabled.

If an error occurs during the ‘modules discovery’ process, then the module discovery is restarted every 5 seconds until successful execution. In this case, the execution of hooks with schedule and kubernetes bindings will be blocked in the “main” queue.

As a result of a ‘module discovery’ process, the tasks for the execution of all enabled modules, deletion of all disabled modules, and execution of all global hooks with the afterAll binding are added to the queue.

Enabled script

A script or an executable file that returns the status of the module. The script has access to the module values in $VALUES_PATH and $CONFIG_VALUES_PATH files, more details about the values are available here. The variable $MODULE_ENABLED_RESULT passes the path to the file into which the script should write the module status: true or false.

Below is an example of the enabled script that disables the module when parameter param2 is set to “stopMePlease”.

#!/usr/bin/env bash

param2=$(jq -r '.simpleModule.param2' $VALUES_PATH)

if [[ $param2 == "stopMePlease" ]] ; then
  echo "false" > $MODULE_ENABLED_RESULT
else
  echo "true" > $MODULE_ENABLED_RESULT
fi

Examples

Keys in values.yaml files

A module named nginx-ingress may have an nginxIngressEnabled flag in two files:

$ cat modules/values.yaml

nginxIngressEnabled: true

$ cat modules/001-nginx-ingress/values.yaml

nginxIngressEnabled: false

Module nginx-ingress is enabled in modules/values.yaml but disabled in modules/001-nginx-ingress/values.yaml. The final result is that the module is disabled.

Also, note that the module’s directory name is kebab-cased but keys in values.yaml are camelCased (see VALUES).

values.yaml and ConfigMap

A module named ‘some-module’ has no someModuleEnabled flag in modules/001-some-module/values.yaml but this flag is defined in a ConfigMap and the module has enabled script:

$ cat modules/values.yaml

global:
  param1: 100
someModuleEnabled: false

$ cat modules/001-some-module/values.yaml

someModule:
  param1: "String"


$ kubectl -n addon-operator get cm/addon-operator -o yaml

data:
  global: |
    param1: 200
  someModule: |
    param1: "Long string"
    param2: "FOO"
  someModuleEnabled: "true"

$ cat modules/01-some-module/enabled

#!/bin/bash

echo false > $MODULE_ENABLED_RESULT

Module some-module is explicitly disabled in modules/values.yaml but enabled by someModuleEnabled key in ConfigMap/addon-operator. Thus enabled script is executed and returns false. So the final result is that the module is disabled.

Task queues

Task queues are simple FIFO queues. The Addon-operator processes an event, creates a task and adds it to the particular named queue. Each named queue has a queue handler which runs the first task and proceeds to the next.

Each task is processed until successful completion. In case of an error, the task is returned to the start of the queue and executed with an exponentially growing delay (from 5s to 30s). When executing tasks for the kubernetes and schedule events, the queue handler ignores execution errors if the allowFailure: true flag is specified in the binding configuration.

Queue monitoring

You can use Prometheus metrics to monitor the queue. For details, see METRICS.