Hooks
A hook is an executable file that Shell-operator runs when some event occurs. It can be a script or a compiled program written in any programming language. For illustrative purposes, we will use bash scripts. An example with a hook in the form of a Python script is available here: 002-startup-python.
The hook receives the data and returns the result via files. Paths to files are passed to the hook via environment variables.
Shell-operator lifecycle
At startup Shell-operator initializes the hooks:
- The recursive search for hook files is performed in the hooks directory. You can specify it with
--hooks-dir
command-line argument or with theSHELL_OPERATOR_HOOKS_DIR
environment variable (the default path is/hooks
).- Every executable file found in the path is considered a hook (please note,
lib
subdirectory ignored).
- Every executable file found in the path is considered a hook (please note,
- Found hooks are sorted alphabetically according to the directories’ and hooks’ names. Then they are executed with the
--config
flag to get bindings to events in YAML or JSON format. - If hook’s configuration is successful, the working queue named “main” is filled with
onStartup
hooks. - Then, the “main” queue is filled with
kubernetes
hooks withSynchronization
binding context type, so that each hook receives all existing objects described in hook’s configuration. - After executing
kubernetes
hook withSynchronization
binding context, Shell-operator starts a monitor of Kubernetes events according to configuredkubernetes
binding.- Each monitor stores a snapshot - a refreshable list of all Kubernetes objects that match a binding definition.
Next, the main cycle is started:
-
Event handler adds hooks to the named queues on events:
kubernetes
hooks are added to the queue when desired WatchEvent is received from Kubernetes,schedule
hooks are added according to the schedule,kubernetes
andschedule
hooks are added to the “main” queue or the named queue ifqueue
field was specified.
-
Each named queue has its queue handler which executes hooks strictly sequentially. If hook fails with an error (non-zero exit code), Shell-operator restarts it (every 5 seconds) until it succeeds. In case of an erroneous execution of a hook, when other events occur, a queue will be filled with new tasks, but their execution will be blocked until the failing hook succeeds.
- You can change this behavior for a specific hook by adding
allowFailure: true
to the binding configuration (not available foronStartup
hooks).
- You can change this behavior for a specific hook by adding
-
Each hook is executed with a binding context, that describes an already occurred event:
kubernetes
hook receivesEvent
binding context with an object related to the event.schedule
hook receives a name of triggered schedule binding.
-
If there is a sequence of hook executions in a queue, then hook is executed once with array of binding contexts.
- If binding contains
group
key, then a sequence of binding context with similargroup
key is compacted into one binding context.
- If binding contains
-
Several metrics are available for monitoring the activity of the queues and hooks: queues size, number of execution errors for specific hooks, etc. See METRICS for more details.
Hook configuration
Shell-operator runs the hook with the --config
flag. In response, the hook should print its event binding configuration to stdout. The response can be in YAML format:
configVersion: v1
onStartup: ORDER,
schedule:
- {SCHEDULE_PARAMETERS}
- {SCHEDULE_PARAMETERS}
kubernetes:
- {KUBERNETES_PARAMETERS}
- {KUBERNETES_PARAMETERS}
kubernetesValidating:
- {VALIDATING_PARAMETERS}
- {VALIDATING_PARAMETERS}
settings:
SETTINGS_PARAMETERS
or in JSON format:
{
"configVersion": "v1",
"onStartup": STARTUP_ORDER,
"schedule": [
{SCHEDULE_PARAMETERS},
{SCHEDULE_PARAMETERS}
],
"kubernetes": [
{KUBERNETES_PARAMETERS},
{KUBERNETES_PARAMETERS}
],
"kubernetesValidating": [
{VALIDATING_PARAMETERS},
{VALIDATING_PARAMETERS}
],
"settings": {SETTINGS_PARAMETERS}
}
configVersion
field specifies a version of configuration schema. The latest schema version is v1 and it is described below.
Event binding is an event type (one of “onStartup”, “schedule”, “kubernetes” or “kubernetesValidating”) plus parameters required for a subscription.
onStartup
Use this binding type to execute a hook at the Shell-operator’s startup.
Syntax:
configVersion: v1
onStartup: ORDER
Parameters:
ORDER
— an integer value that specifies an execution order. “OnStartup” hooks will be sorted by this value and then alphabetically by file name.
schedule
Scheduled execution. You can bind a hook to any number of schedules.
Syntax
configVersion: v1
schedule:
- crontab: "*/5 * * * *"
allowFailure: true|false
- name: "Every 20 minutes"
crontab: "*/20 * * * *"
allowFailure: true|false
- name: "every 10 seconds",
crontab: "*/10 * * * * *"
allowFailure: true|false
queue: "every-ten"
includeSnapshotsFrom: "monitor-pods"
- name: "every minute"
crontab: "* * * * *"
allowFailure: true|false
group: "pods"
...
Parameters
-
name
— is an optional identifier. It is used to distinguish between multiple schedules during runtime. For more information see binding context. -
crontab
– is a mandatory schedule with a regular crontab syntax with 5 fields. 6 fields style crontab also supported, for more information see documentation on robfig/cron.v2 library. -
allowFailure
— if ‘true’, Shell-operator skips the hook execution errors. If ‘false’ or the parameter is not set, the hook is restarted after a 5 seconds delay in case of an error. -
queue
— a name of a separate queue. It can be used to execute long-running hooks in parallel with other hooks. -
includeSnapshotsFrom
— a list of names ofkubernetes
bindings. When specified, all monitored objects will be added to the binding context in asnapshots
field. -
group
— a key that define a group ofschedule
andkubernetes
bindings. See grouping.
kubernetes
Run a hook on a Kubernetes object changes.
Syntax
configVersion: v1
kubernetes:
- name: "Monitor pods in cache tier"
apiVersion: v1
kind: Pod # required
executeHookOnEvent: [ "Added", "Modified", "Deleted" ]
executeHookOnSynchronization: true|false # default is true
keepFullObjectsInMemory: true|false # default is true
nameSelector:
matchNames:
- pod-0
- pod-1
labelSelector:
matchLabels:
myLabel: myLabelValue
someKey: someValue
matchExpressions:
- key: "tier"
operator: "In"
values: ["cache"]
# - ...
fieldSelector:
matchExpressions:
- field: "status.phase"
operator: "Equals"
value: "Pending"
# - ...
namespace:
nameSelector:
matchNames: ["somenamespace", "proj-production", "proj-stage"]
labelSelector:
matchLabels:
myLabel: "myLabelValue"
someKey: "someValue"
matchExpressions:
- key: "env"
operator: "In"
values: ["production"]
# - ...
jqFilter: ".metadata.labels"
includeSnapshotsFrom:
- "Monitor pods in cache tier"
- "monitor Pods"
- ...
allowFailure: true|false # default is false
queue: "cache-pods"
group: "pods"
- name: "monitor Pods"
kind: "pod"
# ...
Parameters
-
name
is an optional identifier. It is used to distinguish different bindings during runtime. See also binding context. -
apiVersion
is an optional group and version of object API. For example, it isv1
for core objects (Pod, etc.),rbac.authorization.k8s.io/v1beta1
for ClusterRole andmonitoring.coreos.com/v1
for prometheus-operator. -
kind
is the type of a monitored Kubernetes resource. This field is required. CRDs are supported, but the resource should be registered in the cluster before Shell-operator starts. This can be checked withkubectl api-resources
command. You can specify a case-insensitive name, kind or short name in this field. For example, to monitor a DaemonSet these forms are valid:"kind": "DaemonSet" "kind": "Daemonset" "kind": "daemonsets" "kind": "DaemonSets" "kind": "ds"
-
executeHookOnEvent
— the list of events which led to a hook’s execution. By default, all events are used to execute a hook: “Added”, “Modified” and “Deleted”. Docs: Using API WatchEvent. Empty array can be used to prevent hook execution, it is useful when binding is used only to define a snapshot. -
executeHookOnSynchronization
— iffalse
, Shell-operator skips the hook execution with Synchronization binding context. See binding context. -
nameSelector
— selector of objects by their name. If this selector is not set, then all objects of a specified Kind are monitored. -
labelSelector
— standard selector of objects by labels (examples of use). If the selector is not set, then all objects of a specified kind are monitored. -
fieldSelector
— selector of objects by their fields, works like--field-selector=''
flag ofkubectl
. Supported operators are Equals (or=
,==
) and NotEquals (or!=
) and all expressions are combined with AND. Also, note that fieldSelector with ‘metadata.name’ the field is mutually exclusive with nameSelector. There are limits on fields, see Note. -
namespace
— filters to choose namespaces. If omitted, events from all namespaces will be monitored. -
namespace.nameSelector
— this filter can be used to monitor events from objects in a particular list of namespaces. -
namespace.labelSelector
— this filter works likelabelSelector
but for namespaces and Shell-operator dynamically subscribes to events from matched namespaces. -
jqFilter
— an optional parameter that specifies event filtering using jq syntax. The hook will be triggered on the “Modified” event only if the filter result is changed after the last event. See example 102-monitor-namespaces. -
allowFailure
— iftrue
, Shell-operator skips the hook execution errors. Iffalse
or the parameter is not set, the hook is restarted after a 5 seconds delay in case of an error. -
queue
— a name of a separate queue. It can be used to execute long-running hooks in parallel with hooks in the “main” queue. -
includeSnapshotsFrom
— an array of names ofkubernetes
bindings in a hook. When specified, a list of monitored objects from that bindings will be added to the binding context in asnapshots
field. Self-include is also possible. -
keepFullObjectsInMemory
— if not set ortrue
, dumps of Kubernetes resources are cached for this binding, and the snapshot includes them asobject
fields. Set tofalse
if the hook does not rely on full objects to reduce the memory footprint. -
group
— a key that define a group ofschedule
andkubernetes
bindings. See grouping.
Example
configVersion: v1
kubernetes:
# Trigger on labels changes of Pods with myLabel:myLabelValue in any namespace
- name: "label-changes-of-mylabel-pods"
kind: pod
executeHookOnEvent: ["Modified"]
labelSelector:
matchLabels:
myLabel: "myLabelValue"
namespace:
nameSelector: ["default"]
jqFilter: .metadata.labels
allowFailure: true
includeSnapshotsFrom: ["label-changes-of-mylabel-pods"]
This hook configuration will execute hook on each change in labels of pods labeled with myLabel=myLabelValue
in “default” namespace. The binding context will contain all pods with myLabel=myLabelValue
from “default” namespace.
Notes
default Namespace
Unlike kubectl
you should explicitly define namespace.nameSelector
to monitor events from default
namespace.
namespace:
nameSelector: ["default"]
RBAC is required
Shell-operator requires a ServiceAccount with the appropriate RBAC permissions. See examples with RBAC: monitor-pods and monitor-namespaces.
jqFilter
This filter is used to ignore superfluous “Modified” events, and to exclude object from event subscription. For example, if the hook should track changes of object’s labels, jqFilter: ".metadata.labels"
can be used to ignore changes in other properties (.status
,.metadata.annotations
, etc.).
The result of applying the filter to the event’s object is passed to the hook in a filterResult
field of a binding context.
You can use JQ_LIBRARY_PATH
environment variable to set a path with jq
modules.
In case you need to filter by multiple fields, you can use the form of an object or an array:
-
jqFilter: "{nodeName: .spec.nodeName, name: .metadata.labels}"
returns filterResult as object:"filterResult": { "labels": { "app": "proxy", "pod-template-hash": "cfdbfcbb8" }, "nodeName": "node-01" }
-
jqFilter: "[.spec.nodeName, .metadata.labels]"
returns filterResult as array:"filterResult": [ "node-01", { "app": "proxy", "pod-template-hash": "cfdbfcbb8" } ]
Added != Object created
Consider that the “Added” event is not always equal to “Object created” if labelSelector
, fieldSelector
or namespace.labelSelector
is specified in the binding
. If objects and/or namespace are updated in Kubernetes, the binding
may suddenly start matching them, with the “Added” event. The same with “Deleted”, event “Deleted” is not always equal to “Object removed”, the object can just move out of a scope of selectors.
fieldSelector
There is no support for filtering by arbitrary field neither for core resources nor for custom resources (see issue#53459). Only metadata.name
and metadata.namespace
fields are commonly supported.
However fieldSelector can be useful for some resources with extended set of supported fields:
kind | fieldSelector | src url |
---|---|---|
Pod | spec.nodeName spec.restartPolicy spec.schedulerName spec.serviceAccountName status.phase status.podIP status.nominatedNodeName | 1.16 |
Event | involvedObject.kind involvedObject.namespace involvedObject.name involvedObject.uid involvedObject.apiVersion involvedObject.resourceVersion involvedObject.fieldPath reason source type | 1.16 |
Secret | type | 1.16 |
Namespace | status.phase | 1.16 |
ReplicaSet | status.replicas | 1.16 |
Job | status.successful | 1.16 |
Node | spec.unschedulable | 1.16 |
Example of selecting Pods by ‘Running’ phase:
kind: Pod
fieldSelector:
matchExpressions:
- field: "status.phase"
operator: Equals
value: Running
fieldSelector and labelSelector expressions are ANDed
Objects should match all expressions defined in fieldSelector
and labelSelector
, so, for example, multiple fieldSelector
expressions with metadata.name
field and different values will not match any object.
kubernetesValidating
Use a hook as handler for ValidationWebhookConfiguration.
See syntax and parameters in BINDING_VALIDATING.md
kubernetesCustomResourceConversion
Use a hook as handler for custom resource conversion.
See syntax and parameters in BINDING_CONVERSION.md
Binding context
When an event associated with a hook is triggered, Shell-operator executes the hook without arguments. The information about the event that led to the hook execution is called the binding context and is written in JSON format to a temporary file. The path to this file is available to hook via environment variable BINDING_CONTEXT_PATH
.
Temporary files have unique names to prevent collisions between queues and are deleted after the hook run.
Binging context is a JSON-array of structures with the following fields:
binding
— a string from thename
parameter. If this parameter has not been set in the binding configuration, then strings “schedule” or “kubernetes” are used. For a hook executed at startup, this value is always “onStartup”.type
— “Schedule” forschedule
bindings. “Synchronization” or “Event” forkubernetes
bindings. “Group” ifgroup
is defined.
The hook receives “Event”-type binding context on Kubernetes event and it contains more fields:
watchEvent
— the possible value is one of the values you can use withexecuteHookOnEvent
parameter: “Added”, “Modified” or “Deleted”.object
— a JSON dump of the full object related to the event. It contains an exact copy of the corresponding field in WatchEvent response, so it’s the object state at the moment of the event (not at the moment of the hook execution).filterResult
— the result ofjq
execution with specifiedjqFilter
on the above mentioned object. IfjqFilter
is not specified, thenfilterResult
is omitted.
The hook receives existed objects on startup for each binding with “Synchronization”-type binding context:
objects
— a list of existing objects that match selectors in binding configuration. Each item of this list containsobject
andfilterResult
fields. The state of items is actual for the moment of the hook execution. If the list is empty, the value ofobjects
is an empty array.
If group
or includeSnapshotsFrom
are defined, the hook receives binding context with additional field:
snapshots
— a map that contains an up-to-date lists of objects for each binding name fromincludeSnapshotsFrom
or for eachkubernetes
binding with a similargroup
. IfincludeSnapshotsFrom
list is empty, the field is omitted.
onStartup
binding context example
Hook with this configuration:
configVersion: v1
onStartup: 1
will be executed with this binding context at startup:
[{"binding": "onStartup"}]
schedule
binding context example
For example, if you have the following configuration in a hook:
configVersion: v1
schedule:
- name: incremental
crontab: "0 2 */3 * * *"
allowFailure: true
then at 12:02, it will be executed with the following binding context:
[{ "binding": "incremental", "type":"Schedule"}]
kubernetes
binding context example
A hook can monitor Pods in all namespaces with this simple configuration:
configVersion: v1
kubernetes:
- kind: Pod
“Synchronization” binding context
During startup, the hook receives all existing objects with “Synchronization”-type binding context:
[
{
"binding": "kubernetes",
"type": "Synchronization",
"objects": [
{
"object": {
"kind": "Pod",
"metadata":{
"name":"etcd-...",
"namespace":"kube-system",
...
},
}
},
{
"object": {
"kind": "Pod",
"metadata": {
"name": "kube-proxy-...",
"namespace": "kube-system",
...
},
}
},
...
]
}
]
Note: hook execution at startup with “Synchronization” binding context can be turned off with
executeHookOnSynchronization: false
“Event” binding context
If pod pod-321d12
is then added into namespace ‘default’, then the hook will be executed with the “Event”-type binding context:
[
{
"binding": "kubernetes",
"type": "Event",
"watchEvent": "Added",
"object": {
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "pod-321d12",
"namespace": "default",
...
},
"spec": {
...
},
...
}
}
]
Snapshots
“Event”-type binding context contains an object state at the moment of the event. Actual objects’ state for the moment of the execution can be received in a form of Snapshots.
Shell-operator maintains an up-to-date list of objects for each kubernetes
binding. schedule
and kubernetes
bindings can be configured to receive these lists via includeSnapshotsFrom
parameter. Also, there is a group
parameter to automatically receive all snapshots from multiple bindings and to deduplicate executions.
Snapshot is a JSON array of Kubernetes objects and corresponding jqFilter results. To access the snapshot during the hook execution, there is a map snapshots
in the binding context. The key of this map is a binding name, and the value is the snapshot.
snapshots
example:
[
{ "binding": ...,
"snapshots": {
"binding-name-1": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
]
}}]
object
— a JSON dump of Kubernetes object.filterResult
— a JSON result of applyingjqFilter
to the Kubernetes object.
Keeping dumps for object
fields can take a lot of memory. There is a parameter keepFullObjectsInMemory: false
to disable full dumps.
Note that disabling full objects make sense only if jqFilter
is defined, as it disables full objects in snapshots
field, objects
field of “Synchronization” binding context and object
field of “Event” binding context.
For example, this binding configuration will execute hook with empty items in objects
field of “Synchronization” binding context:
kubernetes:
- name: pods
kinds: Pod
keepFullObjectsInMemory: false
Snapshots example
To illustrate the includeSnapshotsFrom
parameter, consider the hook that reacts to changes of labels of all Pods and requires the content of the ConfigMap named “settings-for-my-hook”. There is also a schedule to do periodic checks:
configVersion: v1
schedule:
- name: periodic-checking
crontab: "0 */3 * * *"
includeSnapshotsFrom: ["monitor-pods", "configmap-content"]
kubernetes:
- name: configmap-content
kind: ConfigMap
nameSelector:
matchNames: ["settings-for-my-hook"]
executeHookOnSynchronization: false
executeHookOnEvent: []
- name: monitor-pods
kind: Pod
jqFilter: '.metadata.labels'
includeSnapshotsFrom: ["configmap-content"]
This hook will not be executed for events related to the binding “configmap-content”. executeHookOnSynchronization: false
accompanied by executeHookOnEvent: []
defines a “snapshot-only” binding. This is one of the techniques to reduce the number of kubectl
invocations.
“Synchronization” binding context with snapshots
During startup, the hook will be executed with the “Synchronization” binding context with snapshots
and objects
:
[
{
"binding": "monitor-pods",
"type": "Synchronization",
"objects": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
"labels": { ... },
...
},
},
"filterResult": {
"label1": "value",
...
}
},
{
"object": {
"kind": "Pod",
"metadata": {
"name": "kube-proxy-...",
"namespace": "kube-system",
...
},
},
"filterResult": {
"label1": "value",
...
}
},
...
],
"snapshots": {
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
]
“Event” binding context with snapshots
If pod pod-321d12
is then added into the “default” namespace, then the hook will be executed with the “Event” binding context with object
and filterResult
fields:
[
{
"binding": "monitor-pods",
"type": "Event",
"watchEvent": "Added",
"object": {
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "pod-321d12",
"namespace": "default",
...
},
"spec": {
...
},
...
},
"filterResult": { ... },
"snapshots": {
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
]
“Schedule” binding context with snapshots
Every 3 hours, the hook will be executed with the binding context that include 2 snapshots (”monitor-pods” and “configmap-content”):
[
{
"binding": "periodic-checking",
"type": "Schedule",
"snapshots": {
"monitor-pods": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
],
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
"metadata": {"name": "settings-for-my-hook", ... },
"data": {"field1": ... }
}
}
]
}
}
]
Binding context of grouped bindings
group
parameter defines a named group of bindings. Group is used when the source of the event is not important, and data in snapshots is enough for the hook. When binding with group
is triggered with the event, the hook receives snapshots from all kubernetes
bindings with the same group
name.
Adjacent tasks for kubernetes
and schedule
bindings with the same group
and queue
are “compacted”, and the hook is executed only once. So it is wise to use the same queue
for all hooks in a group. This “compaction” mechanism is not available for kubernetesValidating
and kubernetesCustomResourceConversion
bindings as they’re not queued.
executeHookOnSynchronization
, executeHookOnEvent
and keepFullObjectsInMemory
can be used with group
. Their effects are as described above for non-grouped bindings.
group
parameter is compatible with includeSnapshotsFrom
parameter. includeSnapshotsFrom
can be used to include additional snapshots into binding context.
Binding context for a group contains:
binding
field with the original binding name or, if the name field wasn’t set in the binding configuration, then strings “schedule” or “kubernetes” are used.type
field with the value “Group”.snapshots
field if there is at least onekubernetes
binding in the group orincludeSnapshotsFrom
is not empty.
Group binding context example
Consider the hook that is executed on changes of labels of all Pods, changes in ConfigMap’s data and also on schedule:
configVersion: v1
schedule:
- name: periodic-checking
crontab: "0 */3 * * *"
group: "pods"
kubernetes:
- name: monitor-pods
apiVersion: v1
kind: Pod
jqFilter: '.metadata.labels'
group: "pods"
- name: configmap-content
apiVersion: v1
kind: ConfigMap
nameSelector:
matchNames: ["settings-for-my-hook"]
jqFilter: '.data'
group: "pods"
binding context for grouped bindings
Grouped bindings is used when only the occurrence of an event is important. So, the hook receives actual state of Pods and the ConfigMap on every of these events:
- During startup.
- A new Pod is added.
- The Pod is deleted.
- Labels of the Pod are changed.
- ConfigMap/settings-for-my-hook is deleted.
- ConfigMap/settings-for-my-hook is added.
- Data field is changed in ConfigMap/settings-for-my-hook.
- Every 3 hours.
Binding contexts for these events will be pretty the same, except for the binding
field, as it will be equal to the corresponding name
field of the binding:
[
{
"binding": "monitor-pods",
"type": "Group",
"snapshots": {
"monitor-pods": [
{
"object": {
"kind": "Pod",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
],
"configmap-content": [
{
"object": {
"kind": "ConfigMap",
"metadata": {
"name": "etcd-...",
"namespace": "kube-system",
...
},
},
"filterResult": { ... },
},
...
]
}
}
]
settings
An optional block with hook-level settings.
Syntax
configVersion: v1
settings:
executionMinInterval: 3s
executionBurst: 1
Parameters
executionMinInterval
defines a minimum time between hook executions.executionBurst
a number of allowed executions during a period.
Execution rate
executionMinInterval
and executionBurst
are parameters for “token bucket” algorithm. These parameters are used to throttle hook executions and wait for more events in the queue. It is wise to use a separate queue for bindings in such a hook, as a hook with execution rate settings and with default (”main”) queue can hold the execution of other hooks.
Example
configVersion: v1
kubernetes:
- name: "all-pods-in-ns"
kind: pod
executeHookOnEvent: ["Modified"]
namespace:
nameSelector: ["default"]
queue: handle-pods-queue
settings:
executionMinInterval: 3s
executionBurst: 1
If the Shell-operator will receive a lot of events for the “all-pods-in-ns” binding, the hook will be executed no more than once in 3 seconds.