|
|
c17bfd |
diff --git a/source/configuration/modules/mmkubernetes.rst b/source/configuration/modules/mmkubernetes.rst
|
|
|
c17bfd |
new file mode 100644
|
|
|
c17bfd |
index 0000000..1cd3d2a
|
|
|
c17bfd |
--- /dev/null
|
|
|
c17bfd |
+++ b/source/configuration/modules/mmkubernetes.rst
|
|
|
c17bfd |
@@ -0,0 +1,378 @@
|
|
|
c17bfd |
+*****************************************
|
|
|
c17bfd |
+Kubernetes Metadata Module (mmkubernetes)
|
|
|
c17bfd |
+*****************************************
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+=========================== ===========================================================================
|
|
|
c17bfd |
+**Module Name:** **mmkubernetes**
|
|
|
c17bfd |
+**Author:** `Tomáš Heinrich`
|
|
|
c17bfd |
+ `Rich Megginson` <rmeggins@redhat.com>
|
|
|
c17bfd |
+=========================== ===========================================================================
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Purpose
|
|
|
c17bfd |
+=======
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+This module is used to add `Kubernetes <https://kubernetes.io/>`
|
|
|
c17bfd |
+metadata to log messages logged by containers running in Kubernetes.
|
|
|
c17bfd |
+It will add the namespace uuid, pod uuid, pod and namespace labels and
|
|
|
c17bfd |
+annotations, and other metadata associated with the pod and
|
|
|
c17bfd |
+namespace.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. note::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ This **only** works with log files in `/var/log/containers/*.log`
|
|
|
c17bfd |
+ (docker `--log-driver=json-file`), or with journald entries with
|
|
|
c17bfd |
+ message properties `CONTAINER_NAME` and `CONTAINER_ID_FULL` (docker
|
|
|
c17bfd |
+ `--log-driver=journald`), and when the application running inside
|
|
|
c17bfd |
+ the container writes logs to `stdout`/`stderr`. This **does not**
|
|
|
c17bfd |
+ currently work with other log drivers.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+For json-file logs, you must use the `imfile` module with the
|
|
|
c17bfd |
+`addmetadata="on"` parameter, and the filename must match the
|
|
|
c17bfd |
+liblognorm rules specified by the `filenamerules`
|
|
|
c17bfd |
+(:ref:`filenamerules`) or `filenamerulebase` (:ref:`filenamerulebase`)
|
|
|
c17bfd |
+parameter values.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+For journald logs, there must be a message property `CONTAINER_NAME`
|
|
|
c17bfd |
+which matches the liblognorm rules specified by the `containerrules`
|
|
|
c17bfd |
+(:ref:`containerrules`) or `containerrulebase`
|
|
|
c17bfd |
+(:ref:`containerrulebase`) parameter values. The record must also have
|
|
|
c17bfd |
+the message property `CONTAINER_ID_FULL`.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+This module is implemented via the output module interface. This means
|
|
|
c17bfd |
+that mmkubernetes should be called just like an action. After it has
|
|
|
c17bfd |
+been called, there will be two new message properties: `kubernetes`
|
|
|
c17bfd |
+and `docker`. There will be subfields of each one for the various
|
|
|
c17bfd |
+metadata items: `$!kubernetes!namespace_name`
|
|
|
c17bfd |
+`$!kubernetes!labels!this-is-my-label`, etc. There is currently only
|
|
|
c17bfd |
+1 docker subfield: `$!docker!container_id`. See
|
|
|
c17bfd |
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml
|
|
|
c17bfd |
+and
|
|
|
c17bfd |
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml
|
|
|
c17bfd |
+for more details.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Configuration Parameters
|
|
|
c17bfd |
+========================
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. note::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ Parameter names are case-insensitive.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Module Parameters and Action Parameters
|
|
|
c17bfd |
+---------------------------------------
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _kubernetesurl:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+KubernetesURL
|
|
|
c17bfd |
+^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "https://kubernetes.default.svc.cluster.local:443", "yes", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+The URL of the Kubernetes API server. Example: `https://localhost:8443`.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _mmkubernetes-tls.cacert:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+tls.cacert
|
|
|
c17bfd |
+^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "none", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Full path and file name of file containing the CA cert of the
|
|
|
c17bfd |
+Kubernetes API server cert issuer. Example: `/etc/rsyslog.d/mmk8s-ca.crt`.
|
|
|
c17bfd |
+This parameter is not mandatory if using an `http` scheme instead of `https` in
|
|
|
c17bfd |
+`kubernetesurl`, or if using `allowunsignedcerts="yes"`.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _tokenfile:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+tokenfile
|
|
|
c17bfd |
+^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "none", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+The file containing the token to use to authenticate to the Kubernetes API
|
|
|
c17bfd |
+server. One of `tokenfile` or `token` is required if Kubernetes is configured
|
|
|
c17bfd |
+with access control. Example: `/etc/rsyslog.d/mmk8s.token`
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _token:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+token
|
|
|
c17bfd |
+^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "none", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+The token to use to authenticate to the Kubernetes API server. One of `token`
|
|
|
c17bfd |
+or `tokenfile` is required if Kubernetes is configured with access control.
|
|
|
c17bfd |
+Example: `UxMU46ptoEWOSqLNa1bFmH`
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _annotation_match:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+annotation_match
|
|
|
c17bfd |
+^^^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "array", "none", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+By default no pod or namespace annotations will be added to the
|
|
|
c17bfd |
+messages. This parameter is an array of patterns to match the keys of
|
|
|
c17bfd |
+the `annotations` field in the pod and namespace metadata to include
|
|
|
c17bfd |
+in the `$!kubernetes!annotations` (for pod annotations) or the
|
|
|
c17bfd |
+`$!kubernetes!namespace_annotations` (for namespace annotations)
|
|
|
c17bfd |
+message properties. Example: `["k8s.*master","k8s.*node"]`
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _srcmetadatapath:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+srcmetadatapath
|
|
|
c17bfd |
+^^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "$!metadata!filename", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+When reading json-file logs, with `imfile` and `addmetadata="on"`,
|
|
|
c17bfd |
+this is the property where the filename is stored.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _dstmetadatapath:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+dstmetadatapath
|
|
|
c17bfd |
+^^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "$!", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+This is the where the `kubernetes` and `docker` properties will be
|
|
|
c17bfd |
+written. By default, the module will add `$!kubernetes` and
|
|
|
c17bfd |
+`$!docker`.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _allowunsignedcerts:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+allowunsignedcerts
|
|
|
c17bfd |
+^^^^^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "boolean", "off", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to
|
|
|
c17bfd |
+`0`. You are strongly discouraged to set this to `"on"`. It is
|
|
|
c17bfd |
+primarily useful only for debugging or testing.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _de_dot:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+de_dot
|
|
|
c17bfd |
+^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "boolean", "on", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+When processing labels and annotations, if this parameter is set to
|
|
|
c17bfd |
+`"on"`, the key strings will have their `.` characters replaced with
|
|
|
c17bfd |
+the string specified by the `de_dot_separator` parameter.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _de_dot_separator:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+de_dot_separator
|
|
|
c17bfd |
+^^^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "_", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+When processing labels and annotations, if the `de_dot` parameter is
|
|
|
c17bfd |
+set to `"on"`, the key strings will have their `.` characters replaced
|
|
|
c17bfd |
+with the string specified by the string value of this parameter.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _filenamerules:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+filenamerules
|
|
|
c17bfd |
+^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "SEE BELOW", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. note::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ This directive is not supported with liblognorm 2.0.2 and earlier.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+When processing json-file logs, these are the lognorm rules to use to
|
|
|
c17bfd |
+match the filename and extract metadata. The default value is::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%conta\
|
|
|
c17bfd |
+ iner_name:char-to:-%-%container_id:char-to:.%.log
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. note::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ In the above rules, the slashes ``\`` ending each line indicate
|
|
|
c17bfd |
+ line wrapping - they are not part of the rule.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+There are two rules because the `container_hash` is optional.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _filenamerulebase:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+filenamerulebase
|
|
|
c17bfd |
+^^^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "/etc/rsyslog.d/k8s_filename.rulebase", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+When processing json-file logs, this is the rulebase used to
|
|
|
c17bfd |
+match the filename and extract metadata. For the actual rules, see
|
|
|
c17bfd |
+below `filenamerules`.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _containerrules:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+containerrules
|
|
|
c17bfd |
+^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "SEE BELOW", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. note::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ This directive is not supported with liblognorm 2.0.2 and earlier.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+For journald logs, there must be a message property `CONTAINER_NAME`
|
|
|
c17bfd |
+which has a value matching these rules specified by this parameter.
|
|
|
c17bfd |
+The default value is::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:\
|
|
|
c17bfd |
+ _%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_u\
|
|
|
c17bfd |
+ sed_2:rest%
|
|
|
c17bfd |
+ rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%na\
|
|
|
c17bfd |
+ mespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. note::
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ In the above rules, the slashes ``\`` ending each line indicate
|
|
|
c17bfd |
+ line wrapping - they are not part of the rule.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+There are two rules because the `container_hash` is optional.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. _containerrulebase:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+containerrulebase
|
|
|
c17bfd |
+^^^^^^^^^^^^^^^^^
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. csv-table::
|
|
|
c17bfd |
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
|
|
|
c17bfd |
+ :widths: auto
|
|
|
c17bfd |
+ :class: parameter-table
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ "word", "/etc/rsyslog.d/k8s_container_name.rulebase", "no", "none"
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+When processing json-file logs, this is the rulebase used to
|
|
|
c17bfd |
+match the CONTAINER_NAME property value and extract metadata. For the
|
|
|
c17bfd |
+actual rules, see `containerrules`.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Fields
|
|
|
c17bfd |
+------
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+These are the fields added from the metadata in the json-file filename, or from
|
|
|
c17bfd |
+the `CONTAINER_NAME` and `CONTAINER_ID_FULL` fields from the `imjournal` input:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+`$!kubernetes!namespace_name`, `$!kubernetes!pod_name`,
|
|
|
c17bfd |
+`$!kubernetes!container_name`, `$!docker!id`, `$!kubernetes!master_url`.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+If mmkubernetes can extract the above fields from the input, the following
|
|
|
c17bfd |
+fields will always be present. If they are not present, mmkubernetes
|
|
|
c17bfd |
+failed to look up the namespace or pod in Kubernetes:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+`$!kubernetes!namespace_id`, `$!kubernetes!pod_id`,
|
|
|
c17bfd |
+`$!kubernetes!creation_timestamp`, `$!kubernetes!host`
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+The following fields may be present, depending on how the namespace and pod are
|
|
|
c17bfd |
+defined in Kubernetes, and depending on the value of the directive
|
|
|
c17bfd |
+`annotation_match`:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+`$!kubernetes!labels`, `$!kubernetes!annotations`, `$!kubernetes!namespace_labels`,
|
|
|
c17bfd |
+`$!kubernetes!namespace_annotations`
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+More fields may be added in the future.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Example
|
|
|
c17bfd |
+-------
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Assuming you have an `imfile` input reading from docker json-file container
|
|
|
c17bfd |
+logs managed by Kubernetes, with `addmetadata="on"` so that mmkubernetes can
|
|
|
c17bfd |
+get the basic necessary Kubernetes metadata from the filename:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. code-block:: none
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ input(type="imfile" file="/var/log/containers/*.log"
|
|
|
c17bfd |
+ tag="kubernetes" addmetadata="on")
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+and/or an `imjournal` input for docker journald container logs annotated by
|
|
|
c17bfd |
+Kubernetes:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. code-block:: none
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ input(type="imjournal")
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Then mmkubernetes can be used to annotate log records like this:
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+.. code-block:: none
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ module(load="mmkubernetes")
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+ action(type="mmkubernetes")
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+After this, you should have log records with fields described in the `Fields`
|
|
|
c17bfd |
+section above.
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+Credits
|
|
|
c17bfd |
+-------
|
|
|
c17bfd |
+
|
|
|
c17bfd |
+This work is based on
|
|
|
c17bfd |
+https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
|
|
|
c17bfd |
+and has many of the same features.
|