Blame SOURCES/rsyslog-8.24.0-doc-rhbz1539193-mmkubernetes-new-plugin.patch

c17bfd
diff --git a/source/configuration/modules/mmkubernetes.rst b/source/configuration/modules/mmkubernetes.rst
c17bfd
new file mode 100644
c17bfd
index 0000000..1cd3d2a
c17bfd
--- /dev/null
c17bfd
+++ b/source/configuration/modules/mmkubernetes.rst
c17bfd
@@ -0,0 +1,378 @@
c17bfd
+*****************************************
c17bfd
+Kubernetes Metadata Module (mmkubernetes)
c17bfd
+*****************************************
c17bfd
+
c17bfd
+===========================  ===========================================================================
c17bfd
+**Module Name:**             **mmkubernetes**
c17bfd
+**Author:**                  `Tomáš Heinrich`
c17bfd
+                             `Rich Megginson` <rmeggins@redhat.com>
c17bfd
+===========================  ===========================================================================
c17bfd
+
c17bfd
+Purpose
c17bfd
+=======
c17bfd
+
c17bfd
+This module is used to add `Kubernetes <https://kubernetes.io/>`
c17bfd
+metadata to log messages logged by containers running in Kubernetes.
c17bfd
+It will add the namespace uuid, pod uuid, pod and namespace labels and
c17bfd
+annotations, and other metadata associated with the pod and
c17bfd
+namespace.
c17bfd
+
c17bfd
+.. note::
c17bfd
+
c17bfd
+   This **only** works with log files in `/var/log/containers/*.log`
c17bfd
+   (docker `--log-driver=json-file`), or with journald entries with
c17bfd
+   message properties `CONTAINER_NAME` and `CONTAINER_ID_FULL` (docker
c17bfd
+   `--log-driver=journald`), and when the application running inside
c17bfd
+   the container writes logs to `stdout`/`stderr`.  This **does not**
c17bfd
+   currently work with other log drivers.
c17bfd
+
c17bfd
+For json-file logs, you must use the `imfile` module with the
c17bfd
+`addmetadata="on"` parameter, and the filename must match the
c17bfd
+liblognorm rules specified by the `filenamerules`
c17bfd
+(:ref:`filenamerules`) or `filenamerulebase` (:ref:`filenamerulebase`)
c17bfd
+parameter values.
c17bfd
+
c17bfd
+For journald logs, there must be a message property `CONTAINER_NAME`
c17bfd
+which matches the liblognorm rules specified by the `containerrules`
c17bfd
+(:ref:`containerrules`) or `containerrulebase`
c17bfd
+(:ref:`containerrulebase`) parameter values. The record must also have
c17bfd
+the message property `CONTAINER_ID_FULL`.
c17bfd
+
c17bfd
+This module is implemented via the output module interface. This means
c17bfd
+that mmkubernetes should be called just like an action. After it has
c17bfd
+been called, there will be two new message properties: `kubernetes`
c17bfd
+and `docker`.  There will be subfields of each one for the various
c17bfd
+metadata items: `$!kubernetes!namespace_name`
c17bfd
+`$!kubernetes!labels!this-is-my-label`, etc.  There is currently only
c17bfd
+1 docker subfield: `$!docker!container_id`.  See
c17bfd
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml
c17bfd
+and
c17bfd
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml
c17bfd
+for more details.
c17bfd
+
c17bfd
+Configuration Parameters
c17bfd
+========================
c17bfd
+
c17bfd
+.. note::
c17bfd
+
c17bfd
+   Parameter names are case-insensitive.
c17bfd
+
c17bfd
+Module Parameters and Action Parameters
c17bfd
+---------------------------------------
c17bfd
+
c17bfd
+.. _kubernetesurl:
c17bfd
+
c17bfd
+KubernetesURL
c17bfd
+^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "https://kubernetes.default.svc.cluster.local:443", "yes", "none"
c17bfd
+
c17bfd
+The URL of the Kubernetes API server.  Example: `https://localhost:8443`.
c17bfd
+
c17bfd
+.. _mmkubernetes-tls.cacert:
c17bfd
+
c17bfd
+tls.cacert
c17bfd
+^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "none", "no", "none"
c17bfd
+
c17bfd
+Full path and file name of file containing the CA cert of the
c17bfd
+Kubernetes API server cert issuer.  Example: `/etc/rsyslog.d/mmk8s-ca.crt`.
c17bfd
+This parameter is not mandatory if using an `http` scheme instead of `https` in
c17bfd
+`kubernetesurl`, or if using `allowunsignedcerts="yes"`.
c17bfd
+
c17bfd
+.. _tokenfile:
c17bfd
+
c17bfd
+tokenfile
c17bfd
+^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "none", "no", "none"
c17bfd
+
c17bfd
+The file containing the token to use to authenticate to the Kubernetes API
c17bfd
+server.  One of `tokenfile` or `token` is required if Kubernetes is configured
c17bfd
+with access control.  Example: `/etc/rsyslog.d/mmk8s.token`
c17bfd
+
c17bfd
+.. _token:
c17bfd
+
c17bfd
+token
c17bfd
+^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "none", "no", "none"
c17bfd
+
c17bfd
+The token to use to authenticate to the Kubernetes API server.  One of `token`
c17bfd
+or `tokenfile` is required if Kubernetes is configured with access control.
c17bfd
+Example: `UxMU46ptoEWOSqLNa1bFmH`
c17bfd
+
c17bfd
+.. _annotation_match:
c17bfd
+
c17bfd
+annotation_match
c17bfd
+^^^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "array", "none", "no", "none"
c17bfd
+
c17bfd
+By default no pod or namespace annotations will be added to the
c17bfd
+messages.  This parameter is an array of patterns to match the keys of
c17bfd
+the `annotations` field in the pod and namespace metadata to include
c17bfd
+in the `$!kubernetes!annotations` (for pod annotations) or the
c17bfd
+`$!kubernetes!namespace_annotations` (for namespace annotations)
c17bfd
+message properties.  Example: `["k8s.*master","k8s.*node"]`
c17bfd
+
c17bfd
+.. _srcmetadatapath:
c17bfd
+
c17bfd
+srcmetadatapath
c17bfd
+^^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "$!metadata!filename", "no", "none"
c17bfd
+
c17bfd
+When reading json-file logs, with `imfile` and `addmetadata="on"`,
c17bfd
+this is the property where the filename is stored.
c17bfd
+
c17bfd
+.. _dstmetadatapath:
c17bfd
+
c17bfd
+dstmetadatapath
c17bfd
+^^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "$!", "no", "none"
c17bfd
+
c17bfd
+This is the where the `kubernetes` and `docker` properties will be
c17bfd
+written.  By default, the module will add `$!kubernetes` and
c17bfd
+`$!docker`.
c17bfd
+
c17bfd
+.. _allowunsignedcerts:
c17bfd
+
c17bfd
+allowunsignedcerts
c17bfd
+^^^^^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "boolean", "off", "no", "none"
c17bfd
+
c17bfd
+If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to
c17bfd
+`0`.  You are strongly discouraged to set this to `"on"`.  It is
c17bfd
+primarily useful only for debugging or testing.
c17bfd
+
c17bfd
+.. _de_dot:
c17bfd
+
c17bfd
+de_dot
c17bfd
+^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "boolean", "on", "no", "none"
c17bfd
+
c17bfd
+When processing labels and annotations, if this parameter is set to
c17bfd
+`"on"`, the key strings will have their `.` characters replaced with
c17bfd
+the string specified by the `de_dot_separator` parameter.
c17bfd
+
c17bfd
+.. _de_dot_separator:
c17bfd
+
c17bfd
+de_dot_separator
c17bfd
+^^^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "_", "no", "none"
c17bfd
+
c17bfd
+When processing labels and annotations, if the `de_dot` parameter is
c17bfd
+set to `"on"`, the key strings will have their `.` characters replaced
c17bfd
+with the string specified by the string value of this parameter.
c17bfd
+
c17bfd
+.. _filenamerules:
c17bfd
+
c17bfd
+filenamerules
c17bfd
+^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "SEE BELOW", "no", "none"
c17bfd
+
c17bfd
+.. note::
c17bfd
+
c17bfd
+    This directive is not supported with liblognorm 2.0.2 and earlier.
c17bfd
+
c17bfd
+When processing json-file logs, these are the lognorm rules to use to
c17bfd
+match the filename and extract metadata.  The default value is::
c17bfd
+
c17bfd
+    rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%conta\
c17bfd
+    iner_name:char-to:-%-%container_id:char-to:.%.log
c17bfd
+
c17bfd
+.. note::
c17bfd
+
c17bfd
+    In the above rules, the slashes ``\`` ending each line indicate
c17bfd
+    line wrapping - they are not part of the rule.
c17bfd
+
c17bfd
+There are two rules because the `container_hash` is optional.
c17bfd
+
c17bfd
+.. _filenamerulebase:
c17bfd
+
c17bfd
+filenamerulebase
c17bfd
+^^^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "/etc/rsyslog.d/k8s_filename.rulebase", "no", "none"
c17bfd
+
c17bfd
+When processing json-file logs, this is the rulebase used to
c17bfd
+match the filename and extract metadata.  For the actual rules, see
c17bfd
+below `filenamerules`.
c17bfd
+
c17bfd
+.. _containerrules:
c17bfd
+
c17bfd
+containerrules
c17bfd
+^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "SEE BELOW", "no", "none"
c17bfd
+
c17bfd
+.. note::
c17bfd
+
c17bfd
+    This directive is not supported with liblognorm 2.0.2 and earlier.
c17bfd
+
c17bfd
+For journald logs, there must be a message property `CONTAINER_NAME`
c17bfd
+which has a value matching these rules specified by this parameter.
c17bfd
+The default value is::
c17bfd
+
c17bfd
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:\
c17bfd
+    _%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_u\
c17bfd
+    sed_2:rest%
c17bfd
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%na\
c17bfd
+    mespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%
c17bfd
+
c17bfd
+.. note::
c17bfd
+
c17bfd
+    In the above rules, the slashes ``\`` ending each line indicate
c17bfd
+    line wrapping - they are not part of the rule.
c17bfd
+
c17bfd
+There are two rules because the `container_hash` is optional.
c17bfd
+
c17bfd
+.. _containerrulebase:
c17bfd
+
c17bfd
+containerrulebase
c17bfd
+^^^^^^^^^^^^^^^^^
c17bfd
+
c17bfd
+.. csv-table::
c17bfd
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
c17bfd
+   :widths: auto
c17bfd
+   :class: parameter-table
c17bfd
+
c17bfd
+   "word", "/etc/rsyslog.d/k8s_container_name.rulebase", "no", "none"
c17bfd
+
c17bfd
+When processing json-file logs, this is the rulebase used to
c17bfd
+match the CONTAINER_NAME property value and extract metadata.  For the
c17bfd
+actual rules, see `containerrules`.
c17bfd
+
c17bfd
+Fields
c17bfd
+------
c17bfd
+
c17bfd
+These are the fields added from the metadata in the json-file filename, or from
c17bfd
+the `CONTAINER_NAME` and `CONTAINER_ID_FULL` fields from the `imjournal` input:
c17bfd
+
c17bfd
+`$!kubernetes!namespace_name`, `$!kubernetes!pod_name`,
c17bfd
+`$!kubernetes!container_name`, `$!docker!id`, `$!kubernetes!master_url`.
c17bfd
+
c17bfd
+If mmkubernetes can extract the above fields from the input, the following
c17bfd
+fields will always be present.  If they are not present, mmkubernetes
c17bfd
+failed to look up the namespace or pod in Kubernetes:
c17bfd
+
c17bfd
+`$!kubernetes!namespace_id`, `$!kubernetes!pod_id`,
c17bfd
+`$!kubernetes!creation_timestamp`, `$!kubernetes!host`
c17bfd
+
c17bfd
+The following fields may be present, depending on how the namespace and pod are
c17bfd
+defined in Kubernetes, and depending on the value of the directive
c17bfd
+`annotation_match`:
c17bfd
+
c17bfd
+`$!kubernetes!labels`, `$!kubernetes!annotations`, `$!kubernetes!namespace_labels`,
c17bfd
+`$!kubernetes!namespace_annotations`
c17bfd
+
c17bfd
+More fields may be added in the future.
c17bfd
+
c17bfd
+Example
c17bfd
+-------
c17bfd
+
c17bfd
+Assuming you have an `imfile` input reading from docker json-file container
c17bfd
+logs managed by Kubernetes, with `addmetadata="on"` so that mmkubernetes can
c17bfd
+get the basic necessary Kubernetes metadata from the filename:
c17bfd
+
c17bfd
+.. code-block:: none
c17bfd
+
c17bfd
+    input(type="imfile" file="/var/log/containers/*.log"
c17bfd
+          tag="kubernetes" addmetadata="on")
c17bfd
+
c17bfd
+and/or an `imjournal` input for docker journald container logs annotated by
c17bfd
+Kubernetes:
c17bfd
+
c17bfd
+.. code-block:: none
c17bfd
+
c17bfd
+    input(type="imjournal")
c17bfd
+
c17bfd
+Then mmkubernetes can be used to annotate log records like this:
c17bfd
+
c17bfd
+.. code-block:: none
c17bfd
+
c17bfd
+    module(load="mmkubernetes")
c17bfd
+
c17bfd
+    action(type="mmkubernetes")
c17bfd
+
c17bfd
+After this, you should have log records with fields described in the `Fields`
c17bfd
+section above.
c17bfd
+
c17bfd
+Credits
c17bfd
+-------
c17bfd
+
c17bfd
+This work is based on
c17bfd
+https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
c17bfd
+and has many of the same features.