Blame SOURCES/rsyslog-8.24.0-doc-rhbz1539193-mmkubernetes-new-plugin.patch

fde8c1
diff --git a/source/configuration/modules/mmkubernetes.rst b/source/configuration/modules/mmkubernetes.rst
fde8c1
new file mode 100644
fde8c1
index 0000000..1cd3d2a
fde8c1
--- /dev/null
fde8c1
+++ b/source/configuration/modules/mmkubernetes.rst
fde8c1
@@ -0,0 +1,378 @@
fde8c1
+*****************************************
fde8c1
+Kubernetes Metadata Module (mmkubernetes)
fde8c1
+*****************************************
fde8c1
+
fde8c1
+===========================  ===========================================================================
fde8c1
+**Module Name:**             **mmkubernetes**
fde8c1
+**Author:**                  `Tomáš Heinrich`
fde8c1
+                             `Rich Megginson` <rmeggins@redhat.com>
fde8c1
+===========================  ===========================================================================
fde8c1
+
fde8c1
+Purpose
fde8c1
+=======
fde8c1
+
fde8c1
+This module is used to add `Kubernetes <https://kubernetes.io/>`
fde8c1
+metadata to log messages logged by containers running in Kubernetes.
fde8c1
+It will add the namespace uuid, pod uuid, pod and namespace labels and
fde8c1
+annotations, and other metadata associated with the pod and
fde8c1
+namespace.
fde8c1
+
fde8c1
+.. note::
fde8c1
+
fde8c1
+   This **only** works with log files in `/var/log/containers/*.log`
fde8c1
+   (docker `--log-driver=json-file`), or with journald entries with
fde8c1
+   message properties `CONTAINER_NAME` and `CONTAINER_ID_FULL` (docker
fde8c1
+   `--log-driver=journald`), and when the application running inside
fde8c1
+   the container writes logs to `stdout`/`stderr`.  This **does not**
fde8c1
+   currently work with other log drivers.
fde8c1
+
fde8c1
+For json-file logs, you must use the `imfile` module with the
fde8c1
+`addmetadata="on"` parameter, and the filename must match the
fde8c1
+liblognorm rules specified by the `filenamerules`
fde8c1
+(:ref:`filenamerules`) or `filenamerulebase` (:ref:`filenamerulebase`)
fde8c1
+parameter values.
fde8c1
+
fde8c1
+For journald logs, there must be a message property `CONTAINER_NAME`
fde8c1
+which matches the liblognorm rules specified by the `containerrules`
fde8c1
+(:ref:`containerrules`) or `containerrulebase`
fde8c1
+(:ref:`containerrulebase`) parameter values. The record must also have
fde8c1
+the message property `CONTAINER_ID_FULL`.
fde8c1
+
fde8c1
+This module is implemented via the output module interface. This means
fde8c1
+that mmkubernetes should be called just like an action. After it has
fde8c1
+been called, there will be two new message properties: `kubernetes`
fde8c1
+and `docker`.  There will be subfields of each one for the various
fde8c1
+metadata items: `$!kubernetes!namespace_name`
fde8c1
+`$!kubernetes!labels!this-is-my-label`, etc.  There is currently only
fde8c1
+1 docker subfield: `$!docker!container_id`.  See
fde8c1
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml
fde8c1
+and
fde8c1
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml
fde8c1
+for more details.
fde8c1
+
fde8c1
+Configuration Parameters
fde8c1
+========================
fde8c1
+
fde8c1
+.. note::
fde8c1
+
fde8c1
+   Parameter names are case-insensitive.
fde8c1
+
fde8c1
+Module Parameters and Action Parameters
fde8c1
+---------------------------------------
fde8c1
+
fde8c1
+.. _kubernetesurl:
fde8c1
+
fde8c1
+KubernetesURL
fde8c1
+^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "https://kubernetes.default.svc.cluster.local:443", "yes", "none"
fde8c1
+
fde8c1
+The URL of the Kubernetes API server.  Example: `https://localhost:8443`.
fde8c1
+
fde8c1
+.. _mmkubernetes-tls.cacert:
fde8c1
+
fde8c1
+tls.cacert
fde8c1
+^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "none", "no", "none"
fde8c1
+
fde8c1
+Full path and file name of file containing the CA cert of the
fde8c1
+Kubernetes API server cert issuer.  Example: `/etc/rsyslog.d/mmk8s-ca.crt`.
fde8c1
+This parameter is not mandatory if using an `http` scheme instead of `https` in
fde8c1
+`kubernetesurl`, or if using `allowunsignedcerts="yes"`.
fde8c1
+
fde8c1
+.. _tokenfile:
fde8c1
+
fde8c1
+tokenfile
fde8c1
+^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "none", "no", "none"
fde8c1
+
fde8c1
+The file containing the token to use to authenticate to the Kubernetes API
fde8c1
+server.  One of `tokenfile` or `token` is required if Kubernetes is configured
fde8c1
+with access control.  Example: `/etc/rsyslog.d/mmk8s.token`
fde8c1
+
fde8c1
+.. _token:
fde8c1
+
fde8c1
+token
fde8c1
+^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "none", "no", "none"
fde8c1
+
fde8c1
+The token to use to authenticate to the Kubernetes API server.  One of `token`
fde8c1
+or `tokenfile` is required if Kubernetes is configured with access control.
fde8c1
+Example: `UxMU46ptoEWOSqLNa1bFmH`
fde8c1
+
fde8c1
+.. _annotation_match:
fde8c1
+
fde8c1
+annotation_match
fde8c1
+^^^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "array", "none", "no", "none"
fde8c1
+
fde8c1
+By default no pod or namespace annotations will be added to the
fde8c1
+messages.  This parameter is an array of patterns to match the keys of
fde8c1
+the `annotations` field in the pod and namespace metadata to include
fde8c1
+in the `$!kubernetes!annotations` (for pod annotations) or the
fde8c1
+`$!kubernetes!namespace_annotations` (for namespace annotations)
fde8c1
+message properties.  Example: `["k8s.*master","k8s.*node"]`
fde8c1
+
fde8c1
+.. _srcmetadatapath:
fde8c1
+
fde8c1
+srcmetadatapath
fde8c1
+^^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "$!metadata!filename", "no", "none"
fde8c1
+
fde8c1
+When reading json-file logs, with `imfile` and `addmetadata="on"`,
fde8c1
+this is the property where the filename is stored.
fde8c1
+
fde8c1
+.. _dstmetadatapath:
fde8c1
+
fde8c1
+dstmetadatapath
fde8c1
+^^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "$!", "no", "none"
fde8c1
+
fde8c1
+This is the where the `kubernetes` and `docker` properties will be
fde8c1
+written.  By default, the module will add `$!kubernetes` and
fde8c1
+`$!docker`.
fde8c1
+
fde8c1
+.. _allowunsignedcerts:
fde8c1
+
fde8c1
+allowunsignedcerts
fde8c1
+^^^^^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "boolean", "off", "no", "none"
fde8c1
+
fde8c1
+If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to
fde8c1
+`0`.  You are strongly discouraged to set this to `"on"`.  It is
fde8c1
+primarily useful only for debugging or testing.
fde8c1
+
fde8c1
+.. _de_dot:
fde8c1
+
fde8c1
+de_dot
fde8c1
+^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "boolean", "on", "no", "none"
fde8c1
+
fde8c1
+When processing labels and annotations, if this parameter is set to
fde8c1
+`"on"`, the key strings will have their `.` characters replaced with
fde8c1
+the string specified by the `de_dot_separator` parameter.
fde8c1
+
fde8c1
+.. _de_dot_separator:
fde8c1
+
fde8c1
+de_dot_separator
fde8c1
+^^^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "_", "no", "none"
fde8c1
+
fde8c1
+When processing labels and annotations, if the `de_dot` parameter is
fde8c1
+set to `"on"`, the key strings will have their `.` characters replaced
fde8c1
+with the string specified by the string value of this parameter.
fde8c1
+
fde8c1
+.. _filenamerules:
fde8c1
+
fde8c1
+filenamerules
fde8c1
+^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "SEE BELOW", "no", "none"
fde8c1
+
fde8c1
+.. note::
fde8c1
+
fde8c1
+    This directive is not supported with liblognorm 2.0.2 and earlier.
fde8c1
+
fde8c1
+When processing json-file logs, these are the lognorm rules to use to
fde8c1
+match the filename and extract metadata.  The default value is::
fde8c1
+
fde8c1
+    rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%conta\
fde8c1
+    iner_name:char-to:-%-%container_id:char-to:.%.log
fde8c1
+
fde8c1
+.. note::
fde8c1
+
fde8c1
+    In the above rules, the slashes ``\`` ending each line indicate
fde8c1
+    line wrapping - they are not part of the rule.
fde8c1
+
fde8c1
+There are two rules because the `container_hash` is optional.
fde8c1
+
fde8c1
+.. _filenamerulebase:
fde8c1
+
fde8c1
+filenamerulebase
fde8c1
+^^^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "/etc/rsyslog.d/k8s_filename.rulebase", "no", "none"
fde8c1
+
fde8c1
+When processing json-file logs, this is the rulebase used to
fde8c1
+match the filename and extract metadata.  For the actual rules, see
fde8c1
+below `filenamerules`.
fde8c1
+
fde8c1
+.. _containerrules:
fde8c1
+
fde8c1
+containerrules
fde8c1
+^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "SEE BELOW", "no", "none"
fde8c1
+
fde8c1
+.. note::
fde8c1
+
fde8c1
+    This directive is not supported with liblognorm 2.0.2 and earlier.
fde8c1
+
fde8c1
+For journald logs, there must be a message property `CONTAINER_NAME`
fde8c1
+which has a value matching these rules specified by this parameter.
fde8c1
+The default value is::
fde8c1
+
fde8c1
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:\
fde8c1
+    _%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_u\
fde8c1
+    sed_2:rest%
fde8c1
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%na\
fde8c1
+    mespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%
fde8c1
+
fde8c1
+.. note::
fde8c1
+
fde8c1
+    In the above rules, the slashes ``\`` ending each line indicate
fde8c1
+    line wrapping - they are not part of the rule.
fde8c1
+
fde8c1
+There are two rules because the `container_hash` is optional.
fde8c1
+
fde8c1
+.. _containerrulebase:
fde8c1
+
fde8c1
+containerrulebase
fde8c1
+^^^^^^^^^^^^^^^^^
fde8c1
+
fde8c1
+.. csv-table::
fde8c1
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
fde8c1
+   :widths: auto
fde8c1
+   :class: parameter-table
fde8c1
+
fde8c1
+   "word", "/etc/rsyslog.d/k8s_container_name.rulebase", "no", "none"
fde8c1
+
fde8c1
+When processing json-file logs, this is the rulebase used to
fde8c1
+match the CONTAINER_NAME property value and extract metadata.  For the
fde8c1
+actual rules, see `containerrules`.
fde8c1
+
fde8c1
+Fields
fde8c1
+------
fde8c1
+
fde8c1
+These are the fields added from the metadata in the json-file filename, or from
fde8c1
+the `CONTAINER_NAME` and `CONTAINER_ID_FULL` fields from the `imjournal` input:
fde8c1
+
fde8c1
+`$!kubernetes!namespace_name`, `$!kubernetes!pod_name`,
fde8c1
+`$!kubernetes!container_name`, `$!docker!id`, `$!kubernetes!master_url`.
fde8c1
+
fde8c1
+If mmkubernetes can extract the above fields from the input, the following
fde8c1
+fields will always be present.  If they are not present, mmkubernetes
fde8c1
+failed to look up the namespace or pod in Kubernetes:
fde8c1
+
fde8c1
+`$!kubernetes!namespace_id`, `$!kubernetes!pod_id`,
fde8c1
+`$!kubernetes!creation_timestamp`, `$!kubernetes!host`
fde8c1
+
fde8c1
+The following fields may be present, depending on how the namespace and pod are
fde8c1
+defined in Kubernetes, and depending on the value of the directive
fde8c1
+`annotation_match`:
fde8c1
+
fde8c1
+`$!kubernetes!labels`, `$!kubernetes!annotations`, `$!kubernetes!namespace_labels`,
fde8c1
+`$!kubernetes!namespace_annotations`
fde8c1
+
fde8c1
+More fields may be added in the future.
fde8c1
+
fde8c1
+Example
fde8c1
+-------
fde8c1
+
fde8c1
+Assuming you have an `imfile` input reading from docker json-file container
fde8c1
+logs managed by Kubernetes, with `addmetadata="on"` so that mmkubernetes can
fde8c1
+get the basic necessary Kubernetes metadata from the filename:
fde8c1
+
fde8c1
+.. code-block:: none
fde8c1
+
fde8c1
+    input(type="imfile" file="/var/log/containers/*.log"
fde8c1
+          tag="kubernetes" addmetadata="on")
fde8c1
+
fde8c1
+and/or an `imjournal` input for docker journald container logs annotated by
fde8c1
+Kubernetes:
fde8c1
+
fde8c1
+.. code-block:: none
fde8c1
+
fde8c1
+    input(type="imjournal")
fde8c1
+
fde8c1
+Then mmkubernetes can be used to annotate log records like this:
fde8c1
+
fde8c1
+.. code-block:: none
fde8c1
+
fde8c1
+    module(load="mmkubernetes")
fde8c1
+
fde8c1
+    action(type="mmkubernetes")
fde8c1
+
fde8c1
+After this, you should have log records with fields described in the `Fields`
fde8c1
+section above.
fde8c1
+
fde8c1
+Credits
fde8c1
+-------
fde8c1
+
fde8c1
+This work is based on
fde8c1
+https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
fde8c1
+and has many of the same features.