Blame SOURCES/rsyslog-8.24.0-doc-rhbz1539193-mmkubernetes-new-plugin.patch

f656cf
diff --git a/source/configuration/modules/mmkubernetes.rst b/source/configuration/modules/mmkubernetes.rst
f656cf
new file mode 100644
f656cf
index 0000000..1cd3d2a
f656cf
--- /dev/null
f656cf
+++ b/source/configuration/modules/mmkubernetes.rst
f656cf
@@ -0,0 +1,378 @@
f656cf
+*****************************************
f656cf
+Kubernetes Metadata Module (mmkubernetes)
f656cf
+*****************************************
f656cf
+
f656cf
+===========================  ===========================================================================
f656cf
+**Module Name:**             **mmkubernetes**
f656cf
+**Author:**                  `Tomáš Heinrich`
f656cf
+                             `Rich Megginson` <rmeggins@redhat.com>
f656cf
+===========================  ===========================================================================
f656cf
+
f656cf
+Purpose
f656cf
+=======
f656cf
+
f656cf
+This module is used to add `Kubernetes <https://kubernetes.io/>`
f656cf
+metadata to log messages logged by containers running in Kubernetes.
f656cf
+It will add the namespace uuid, pod uuid, pod and namespace labels and
f656cf
+annotations, and other metadata associated with the pod and
f656cf
+namespace.
f656cf
+
f656cf
+.. note::
f656cf
+
f656cf
+   This **only** works with log files in `/var/log/containers/*.log`
f656cf
+   (docker `--log-driver=json-file`), or with journald entries with
f656cf
+   message properties `CONTAINER_NAME` and `CONTAINER_ID_FULL` (docker
f656cf
+   `--log-driver=journald`), and when the application running inside
f656cf
+   the container writes logs to `stdout`/`stderr`.  This **does not**
f656cf
+   currently work with other log drivers.
f656cf
+
f656cf
+For json-file logs, you must use the `imfile` module with the
f656cf
+`addmetadata="on"` parameter, and the filename must match the
f656cf
+liblognorm rules specified by the `filenamerules`
f656cf
+(:ref:`filenamerules`) or `filenamerulebase` (:ref:`filenamerulebase`)
f656cf
+parameter values.
f656cf
+
f656cf
+For journald logs, there must be a message property `CONTAINER_NAME`
f656cf
+which matches the liblognorm rules specified by the `containerrules`
f656cf
+(:ref:`containerrules`) or `containerrulebase`
f656cf
+(:ref:`containerrulebase`) parameter values. The record must also have
f656cf
+the message property `CONTAINER_ID_FULL`.
f656cf
+
f656cf
+This module is implemented via the output module interface. This means
f656cf
+that mmkubernetes should be called just like an action. After it has
f656cf
+been called, there will be two new message properties: `kubernetes`
f656cf
+and `docker`.  There will be subfields of each one for the various
f656cf
+metadata items: `$!kubernetes!namespace_name`
f656cf
+`$!kubernetes!labels!this-is-my-label`, etc.  There is currently only
f656cf
+1 docker subfield: `$!docker!container_id`.  See
f656cf
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml
f656cf
+and
f656cf
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml
f656cf
+for more details.
f656cf
+
f656cf
+Configuration Parameters
f656cf
+========================
f656cf
+
f656cf
+.. note::
f656cf
+
f656cf
+   Parameter names are case-insensitive.
f656cf
+
f656cf
+Module Parameters and Action Parameters
f656cf
+---------------------------------------
f656cf
+
f656cf
+.. _kubernetesurl:
f656cf
+
f656cf
+KubernetesURL
f656cf
+^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "https://kubernetes.default.svc.cluster.local:443", "yes", "none"
f656cf
+
f656cf
+The URL of the Kubernetes API server.  Example: `https://localhost:8443`.
f656cf
+
f656cf
+.. _mmkubernetes-tls.cacert:
f656cf
+
f656cf
+tls.cacert
f656cf
+^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "none", "no", "none"
f656cf
+
f656cf
+Full path and file name of file containing the CA cert of the
f656cf
+Kubernetes API server cert issuer.  Example: `/etc/rsyslog.d/mmk8s-ca.crt`.
f656cf
+This parameter is not mandatory if using an `http` scheme instead of `https` in
f656cf
+`kubernetesurl`, or if using `allowunsignedcerts="yes"`.
f656cf
+
f656cf
+.. _tokenfile:
f656cf
+
f656cf
+tokenfile
f656cf
+^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "none", "no", "none"
f656cf
+
f656cf
+The file containing the token to use to authenticate to the Kubernetes API
f656cf
+server.  One of `tokenfile` or `token` is required if Kubernetes is configured
f656cf
+with access control.  Example: `/etc/rsyslog.d/mmk8s.token`
f656cf
+
f656cf
+.. _token:
f656cf
+
f656cf
+token
f656cf
+^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "none", "no", "none"
f656cf
+
f656cf
+The token to use to authenticate to the Kubernetes API server.  One of `token`
f656cf
+or `tokenfile` is required if Kubernetes is configured with access control.
f656cf
+Example: `UxMU46ptoEWOSqLNa1bFmH`
f656cf
+
f656cf
+.. _annotation_match:
f656cf
+
f656cf
+annotation_match
f656cf
+^^^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "array", "none", "no", "none"
f656cf
+
f656cf
+By default no pod or namespace annotations will be added to the
f656cf
+messages.  This parameter is an array of patterns to match the keys of
f656cf
+the `annotations` field in the pod and namespace metadata to include
f656cf
+in the `$!kubernetes!annotations` (for pod annotations) or the
f656cf
+`$!kubernetes!namespace_annotations` (for namespace annotations)
f656cf
+message properties.  Example: `["k8s.*master","k8s.*node"]`
f656cf
+
f656cf
+.. _srcmetadatapath:
f656cf
+
f656cf
+srcmetadatapath
f656cf
+^^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "$!metadata!filename", "no", "none"
f656cf
+
f656cf
+When reading json-file logs, with `imfile` and `addmetadata="on"`,
f656cf
+this is the property where the filename is stored.
f656cf
+
f656cf
+.. _dstmetadatapath:
f656cf
+
f656cf
+dstmetadatapath
f656cf
+^^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "$!", "no", "none"
f656cf
+
f656cf
+This is the where the `kubernetes` and `docker` properties will be
f656cf
+written.  By default, the module will add `$!kubernetes` and
f656cf
+`$!docker`.
f656cf
+
f656cf
+.. _allowunsignedcerts:
f656cf
+
f656cf
+allowunsignedcerts
f656cf
+^^^^^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "boolean", "off", "no", "none"
f656cf
+
f656cf
+If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to
f656cf
+`0`.  You are strongly discouraged to set this to `"on"`.  It is
f656cf
+primarily useful only for debugging or testing.
f656cf
+
f656cf
+.. _de_dot:
f656cf
+
f656cf
+de_dot
f656cf
+^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "boolean", "on", "no", "none"
f656cf
+
f656cf
+When processing labels and annotations, if this parameter is set to
f656cf
+`"on"`, the key strings will have their `.` characters replaced with
f656cf
+the string specified by the `de_dot_separator` parameter.
f656cf
+
f656cf
+.. _de_dot_separator:
f656cf
+
f656cf
+de_dot_separator
f656cf
+^^^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "_", "no", "none"
f656cf
+
f656cf
+When processing labels and annotations, if the `de_dot` parameter is
f656cf
+set to `"on"`, the key strings will have their `.` characters replaced
f656cf
+with the string specified by the string value of this parameter.
f656cf
+
f656cf
+.. _filenamerules:
f656cf
+
f656cf
+filenamerules
f656cf
+^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "SEE BELOW", "no", "none"
f656cf
+
f656cf
+.. note::
f656cf
+
f656cf
+    This directive is not supported with liblognorm 2.0.2 and earlier.
f656cf
+
f656cf
+When processing json-file logs, these are the lognorm rules to use to
f656cf
+match the filename and extract metadata.  The default value is::
f656cf
+
f656cf
+    rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%conta\
f656cf
+    iner_name:char-to:-%-%container_id:char-to:.%.log
f656cf
+
f656cf
+.. note::
f656cf
+
f656cf
+    In the above rules, the slashes ``\`` ending each line indicate
f656cf
+    line wrapping - they are not part of the rule.
f656cf
+
f656cf
+There are two rules because the `container_hash` is optional.
f656cf
+
f656cf
+.. _filenamerulebase:
f656cf
+
f656cf
+filenamerulebase
f656cf
+^^^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "/etc/rsyslog.d/k8s_filename.rulebase", "no", "none"
f656cf
+
f656cf
+When processing json-file logs, this is the rulebase used to
f656cf
+match the filename and extract metadata.  For the actual rules, see
f656cf
+below `filenamerules`.
f656cf
+
f656cf
+.. _containerrules:
f656cf
+
f656cf
+containerrules
f656cf
+^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "SEE BELOW", "no", "none"
f656cf
+
f656cf
+.. note::
f656cf
+
f656cf
+    This directive is not supported with liblognorm 2.0.2 and earlier.
f656cf
+
f656cf
+For journald logs, there must be a message property `CONTAINER_NAME`
f656cf
+which has a value matching these rules specified by this parameter.
f656cf
+The default value is::
f656cf
+
f656cf
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:\
f656cf
+    _%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_u\
f656cf
+    sed_2:rest%
f656cf
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%na\
f656cf
+    mespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%
f656cf
+
f656cf
+.. note::
f656cf
+
f656cf
+    In the above rules, the slashes ``\`` ending each line indicate
f656cf
+    line wrapping - they are not part of the rule.
f656cf
+
f656cf
+There are two rules because the `container_hash` is optional.
f656cf
+
f656cf
+.. _containerrulebase:
f656cf
+
f656cf
+containerrulebase
f656cf
+^^^^^^^^^^^^^^^^^
f656cf
+
f656cf
+.. csv-table::
f656cf
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
f656cf
+   :widths: auto
f656cf
+   :class: parameter-table
f656cf
+
f656cf
+   "word", "/etc/rsyslog.d/k8s_container_name.rulebase", "no", "none"
f656cf
+
f656cf
+When processing json-file logs, this is the rulebase used to
f656cf
+match the CONTAINER_NAME property value and extract metadata.  For the
f656cf
+actual rules, see `containerrules`.
f656cf
+
f656cf
+Fields
f656cf
+------
f656cf
+
f656cf
+These are the fields added from the metadata in the json-file filename, or from
f656cf
+the `CONTAINER_NAME` and `CONTAINER_ID_FULL` fields from the `imjournal` input:
f656cf
+
f656cf
+`$!kubernetes!namespace_name`, `$!kubernetes!pod_name`,
f656cf
+`$!kubernetes!container_name`, `$!docker!id`, `$!kubernetes!master_url`.
f656cf
+
f656cf
+If mmkubernetes can extract the above fields from the input, the following
f656cf
+fields will always be present.  If they are not present, mmkubernetes
f656cf
+failed to look up the namespace or pod in Kubernetes:
f656cf
+
f656cf
+`$!kubernetes!namespace_id`, `$!kubernetes!pod_id`,
f656cf
+`$!kubernetes!creation_timestamp`, `$!kubernetes!host`
f656cf
+
f656cf
+The following fields may be present, depending on how the namespace and pod are
f656cf
+defined in Kubernetes, and depending on the value of the directive
f656cf
+`annotation_match`:
f656cf
+
f656cf
+`$!kubernetes!labels`, `$!kubernetes!annotations`, `$!kubernetes!namespace_labels`,
f656cf
+`$!kubernetes!namespace_annotations`
f656cf
+
f656cf
+More fields may be added in the future.
f656cf
+
f656cf
+Example
f656cf
+-------
f656cf
+
f656cf
+Assuming you have an `imfile` input reading from docker json-file container
f656cf
+logs managed by Kubernetes, with `addmetadata="on"` so that mmkubernetes can
f656cf
+get the basic necessary Kubernetes metadata from the filename:
f656cf
+
f656cf
+.. code-block:: none
f656cf
+
f656cf
+    input(type="imfile" file="/var/log/containers/*.log"
f656cf
+          tag="kubernetes" addmetadata="on")
f656cf
+
f656cf
+and/or an `imjournal` input for docker journald container logs annotated by
f656cf
+Kubernetes:
f656cf
+
f656cf
+.. code-block:: none
f656cf
+
f656cf
+    input(type="imjournal")
f656cf
+
f656cf
+Then mmkubernetes can be used to annotate log records like this:
f656cf
+
f656cf
+.. code-block:: none
f656cf
+
f656cf
+    module(load="mmkubernetes")
f656cf
+
f656cf
+    action(type="mmkubernetes")
f656cf
+
f656cf
+After this, you should have log records with fields described in the `Fields`
f656cf
+section above.
f656cf
+
f656cf
+Credits
f656cf
+-------
f656cf
+
f656cf
+This work is based on
f656cf
+https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
f656cf
+and has many of the same features.