Blob Blame History Raw
diff --git a/source/configuration/modules/mmkubernetes.rst b/source/configuration/modules/mmkubernetes.rst
new file mode 100644
index 0000000..1cd3d2a
--- /dev/null
+++ b/source/configuration/modules/mmkubernetes.rst
@@ -0,0 +1,378 @@
+*****************************************
+Kubernetes Metadata Module (mmkubernetes)
+*****************************************
+
+===========================  ===========================================================================
+**Module Name:**             **mmkubernetes**
+**Author:**                  `Tomáš Heinrich`
+                             `Rich Megginson` <rmeggins@redhat.com>
+===========================  ===========================================================================
+
+Purpose
+=======
+
+This module is used to add `Kubernetes <https://kubernetes.io/>`
+metadata to log messages logged by containers running in Kubernetes.
+It will add the namespace uuid, pod uuid, pod and namespace labels and
+annotations, and other metadata associated with the pod and
+namespace.
+
+.. note::
+
+   This **only** works with log files in `/var/log/containers/*.log`
+   (docker `--log-driver=json-file`), or with journald entries with
+   message properties `CONTAINER_NAME` and `CONTAINER_ID_FULL` (docker
+   `--log-driver=journald`), and when the application running inside
+   the container writes logs to `stdout`/`stderr`.  This **does not**
+   currently work with other log drivers.
+
+For json-file logs, you must use the `imfile` module with the
+`addmetadata="on"` parameter, and the filename must match the
+liblognorm rules specified by the `filenamerules`
+(:ref:`filenamerules`) or `filenamerulebase` (:ref:`filenamerulebase`)
+parameter values.
+
+For journald logs, there must be a message property `CONTAINER_NAME`
+which matches the liblognorm rules specified by the `containerrules`
+(:ref:`containerrules`) or `containerrulebase`
+(:ref:`containerrulebase`) parameter values. The record must also have
+the message property `CONTAINER_ID_FULL`.
+
+This module is implemented via the output module interface. This means
+that mmkubernetes should be called just like an action. After it has
+been called, there will be two new message properties: `kubernetes`
+and `docker`.  There will be subfields of each one for the various
+metadata items: `$!kubernetes!namespace_name`
+`$!kubernetes!labels!this-is-my-label`, etc.  There is currently only
+1 docker subfield: `$!docker!container_id`.  See
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml
+and
+https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml
+for more details.
+
+Configuration Parameters
+========================
+
+.. note::
+
+   Parameter names are case-insensitive.
+
+Module Parameters and Action Parameters
+---------------------------------------
+
+.. _kubernetesurl:
+
+KubernetesURL
+^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "https://kubernetes.default.svc.cluster.local:443", "yes", "none"
+
+The URL of the Kubernetes API server.  Example: `https://localhost:8443`.
+
+.. _mmkubernetes-tls.cacert:
+
+tls.cacert
+^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "none", "no", "none"
+
+Full path and file name of file containing the CA cert of the
+Kubernetes API server cert issuer.  Example: `/etc/rsyslog.d/mmk8s-ca.crt`.
+This parameter is not mandatory if using an `http` scheme instead of `https` in
+`kubernetesurl`, or if using `allowunsignedcerts="yes"`.
+
+.. _tokenfile:
+
+tokenfile
+^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "none", "no", "none"
+
+The file containing the token to use to authenticate to the Kubernetes API
+server.  One of `tokenfile` or `token` is required if Kubernetes is configured
+with access control.  Example: `/etc/rsyslog.d/mmk8s.token`
+
+.. _token:
+
+token
+^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "none", "no", "none"
+
+The token to use to authenticate to the Kubernetes API server.  One of `token`
+or `tokenfile` is required if Kubernetes is configured with access control.
+Example: `UxMU46ptoEWOSqLNa1bFmH`
+
+.. _annotation_match:
+
+annotation_match
+^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "array", "none", "no", "none"
+
+By default no pod or namespace annotations will be added to the
+messages.  This parameter is an array of patterns to match the keys of
+the `annotations` field in the pod and namespace metadata to include
+in the `$!kubernetes!annotations` (for pod annotations) or the
+`$!kubernetes!namespace_annotations` (for namespace annotations)
+message properties.  Example: `["k8s.*master","k8s.*node"]`
+
+.. _srcmetadatapath:
+
+srcmetadatapath
+^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "$!metadata!filename", "no", "none"
+
+When reading json-file logs, with `imfile` and `addmetadata="on"`,
+this is the property where the filename is stored.
+
+.. _dstmetadatapath:
+
+dstmetadatapath
+^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "$!", "no", "none"
+
+This is the where the `kubernetes` and `docker` properties will be
+written.  By default, the module will add `$!kubernetes` and
+`$!docker`.
+
+.. _allowunsignedcerts:
+
+allowunsignedcerts
+^^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "boolean", "off", "no", "none"
+
+If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to
+`0`.  You are strongly discouraged to set this to `"on"`.  It is
+primarily useful only for debugging or testing.
+
+.. _de_dot:
+
+de_dot
+^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "boolean", "on", "no", "none"
+
+When processing labels and annotations, if this parameter is set to
+`"on"`, the key strings will have their `.` characters replaced with
+the string specified by the `de_dot_separator` parameter.
+
+.. _de_dot_separator:
+
+de_dot_separator
+^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "_", "no", "none"
+
+When processing labels and annotations, if the `de_dot` parameter is
+set to `"on"`, the key strings will have their `.` characters replaced
+with the string specified by the string value of this parameter.
+
+.. _filenamerules:
+
+filenamerules
+^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "SEE BELOW", "no", "none"
+
+.. note::
+
+    This directive is not supported with liblognorm 2.0.2 and earlier.
+
+When processing json-file logs, these are the lognorm rules to use to
+match the filename and extract metadata.  The default value is::
+
+    rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%conta\
+    iner_name:char-to:-%-%container_id:char-to:.%.log
+
+.. note::
+
+    In the above rules, the slashes ``\`` ending each line indicate
+    line wrapping - they are not part of the rule.
+
+There are two rules because the `container_hash` is optional.
+
+.. _filenamerulebase:
+
+filenamerulebase
+^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "/etc/rsyslog.d/k8s_filename.rulebase", "no", "none"
+
+When processing json-file logs, this is the rulebase used to
+match the filename and extract metadata.  For the actual rules, see
+below `filenamerules`.
+
+.. _containerrules:
+
+containerrules
+^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "SEE BELOW", "no", "none"
+
+.. note::
+
+    This directive is not supported with liblognorm 2.0.2 and earlier.
+
+For journald logs, there must be a message property `CONTAINER_NAME`
+which has a value matching these rules specified by this parameter.
+The default value is::
+
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:\
+    _%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_u\
+    sed_2:rest%
+    rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%na\
+    mespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%
+
+.. note::
+
+    In the above rules, the slashes ``\`` ending each line indicate
+    line wrapping - they are not part of the rule.
+
+There are two rules because the `container_hash` is optional.
+
+.. _containerrulebase:
+
+containerrulebase
+^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :header: "type", "default", "mandatory", "obsolete legacy directive"
+   :widths: auto
+   :class: parameter-table
+
+   "word", "/etc/rsyslog.d/k8s_container_name.rulebase", "no", "none"
+
+When processing json-file logs, this is the rulebase used to
+match the CONTAINER_NAME property value and extract metadata.  For the
+actual rules, see `containerrules`.
+
+Fields
+------
+
+These are the fields added from the metadata in the json-file filename, or from
+the `CONTAINER_NAME` and `CONTAINER_ID_FULL` fields from the `imjournal` input:
+
+`$!kubernetes!namespace_name`, `$!kubernetes!pod_name`,
+`$!kubernetes!container_name`, `$!docker!id`, `$!kubernetes!master_url`.
+
+If mmkubernetes can extract the above fields from the input, the following
+fields will always be present.  If they are not present, mmkubernetes
+failed to look up the namespace or pod in Kubernetes:
+
+`$!kubernetes!namespace_id`, `$!kubernetes!pod_id`,
+`$!kubernetes!creation_timestamp`, `$!kubernetes!host`
+
+The following fields may be present, depending on how the namespace and pod are
+defined in Kubernetes, and depending on the value of the directive
+`annotation_match`:
+
+`$!kubernetes!labels`, `$!kubernetes!annotations`, `$!kubernetes!namespace_labels`,
+`$!kubernetes!namespace_annotations`
+
+More fields may be added in the future.
+
+Example
+-------
+
+Assuming you have an `imfile` input reading from docker json-file container
+logs managed by Kubernetes, with `addmetadata="on"` so that mmkubernetes can
+get the basic necessary Kubernetes metadata from the filename:
+
+.. code-block:: none
+
+    input(type="imfile" file="/var/log/containers/*.log"
+          tag="kubernetes" addmetadata="on")
+
+and/or an `imjournal` input for docker journald container logs annotated by
+Kubernetes:
+
+.. code-block:: none
+
+    input(type="imjournal")
+
+Then mmkubernetes can be used to annotate log records like this:
+
+.. code-block:: none
+
+    module(load="mmkubernetes")
+
+    action(type="mmkubernetes")
+
+After this, you should have log records with fields described in the `Fields`
+section above.
+
+Credits
+-------
+
+This work is based on
+https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
+and has many of the same features.