diff --git a/source/configuration/modules/omelasticsearch.rst b/source/configuration/modules/omelasticsearch.rst
index 914fd67..4aee1ac 100644
--- a/source/configuration/modules/omelasticsearch.rst
+++ b/source/configuration/modules/omelasticsearch.rst
@@ -208,18 +208,354 @@ readability):
reconfiguration (e.g. dropping the mandatory attribute) a resubmit may
be succesful.
-**Samples:**
+.. _tls.cacert:
+
+tls.cacert
+^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "word", "none", "no", "none"
+
+This is the full path and file name of the file containing the CA cert for the
+CA that issued the Elasticsearch server cert. This file is in PEM format. For
+example: `/etc/rsyslog.d/es-ca.crt`
+
+.. _tls.mycert:
+
+tls.mycert
+^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "word", "none", "no", "none"
+
+This is the full path and file name of the file containing the client cert for
+doing client cert auth against Elasticsearch. This file is in PEM format. For
+example: `/etc/rsyslog.d/es-client-cert.pem`
+
+.. _tls.myprivkey:
+
+tls.myprivkey
+^^^^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "word", "none", "no", "none"
+
+This is the full path and file name of the file containing the private key
+corresponding to the cert `tls.mycert` used for doing client cert auth against
+Elasticsearch. This file is in PEM format, and must be unencrypted, so take
+care to secure it properly. For example: `/etc/rsyslog.d/es-client-key.pem`
+
+.. _omelasticsearch-bulkid:
+
+bulkid
+^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "word", "none", "no", "none"
+
+This is the unique id to assign to the record. The `bulk` part is misleading - this
+can be used in both bulk mode or in index
+(record at a time) mode. Although you can specify a static value for this
+parameter, you will almost always want to specify a *template* for the value of
+this parameter, and set `dynbulkid="on"` :ref:`omelasticsearch-dynbulkid`. NOTE:
+you must use `bulkid` and `dynbulkid` in order to use `writeoperation="create"`
+:ref:`omelasticsearch-writeoperation`.
+
+.. _omelasticsearch-dynbulkid:
+
+dynbulkid
+^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "binary", "off", "no", "none"
+
+If this parameter is set to `"on"`, then the `bulkid` parameter :ref:`omelasticsearch-bulkid`
+specifies a *template* to use to generate the unique id value to assign to the record. If
+using `bulkid` you will almost always want to set this parameter to `"on"` to assign
+a different unique id value to each record. NOTE:
+you must use `bulkid` and `dynbulkid` in order to use `writeoperation="create"`
+:ref:`omelasticsearch-writeoperation`.
+
+.. _omelasticsearch-writeoperation:
+
+writeoperation
+^^^^^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "word", "index", "no", "none"
+
+The value of this parameter is either `"index"` (the default) or `"create"`. If `"create"` is
+used, this means the bulk action/operation will be `create` - create a document only if the
+document does not already exist. The record must have a unique id in order to use `create`.
+See :ref:`omelasticsearch-bulkid` and :ref:`omelasticsearch-dynbulkid`. See
+:ref:`omelasticsearch-writeoperation-example` for an example.
+
+.. _omelasticsearch-retryfailures:
+
+retryfailures
+^^^^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "binary", "off", "no", "none"
+
+If this parameter is set to `"on"`, then the module will look for an
+`"errors":true` in the bulk index response. If found, each element in the
+response will be parsed to look for errors, since a bulk request may have some
+records which are successful and some which are failures. Failed requests will
+be converted back into records and resubmitted back to rsyslog for
+reprocessing. Each failed request will be resubmitted with a local variable
+called `$.omes`. This is a hash consisting of the fields from the response.
+See below :ref:`omelasticsearch-retry-example` for an example of how retry
+processing works.
+*NOTE* The retried record will be resubmitted at the "top" of your processing
+pipeline. If your processing pipeline is not idempotent (that is, your
+processing pipeline expects "raw" records), then you can specify a ruleset to
+redirect retries to. See :ref:`omelasticsearch-retryruleset` below.
+
+`$.omes` fields:
+
+* writeoperation - the operation used to submit the request - for rsyslog
+ omelasticsearch this currently means either `"index"` or `"create"`
+* status - the HTTP status code - typically an error will have a `4xx` or `5xx`
+ code - of particular note is `429` - this means Elasticsearch was unable to
+ process this bulk record request due to a temporary condition e.g. the bulk
+ index thread pool queue is full, and rsyslog should retry the operation.
+* _index, _type, _id - the metadata associated with the request
+* error - a hash containing one or more, possibly nested, fields containing
+ more detailed information about a failure. Typically there will be fields
+ `$.omes!error!type` (a keyword) and `$.omes!error!reason` (a longer string)
+ with more detailed information about the rejection. NOTE: The format is
+ apparently not described in great detail, so code must not make any
+ assumption about the availability of `error` or any specific sub-field.
+
+There may be other fields too - the code just copies everything in the
+response. Here is an example of a detailed error response, in JSON format, from
+Elasticsearch 5.6.9:
+
+.. code-block:: json
+
+ {"omes":
+ {"writeoperation": "create",
+ "_index": "rsyslog_testbench",
+ "_type": "test-type",
+ "_id": "92BE7AF79CD44305914C7658AF846A08",
+ "status": 400,
+ "error":
+ {"type": "mapper_parsing_exception",
+ "reason": "failed to parse [msgnum]",
+ "caused_by":
+ {"type": "number_format_exception",
+ "reason": "For input string: \"x00000025\""}}}}
+
+Reference: https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html#bulk
+
+.. _omelasticsearch-retryruleset:
+
+retryruleset
+^^^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "word", "", "no", "none"
+
+If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then
+this parameter has no effect. This parameter specifies the name of a ruleset
+to use to route retries. This is useful if you do not want retried messages to
+be processed starting from the top of your processing pipeline, or if you have
+multiple outputs but do not want to send retried Elasticsearch failures to all
+of your outputs, and you do not want to clutter your processing pipeline with a
+lot of conditionals. See below :ref:`omelasticsearch-retry-example` for an
+example of how retry processing works.
+
+.. _omelasticsearch-ratelimit.interval:
+
+ratelimit.interval
+^^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "integer", "600", "no", "none"
+
+If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then
+this parameter has no effect. Specifies the interval in seconds onto which
+rate-limiting is to be applied. If more than ratelimit.burst messages are read
+during that interval, further messages up to the end of the interval are
+discarded. The number of messages discarded is emitted at the end of the
+interval (if there were any discards).
+Setting this to value zero turns off ratelimiting.
+
+.. _omelasticsearch-ratelimit.burst:
+
+ratelimit.burst
+^^^^^^^^^^^^^^^
+
+.. csv-table::
+ :header: "type", "default", "mandatory", "obsolete legacy directive"
+ :widths: auto
+ :class: parameter-table
+
+ "integer", "20000", "no", "none"
+
+If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then
+this parameter has no effect. Specifies the maximum number of messages that
+can be emitted within the ratelimit.interval interval. For futher information,
+see description there.
+
+.. _omelasticsearch-statistic-counter:
+
+Statistic Counter
+=================
+
+This plugin maintains global statistics ,
+which accumulate all action instances. The statistic is named "omelasticsearch".
+Parameters are:
+
+- **submitted** - number of messages submitted for processing (with both
+ success and error result)
+
+- **fail.httprequests** - the number of times a http request failed. Note
+ that a single http request may be used to submit multiple messages, so this
+ number may be (much) lower than fail.http.
+
+- **fail.http** - number of message failures due to connection like-problems
+ (things like remote server down, broken link etc)
+
+- **fail.es** - number of failures due to elasticsearch error reply; Note that
+ this counter does NOT count the number of failed messages but the number of
+ times a failure occured (a potentially much smaller number). Counting messages
+ would be quite performance-intense and is thus not done.
+
+The following counters are available when `retryfailures="on"` is used:
+
+- **response.success** - number of records successfully sent in bulk index
+ requests - counts the number of successful responses
+
+- **response.bad** - number of times omelasticsearch received a response in a
+ bulk index response that was unrecognized or unable to be parsed. This may
+ indicate that omelasticsearch is attempting to communicate with a version of
+ Elasticsearch that is incompatible, or is otherwise sending back data in the
+ response that cannot be handled
+
+- **response.duplicate** - number of records in the bulk index request that
+ were duplicates of already existing records - this will only be reported if
+ using `writeoperation="create"` and `bulkid` to assign each record a unique
+ ID
+
+- **response.badargument** - number of times omelasticsearch received a
+ response that had a status indicating omelasticsearch sent bad data to
+ Elasticsearch. For example, status `400` and an error message indicating
+ omelasticsearch attempted to store a non-numeric string value in a numeric
+ field.
+
+- **response.bulkrejection** - number of times omelasticsearch received a
+ response that had a status indicating Elasticsearch was unable to process
+ the record at this time - status `429`. The record can be retried.
+
+- **response.other** - number of times omelasticsearch received a
+ response not recognized as one of the above responses, typically some other
+ `4xx` or `5xx` http status.
+
+**The fail.httprequests and fail.http counters reflect only failures that
+omelasticsearch detected.** Once it detects problems, it (usually, depends on
+circumstances) tell the rsyslog core that it wants to be suspended until the
+situation clears (this is a requirement for rsyslog output modules). Once it is
+suspended, it does NOT receive any further messages. Depending on the user
+configuration, messages will be lost during this period. Those lost messages will
+NOT be counted by impstats (as it does not see them).
+
+Note that some previous (pre 7.4.5) versions of this plugin had different counters.
+These were experimental and confusing. The only ones really used were "submits",
+which were the number of successfully processed messages and "connfail" which were
+equivalent to "failed.http".
+
+How Retries Are Handled
+=======================
+
+When using `retryfailures="on"` (:ref:`omelasticsearch-retryfailures`), the
+original `Message` object (that is, the original `smsg_t *msg` object) **is not
+available**. This means none of the metadata associated with that object, such
+as various timestamps, hosts/ip addresses, etc. are not available for the retry
+operation. The only thing available is the original JSON string sent in the
+original request, and whatever data is returned in the error response, which
+will contain the Elasticsearch metadata about the index, type, and id, and will
+be made available in the `$.omes` fields. For the message to retry, the code
+will take the original JSON string and parse it back into an internal `Message`
+object. This means you **may need to use a different template** to output
+messages for your retry ruleset. For example, if you used the following
+template to format the Elasticsearch message for the initial submission:
+
+.. code-block:: none
+
+ template(name="es_output_template"
+ type="list"
+ option.json="on") {
+ constant(value="{")
+ constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
+ constant(value="\",\"message\":\"") property(name="msg")
+ constant(value="\",\"host\":\"") property(name="hostname")
+ constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
+ constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
+ constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
+ constant(value="\"}")
+ }
+
+You would have to use a different template for the retry, since none of the
+`timereported`, `msg`, etc. fields will have the same values for the retry as
+for the initial try.
+
+Examples
+========
+
+Example 1
+^^^^^^^^^
The following sample does the following:
- loads the omelasticsearch module
- outputs all logs to Elasticsearch using the default settings
-::
+.. code-block:: none
module(load="omelasticsearch")
*.* action(type="omelasticsearch")
+Example 2
+^^^^^^^^^
+
The following sample does the following:
- loads the omelasticsearch module
@@ -246,7 +582,7 @@ The following sample does the following:
- retry indefinitely if the HTTP request failed (eg: if the target
server is down)
-::
+.. code-block:: none
module(load="omelasticsearch")
template(name="testTemplate"
@@ -274,6 +610,87 @@ The following sample does the following:
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
+.. _omelasticsearch-writeoperation-example:
+
+Example 3
+^^^^^^^^^
+
+The following sample shows how to use :ref:`omelasticsearch-writeoperation`
+with :ref:`omelasticsearch-dynbulkid` and :ref:`omelasticsearch-bulkid`. For
+simplicity, it assumes rsyslog has been built with `--enable-libuuid` which
+provides the `uuid` property for each record:
+
+.. code-block:: none
+
+ module(load="omelasticsearch")
+ set $!es_record_id = $uuid;
+ template(name="bulkid-template" type="list") { property(name="$!es_record_id") }
+ action(type="omelasticsearch"
+ ...
+ bulkmode="on"
+ bulkid="bulkid-template"
+ dynbulkid="on"
+ writeoperation="create")
+
+
+.. _omelasticsearch-retry-example:
+
+Example 4
+^^^^^^^^^
+
+The following sample shows how to use :ref:`omelasticsearch-retryfailures` to
+process, discard, or retry failed operations. This uses
+`writeoperation="create"` with a unique `bulkid` so that we can check for and
+discard duplicate messages as successful. The `try_es` ruleset is used both
+for the initial attempt and any subsequent retries. The code in the ruleset
+assumes that if `$.omes!status` is set and is non-zero, this is a retry for a
+previously failed operation. If the status was successful, or Elasticsearch
+said this was a duplicate, the record is already in Elasticsearch, so we can
+drop the record. If there was some error processing the response
+e.g. Elasticsearch sent a response formatted in some way that we did not know
+how to process, then submit the record to the `error_es` ruleset. If the
+response was a "hard" error like `400`, then submit the record to the
+`error_es` ruleset. In any other case, such as a status `429` or `5xx`, the
+record will be resubmitted to Elasticsearch. In the example, the `error_es`
+ruleset just dumps the records to a file.
+
+.. code-block:: none
+
+ module(load="omelasticsearch")
+ module(load="omfile")
+ set $!es_record_id = $uuid;
+ template(name="bulkid-template" type="list") { property(name="$!es_record_id") }
+
+ ruleset(name="error_es") {
+ action(type="omfile" template="RSYSLOG_DebugFormat" file="es-bulk-errors.log")
+ }
+
+ ruleset(name="try_es") {
+ if strlen($.omes!status) > 0 then {
+ # retry case
+ if ($.omes!status == 200) or ($.omes!status == 201) or (($.omes!status == 409) and ($.omes!writeoperation == "create")) then {
+ stop # successful
+ }
+ if ($.omes!writeoperation == "unknown") or (strlen($.omes!error!type) == 0) or (strlen($.omes!error!reason) == 0) then {
+ call error_es
+ stop
+ }
+ if ($.omes!status == 400) or ($.omes!status < 200) then {
+ call error_es
+ stop
+ }
+ # else fall through to retry operation
+ }
+ action(type="omelasticsearch"
+ ...
+ bulkmode="on"
+ bulkid="bulkid-template"
+ dynbulkid="on"
+ writeoperation="create"
+ retryfailures="on"
+ retryruleset="try_es")
+ }
+ call try_es
This documentation is part of the `rsyslog <http://www.rsyslog.com/>`_
project.