diff --git a/SOURCES/rsyslog-8.24.0-doc-rhbz1459896-queues-defaults.patch b/SOURCES/rsyslog-8.24.0-doc-rhbz1459896-queues-defaults.patch new file mode 100644 index 0000000..e7bdc91 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-doc-rhbz1459896-queues-defaults.patch @@ -0,0 +1,138 @@ +From c8be9a713a57f07311560af50c24267b30bef21b Mon Sep 17 00:00:00 2001 +From: Jiri Vymazal +Date: Tue, 29 Aug 2017 16:32:15 +0200 +Subject: [PATCH] fixed queue default values + +--- + source/concepts/queues.rst | 7 +++---- + source/configuration/global/index.rst | 6 +++--- + source/configuration/global/options/rsconf1_mainmsgqueuesize.rst | 2 +- + source/rainerscript/queue_parameters.rst | 15 ++++++++++++--- + source/configuration/action/index.rst | 12 ++++++------ + 5 files changed, 24 insertions(+), 16 deletions(-) + +diff --git a/source/concepts/queues.rst b/source/concepts/queues.rst +index c71413c..9b41128 100644 +--- a/source/concepts/queues.rst ++++ b/source/concepts/queues.rst +@@ -273,10 +273,9 @@ actually needed. + The water marks can be set via the "*$QueueHighWatermark*\ " + and  "*$QueueLowWatermark*\ " configuration file directives. + Note that these are actual numbers, not percentages. Be sure they make +-sense (also in respect to "*$QueueSize*\ "), as rsyslodg does +-currently not perform any checks on the numbers provided. It is easy to +-screw up the system here (yes, a feature enhancement request is filed +-;)). ++sense (also in respect to "*$QueueSize*\ "). Rsyslodg does ++perform some checks on the numbers provided, and issues warning when ++numbers are "suspicious". + + Limiting the Queue Size + ----------------------- +diff --git a/source/configuration/global/index.rst b/source/configuration/global/index.rst +index 2738f21..a53ef23 100644 +--- a/source/configuration/global/index.rst ++++ b/source/configuration/global/index.rst +@@ -137,13 +137,13 @@ To understand queue parameters, read + - **$MainMsgQueueDequeueSlowdown** [number is timeout in + *micro*\ seconds (1000000us is 1sec!), default 0 (no delay). Simple + rate-limiting!] +-- **$MainMsgQueueDiscardMark** [default 9750] ++- **$MainMsgQueueDiscardMark** [default 98000] + - **$MainMsgQueueDiscardSeverity** [either a textual or + numerical severity! default 4 (warning)] + - **$MainMsgQueueFileName** +-- **$MainMsgQueueHighWaterMark** [default 8000] ++- **$MainMsgQueueHighWaterMark** [default 80000] + - **$MainMsgQueueImmediateShutdown** [on/**off**] +-- **$MainMsgQueueLowWaterMark** [default 2000] ++- **$MainMsgQueueLowWaterMark** [default 20000] + - **$MainMsgQueueMaxFileSize** , default 1m + - **$MainMsgQueueTimeoutActionCompletion** [number is timeout in + ms (1000ms is 1sec!), default 1000, 0 means immediate!] +diff --git a/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst b/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst +index 050407c..3e902cf 100644 +--- a/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst ++++ b/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst +@@ -3,7 +3,7 @@ $MainMsgQueueSize + + **Type:** global configuration directive + +-**Default:** 10000 ++**Default:** 100000 + + **Description:** + +diff --git a/source/rainerscript/queue_parameters.rst b/source/rainerscript/queue_parameters.rst +index 4453721..3f2b7a2 100644 +--- a/source/rainerscript/queue_parameters.rst ++++ b/source/rainerscript/queue_parameters.rst +@@ -33,8 +33,14 @@ read the :doc:`queues <../concepts/queues>` documentation. + For more information on the current status of this restriction see + the `rsyslog FAQ: "lower bound for queue + sizes" `_. ++ ++ The default depends on queue type and, if you need ++ a specific value, please specify it. Otherwise rsyslog selects what ++ it consideres appropriate. For example, ruleset queues have a default ++ size of 50000 and action queues which are configured to be non-direct ++ have a size of 1000. + - **queue.dequeuebatchsize** number +- default 16 ++ default 128 + - **queue.maxdiskspace** number + The maximum size that all queue files together will use on disk. Note + that the actual size may be slightly larger than the configured max, +@@ -46,8 +47,9 @@ read the :doc:`queues <../concepts/queues>` documentation. + processing, because disk queue mode is very considerably slower than + in-memory queue mode. Going to disk should be reserved for cases + where an output action destination is offline for some period. ++ default 90% of queue size + - **queue.lowwatermark** number +- default 2000 ++ default 70% of queue size + - **queue.fulldelaymark** number + Number of messages when the queue should block delayable messages. + Messages are NO LONGER PROCESSED until the queue has sufficient space +@@ -59,9 +61,11 @@ read the :doc:`queues <../concepts/queues>` documentation. + out of space. Please note that if you use a DA queue, setting the + fulldelaymark BELOW the highwatermark makes the queue never activate + disk mode for delayable inputs. So this is probably not what you want. ++ default 97% of queue size + - **queue.lightdelaymark** number ++ default 70% of queue size + - **queue.discardmark** number +- default 9750 ++ default 80% of queue size + - **queue.discardseverity** number + \*numerical\* severity! default 8 (nothing discarded) + - **queue.checkpointinterval** number +diff --git a/source/configuration/action/index.rst b/source/configuration/action/index.rst +index 3e7cd24..9352866 100644 +--- a/source/configuration/action/index.rst ++++ b/source/configuration/action/index.rst +@@ -163,18 +163,18 @@ following action, only. The next and all other actions will be + in "direct" mode (no real queue) if not explicitely specified otherwise. + + - **$ActionQueueCheckpointInterval** +-- **$ActionQueueDequeueBatchSize** [default 16] ++- **$ActionQueueDequeueBatchSize** [default 128] + - **$ActionQueueDequeueSlowdown** [number is timeout in + *micro*\ seconds (1000000us is 1sec!), default 0 (no delay). Simple + rate-limiting!] +-- **$ActionQueueDiscardMark** [default 9750] +-- **$ActionQueueDiscardSeverity** [\*numerical\* severity! default +- 4 (warning)] ++- **$ActionQueueDiscardMark** [default 80% of queue size] ++- **$ActionQueueDiscardSeverity** [\*numerical\* severity! default ++ 8 (nothing discarded)] + - **$ActionQueueFileName** +-- **$ActionQueueHighWaterMark** [default 8000] ++- **$ActionQueueHighWaterMark** [default 90% of queue size] + - **$ActionQueueImmediateShutdown** [on/**off**] + - **$ActionQueueSize** +-- **$ActionQueueLowWaterMark** [default 2000] ++- **$ActionQueueLowWaterMark** [default 70% of queue size] + - **$ActionQueueMaxFileSize** , default 1m + - **$ActionQueueTimeoutActionCompletion** [number is timeout in ms + (1000ms is 1sec!), default 1000, 0 means immediate!] diff --git a/SOURCES/rsyslog-8.24.0-doc-rhbz1507028-recover_qi.patch b/SOURCES/rsyslog-8.24.0-doc-rhbz1507028-recover_qi.patch new file mode 100644 index 0000000..42a69b1 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-doc-rhbz1507028-recover_qi.patch @@ -0,0 +1,27 @@ +From ff07a7cfc171dc2151cc8afe44776525d34a9e01 Mon Sep 17 00:00:00 2001 +From: jvymazal +Date: Tue, 3 Jan 2017 10:24:26 +0100 +Subject: [PATCH] Update queues.rst + +Update queues.rst +--- + source/concepts/queues.rst | 6 ++++++ + 1 file changed, 6 insertions(+) + +diff --git a/source/concepts/queues.rst b/source/concepts/queues.rst +index eb394e8..c71413c 100644 +--- a/source/concepts/queues.rst ++++ b/source/concepts/queues.rst +@@ -153,6 +153,12 @@ can be requested via "*QueueSyncQueueFiles on/off* with the + default being off. Activating this option has a performance penalty, so + it should not be turned on without reason. + ++If you happen to lose or otherwise need the housekeeping structures and ++have all yours queue chunks you can use perl script included in rsyslog ++package to generate it. ++Usage: recover_qi.pl -w *$WorkDirectory* -f QueueFileName -d 8 > QueueFileName.qi ++ ++ + In-Memory Queues + ~~~~~~~~~~~~~~~~ + diff --git a/SOURCES/rsyslog-8.24.0-doc-rhbz1507145-omelastic-client-cert-and-config.patch b/SOURCES/rsyslog-8.24.0-doc-rhbz1507145-omelastic-client-cert-and-config.patch new file mode 100644 index 0000000..ae24862 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-doc-rhbz1507145-omelastic-client-cert-and-config.patch @@ -0,0 +1,458 @@ +diff --git a/source/configuration/modules/omelasticsearch.rst b/source/configuration/modules/omelasticsearch.rst +index 914fd67..4aee1ac 100644 +--- a/source/configuration/modules/omelasticsearch.rst ++++ b/source/configuration/modules/omelasticsearch.rst +@@ -208,18 +208,354 @@ readability): + reconfiguration (e.g. dropping the mandatory attribute) a resubmit may + be succesful. + +-**Samples:** ++.. _tls.cacert: ++ ++tls.cacert ++^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "none", "no", "none" ++ ++This is the full path and file name of the file containing the CA cert for the ++CA that issued the Elasticsearch server cert. This file is in PEM format. For ++example: `/etc/rsyslog.d/es-ca.crt` ++ ++.. _tls.mycert: ++ ++tls.mycert ++^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "none", "no", "none" ++ ++This is the full path and file name of the file containing the client cert for ++doing client cert auth against Elasticsearch. This file is in PEM format. For ++example: `/etc/rsyslog.d/es-client-cert.pem` ++ ++.. _tls.myprivkey: ++ ++tls.myprivkey ++^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "none", "no", "none" ++ ++This is the full path and file name of the file containing the private key ++corresponding to the cert `tls.mycert` used for doing client cert auth against ++Elasticsearch. This file is in PEM format, and must be unencrypted, so take ++care to secure it properly. For example: `/etc/rsyslog.d/es-client-key.pem` ++ ++.. _omelasticsearch-bulkid: ++ ++bulkid ++^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "none", "no", "none" ++ ++This is the unique id to assign to the record. The `bulk` part is misleading - this ++can be used in both bulk mode or in index ++(record at a time) mode. Although you can specify a static value for this ++parameter, you will almost always want to specify a *template* for the value of ++this parameter, and set `dynbulkid="on"` :ref:`omelasticsearch-dynbulkid`. NOTE: ++you must use `bulkid` and `dynbulkid` in order to use `writeoperation="create"` ++:ref:`omelasticsearch-writeoperation`. ++ ++.. _omelasticsearch-dynbulkid: ++ ++dynbulkid ++^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "binary", "off", "no", "none" ++ ++If this parameter is set to `"on"`, then the `bulkid` parameter :ref:`omelasticsearch-bulkid` ++specifies a *template* to use to generate the unique id value to assign to the record. If ++using `bulkid` you will almost always want to set this parameter to `"on"` to assign ++a different unique id value to each record. NOTE: ++you must use `bulkid` and `dynbulkid` in order to use `writeoperation="create"` ++:ref:`omelasticsearch-writeoperation`. ++ ++.. _omelasticsearch-writeoperation: ++ ++writeoperation ++^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "index", "no", "none" ++ ++The value of this parameter is either `"index"` (the default) or `"create"`. If `"create"` is ++used, this means the bulk action/operation will be `create` - create a document only if the ++document does not already exist. The record must have a unique id in order to use `create`. ++See :ref:`omelasticsearch-bulkid` and :ref:`omelasticsearch-dynbulkid`. See ++:ref:`omelasticsearch-writeoperation-example` for an example. ++ ++.. _omelasticsearch-retryfailures: ++ ++retryfailures ++^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "binary", "off", "no", "none" ++ ++If this parameter is set to `"on"`, then the module will look for an ++`"errors":true` in the bulk index response. If found, each element in the ++response will be parsed to look for errors, since a bulk request may have some ++records which are successful and some which are failures. Failed requests will ++be converted back into records and resubmitted back to rsyslog for ++reprocessing. Each failed request will be resubmitted with a local variable ++called `$.omes`. This is a hash consisting of the fields from the response. ++See below :ref:`omelasticsearch-retry-example` for an example of how retry ++processing works. ++*NOTE* The retried record will be resubmitted at the "top" of your processing ++pipeline. If your processing pipeline is not idempotent (that is, your ++processing pipeline expects "raw" records), then you can specify a ruleset to ++redirect retries to. See :ref:`omelasticsearch-retryruleset` below. ++ ++`$.omes` fields: ++ ++* writeoperation - the operation used to submit the request - for rsyslog ++ omelasticsearch this currently means either `"index"` or `"create"` ++* status - the HTTP status code - typically an error will have a `4xx` or `5xx` ++ code - of particular note is `429` - this means Elasticsearch was unable to ++ process this bulk record request due to a temporary condition e.g. the bulk ++ index thread pool queue is full, and rsyslog should retry the operation. ++* _index, _type, _id - the metadata associated with the request ++* error - a hash containing one or more, possibly nested, fields containing ++ more detailed information about a failure. Typically there will be fields ++ `$.omes!error!type` (a keyword) and `$.omes!error!reason` (a longer string) ++ with more detailed information about the rejection. NOTE: The format is ++ apparently not described in great detail, so code must not make any ++ assumption about the availability of `error` or any specific sub-field. ++ ++There may be other fields too - the code just copies everything in the ++response. Here is an example of a detailed error response, in JSON format, from ++Elasticsearch 5.6.9: ++ ++.. code-block:: json ++ ++ {"omes": ++ {"writeoperation": "create", ++ "_index": "rsyslog_testbench", ++ "_type": "test-type", ++ "_id": "92BE7AF79CD44305914C7658AF846A08", ++ "status": 400, ++ "error": ++ {"type": "mapper_parsing_exception", ++ "reason": "failed to parse [msgnum]", ++ "caused_by": ++ {"type": "number_format_exception", ++ "reason": "For input string: \"x00000025\""}}}} ++ ++Reference: https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html#bulk ++ ++.. _omelasticsearch-retryruleset: ++ ++retryruleset ++^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "", "no", "none" ++ ++If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then ++this parameter has no effect. This parameter specifies the name of a ruleset ++to use to route retries. This is useful if you do not want retried messages to ++be processed starting from the top of your processing pipeline, or if you have ++multiple outputs but do not want to send retried Elasticsearch failures to all ++of your outputs, and you do not want to clutter your processing pipeline with a ++lot of conditionals. See below :ref:`omelasticsearch-retry-example` for an ++example of how retry processing works. ++ ++.. _omelasticsearch-ratelimit.interval: ++ ++ratelimit.interval ++^^^^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "integer", "600", "no", "none" ++ ++If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then ++this parameter has no effect. Specifies the interval in seconds onto which ++rate-limiting is to be applied. If more than ratelimit.burst messages are read ++during that interval, further messages up to the end of the interval are ++discarded. The number of messages discarded is emitted at the end of the ++interval (if there were any discards). ++Setting this to value zero turns off ratelimiting. ++ ++.. _omelasticsearch-ratelimit.burst: ++ ++ratelimit.burst ++^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "integer", "20000", "no", "none" ++ ++If `retryfailures` is not `"on"` (:ref:`omelasticsearch-retryfailures`) then ++this parameter has no effect. Specifies the maximum number of messages that ++can be emitted within the ratelimit.interval interval. For futher information, ++see description there. ++ ++.. _omelasticsearch-statistic-counter: ++ ++Statistic Counter ++================= ++ ++This plugin maintains global statistics , ++which accumulate all action instances. The statistic is named "omelasticsearch". ++Parameters are: ++ ++- **submitted** - number of messages submitted for processing (with both ++ success and error result) ++ ++- **fail.httprequests** - the number of times a http request failed. Note ++ that a single http request may be used to submit multiple messages, so this ++ number may be (much) lower than fail.http. ++ ++- **fail.http** - number of message failures due to connection like-problems ++ (things like remote server down, broken link etc) ++ ++- **fail.es** - number of failures due to elasticsearch error reply; Note that ++ this counter does NOT count the number of failed messages but the number of ++ times a failure occured (a potentially much smaller number). Counting messages ++ would be quite performance-intense and is thus not done. ++ ++The following counters are available when `retryfailures="on"` is used: ++ ++- **response.success** - number of records successfully sent in bulk index ++ requests - counts the number of successful responses ++ ++- **response.bad** - number of times omelasticsearch received a response in a ++ bulk index response that was unrecognized or unable to be parsed. This may ++ indicate that omelasticsearch is attempting to communicate with a version of ++ Elasticsearch that is incompatible, or is otherwise sending back data in the ++ response that cannot be handled ++ ++- **response.duplicate** - number of records in the bulk index request that ++ were duplicates of already existing records - this will only be reported if ++ using `writeoperation="create"` and `bulkid` to assign each record a unique ++ ID ++ ++- **response.badargument** - number of times omelasticsearch received a ++ response that had a status indicating omelasticsearch sent bad data to ++ Elasticsearch. For example, status `400` and an error message indicating ++ omelasticsearch attempted to store a non-numeric string value in a numeric ++ field. ++ ++- **response.bulkrejection** - number of times omelasticsearch received a ++ response that had a status indicating Elasticsearch was unable to process ++ the record at this time - status `429`. The record can be retried. ++ ++- **response.other** - number of times omelasticsearch received a ++ response not recognized as one of the above responses, typically some other ++ `4xx` or `5xx` http status. ++ ++**The fail.httprequests and fail.http counters reflect only failures that ++omelasticsearch detected.** Once it detects problems, it (usually, depends on ++circumstances) tell the rsyslog core that it wants to be suspended until the ++situation clears (this is a requirement for rsyslog output modules). Once it is ++suspended, it does NOT receive any further messages. Depending on the user ++configuration, messages will be lost during this period. Those lost messages will ++NOT be counted by impstats (as it does not see them). ++ ++Note that some previous (pre 7.4.5) versions of this plugin had different counters. ++These were experimental and confusing. The only ones really used were "submits", ++which were the number of successfully processed messages and "connfail" which were ++equivalent to "failed.http". ++ ++How Retries Are Handled ++======================= ++ ++When using `retryfailures="on"` (:ref:`omelasticsearch-retryfailures`), the ++original `Message` object (that is, the original `smsg_t *msg` object) **is not ++available**. This means none of the metadata associated with that object, such ++as various timestamps, hosts/ip addresses, etc. are not available for the retry ++operation. The only thing available is the original JSON string sent in the ++original request, and whatever data is returned in the error response, which ++will contain the Elasticsearch metadata about the index, type, and id, and will ++be made available in the `$.omes` fields. For the message to retry, the code ++will take the original JSON string and parse it back into an internal `Message` ++object. This means you **may need to use a different template** to output ++messages for your retry ruleset. For example, if you used the following ++template to format the Elasticsearch message for the initial submission: ++ ++.. code-block:: none ++ ++ template(name="es_output_template" ++          type="list" ++          option.json="on") { ++            constant(value="{") ++              constant(value="\"timestamp\":\"")      property(name="timereported" dateFormat="rfc3339") ++              constant(value="\",\"message\":\"")     property(name="msg") ++              constant(value="\",\"host\":\"")        property(name="hostname") ++              constant(value="\",\"severity\":\"")    property(name="syslogseverity-text") ++              constant(value="\",\"facility\":\"")    property(name="syslogfacility-text") ++              constant(value="\",\"syslogtag\":\"")   property(name="syslogtag") ++            constant(value="\"}") ++          } ++ ++You would have to use a different template for the retry, since none of the ++`timereported`, `msg`, etc. fields will have the same values for the retry as ++for the initial try. ++ ++Examples ++======== ++ ++Example 1 ++^^^^^^^^^ + + The following sample does the following: + + - loads the omelasticsearch module + - outputs all logs to Elasticsearch using the default settings + +-:: ++.. code-block:: none + + module(load="omelasticsearch") + *.* action(type="omelasticsearch") + ++Example 2 ++^^^^^^^^^ ++ + The following sample does the following: + + - loads the omelasticsearch module +@@ -246,7 +582,7 @@ The following sample does the following: + - retry indefinitely if the HTTP request failed (eg: if the target + server is down) + +-:: ++.. code-block:: none + + module(load="omelasticsearch") + template(name="testTemplate" +@@ -274,6 +610,87 @@ The following sample does the following: +        queue.dequeuebatchsize="300" +        action.resumeretrycount="-1") + ++.. _omelasticsearch-writeoperation-example: ++ ++Example 3 ++^^^^^^^^^ ++ ++The following sample shows how to use :ref:`omelasticsearch-writeoperation` ++with :ref:`omelasticsearch-dynbulkid` and :ref:`omelasticsearch-bulkid`. For ++simplicity, it assumes rsyslog has been built with `--enable-libuuid` which ++provides the `uuid` property for each record: ++ ++.. code-block:: none ++ ++ module(load="omelasticsearch") ++ set $!es_record_id = $uuid; ++ template(name="bulkid-template" type="list") { property(name="$!es_record_id") } ++ action(type="omelasticsearch" ++        ... ++        bulkmode="on" ++        bulkid="bulkid-template" ++        dynbulkid="on" ++        writeoperation="create") ++ ++ ++.. _omelasticsearch-retry-example: ++ ++Example 4 ++^^^^^^^^^ ++ ++The following sample shows how to use :ref:`omelasticsearch-retryfailures` to ++process, discard, or retry failed operations. This uses ++`writeoperation="create"` with a unique `bulkid` so that we can check for and ++discard duplicate messages as successful. The `try_es` ruleset is used both ++for the initial attempt and any subsequent retries. The code in the ruleset ++assumes that if `$.omes!status` is set and is non-zero, this is a retry for a ++previously failed operation. If the status was successful, or Elasticsearch ++said this was a duplicate, the record is already in Elasticsearch, so we can ++drop the record. If there was some error processing the response ++e.g. Elasticsearch sent a response formatted in some way that we did not know ++how to process, then submit the record to the `error_es` ruleset. If the ++response was a "hard" error like `400`, then submit the record to the ++`error_es` ruleset. In any other case, such as a status `429` or `5xx`, the ++record will be resubmitted to Elasticsearch. In the example, the `error_es` ++ruleset just dumps the records to a file. ++ ++.. code-block:: none ++ ++ module(load="omelasticsearch") ++ module(load="omfile") ++ set $!es_record_id = $uuid; ++ template(name="bulkid-template" type="list") { property(name="$!es_record_id") } ++ ++ ruleset(name="error_es") { ++ action(type="omfile" template="RSYSLOG_DebugFormat" file="es-bulk-errors.log") ++ } ++ ++ ruleset(name="try_es") { ++ if strlen($.omes!status) > 0 then { ++ # retry case ++ if ($.omes!status == 200) or ($.omes!status == 201) or (($.omes!status == 409) and ($.omes!writeoperation == "create")) then { ++ stop # successful ++ } ++ if ($.omes!writeoperation == "unknown") or (strlen($.omes!error!type) == 0) or (strlen($.omes!error!reason) == 0) then { ++ call error_es ++ stop ++ } ++ if ($.omes!status == 400) or ($.omes!status < 200) then { ++ call error_es ++ stop ++ } ++ # else fall through to retry operation ++ } ++ action(type="omelasticsearch" ++        ... ++        bulkmode="on" ++        bulkid="bulkid-template" ++        dynbulkid="on" ++        writeoperation="create" ++        retryfailures="on" ++        retryruleset="try_es") ++ } ++ call try_es + + This documentation is part of the `rsyslog `_ + project. diff --git a/SOURCES/rsyslog-8.24.0-doc-rhbz1538372-imjournal-duplicates.patch b/SOURCES/rsyslog-8.24.0-doc-rhbz1538372-imjournal-duplicates.patch new file mode 100644 index 0000000..88e4859 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-doc-rhbz1538372-imjournal-duplicates.patch @@ -0,0 +1,28 @@ +From 1dbb68f3dc5c7ae94bdea5ad37296cbc2224e92b Mon Sep 17 00:00:00 2001 +From: Jiri Vymazal +Date: Wed, 25 Jul 2018 14:24:57 +0200 +Subject: [PATCH] Added WorkAroundJournalBug parameter + +this is documentation for rsyslog/rsyslog#2543 +--- + source/configuration/modules/imjournal.rst | 7 +++++++ + 1 file changed, 7 insertions(+) + +diff --git a/source/configuration/modules/imjournal.rst b/source/configuration/modules/imjournal.rst +index 2530ddfe..85ca9e7d 100644 +--- a/source/configuration/modules/imjournal.rst ++++ b/source/configuration/modules/imjournal.rst +@@ -99,6 +99,13 @@ - **usepidfromsystem** [**off**/on] + Retrieves the trusted systemd parameter, _PID, instead of the user + systemd parameter, SYSLOG_PID, which is the default. + ++- **WorkAroundJournalBug** [**off**/on] ++ ++ When journald instance rotates its files it is possible that duplicate records ++ appear in rsyslog. If you turn on this option imjournal will keep track of cursor ++ with each message to work around this problem. Be aware that in some cases this ++ might result in imjournal performance hit. ++ + **Caveats/Known Bugs:** + + - As stated above, a corrupted systemd journal database can cause major diff --git a/SOURCES/rsyslog-8.24.0-doc-rhbz1539193-mmkubernetes-new-plugin.patch b/SOURCES/rsyslog-8.24.0-doc-rhbz1539193-mmkubernetes-new-plugin.patch new file mode 100644 index 0000000..f2ad9f7 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-doc-rhbz1539193-mmkubernetes-new-plugin.patch @@ -0,0 +1,384 @@ +diff --git a/source/configuration/modules/mmkubernetes.rst b/source/configuration/modules/mmkubernetes.rst +new file mode 100644 +index 0000000..1cd3d2a +--- /dev/null ++++ b/source/configuration/modules/mmkubernetes.rst +@@ -0,0 +1,378 @@ ++***************************************** ++Kubernetes Metadata Module (mmkubernetes) ++***************************************** ++ ++=========================== =========================================================================== ++**Module Name:**  **mmkubernetes** ++**Author:** `Tomáš Heinrich` ++ `Rich Megginson` ++=========================== =========================================================================== ++ ++Purpose ++======= ++ ++This module is used to add `Kubernetes ` ++metadata to log messages logged by containers running in Kubernetes. ++It will add the namespace uuid, pod uuid, pod and namespace labels and ++annotations, and other metadata associated with the pod and ++namespace. ++ ++.. note:: ++ ++ This **only** works with log files in `/var/log/containers/*.log` ++ (docker `--log-driver=json-file`), or with journald entries with ++ message properties `CONTAINER_NAME` and `CONTAINER_ID_FULL` (docker ++ `--log-driver=journald`), and when the application running inside ++ the container writes logs to `stdout`/`stderr`. This **does not** ++ currently work with other log drivers. ++ ++For json-file logs, you must use the `imfile` module with the ++`addmetadata="on"` parameter, and the filename must match the ++liblognorm rules specified by the `filenamerules` ++(:ref:`filenamerules`) or `filenamerulebase` (:ref:`filenamerulebase`) ++parameter values. ++ ++For journald logs, there must be a message property `CONTAINER_NAME` ++which matches the liblognorm rules specified by the `containerrules` ++(:ref:`containerrules`) or `containerrulebase` ++(:ref:`containerrulebase`) parameter values. The record must also have ++the message property `CONTAINER_ID_FULL`. ++ ++This module is implemented via the output module interface. This means ++that mmkubernetes should be called just like an action. After it has ++been called, there will be two new message properties: `kubernetes` ++and `docker`. There will be subfields of each one for the various ++metadata items: `$!kubernetes!namespace_name` ++`$!kubernetes!labels!this-is-my-label`, etc. There is currently only ++1 docker subfield: `$!docker!container_id`. See ++https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml ++and ++https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml ++for more details. ++ ++Configuration Parameters ++======================== ++ ++.. note:: ++ ++ Parameter names are case-insensitive. ++ ++Module Parameters and Action Parameters ++--------------------------------------- ++ ++.. _kubernetesurl: ++ ++KubernetesURL ++^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "https://kubernetes.default.svc.cluster.local:443", "yes", "none" ++ ++The URL of the Kubernetes API server. Example: `https://localhost:8443`. ++ ++.. _mmkubernetes-tls.cacert: ++ ++tls.cacert ++^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "none", "no", "none" ++ ++Full path and file name of file containing the CA cert of the ++Kubernetes API server cert issuer. Example: `/etc/rsyslog.d/mmk8s-ca.crt`. ++This parameter is not mandatory if using an `http` scheme instead of `https` in ++`kubernetesurl`, or if using `allowunsignedcerts="yes"`. ++ ++.. _tokenfile: ++ ++tokenfile ++^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "none", "no", "none" ++ ++The file containing the token to use to authenticate to the Kubernetes API ++server. One of `tokenfile` or `token` is required if Kubernetes is configured ++with access control. Example: `/etc/rsyslog.d/mmk8s.token` ++ ++.. _token: ++ ++token ++^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "none", "no", "none" ++ ++The token to use to authenticate to the Kubernetes API server. One of `token` ++or `tokenfile` is required if Kubernetes is configured with access control. ++Example: `UxMU46ptoEWOSqLNa1bFmH` ++ ++.. _annotation_match: ++ ++annotation_match ++^^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "array", "none", "no", "none" ++ ++By default no pod or namespace annotations will be added to the ++messages. This parameter is an array of patterns to match the keys of ++the `annotations` field in the pod and namespace metadata to include ++in the `$!kubernetes!annotations` (for pod annotations) or the ++`$!kubernetes!namespace_annotations` (for namespace annotations) ++message properties. Example: `["k8s.*master","k8s.*node"]` ++ ++.. _srcmetadatapath: ++ ++srcmetadatapath ++^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "$!metadata!filename", "no", "none" ++ ++When reading json-file logs, with `imfile` and `addmetadata="on"`, ++this is the property where the filename is stored. ++ ++.. _dstmetadatapath: ++ ++dstmetadatapath ++^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "$!", "no", "none" ++ ++This is the where the `kubernetes` and `docker` properties will be ++written. By default, the module will add `$!kubernetes` and ++`$!docker`. ++ ++.. _allowunsignedcerts: ++ ++allowunsignedcerts ++^^^^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "boolean", "off", "no", "none" ++ ++If `"on"`, this will set the curl `CURLOPT_SSL_VERIFYPEER` option to ++`0`. You are strongly discouraged to set this to `"on"`. It is ++primarily useful only for debugging or testing. ++ ++.. _de_dot: ++ ++de_dot ++^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "boolean", "on", "no", "none" ++ ++When processing labels and annotations, if this parameter is set to ++`"on"`, the key strings will have their `.` characters replaced with ++the string specified by the `de_dot_separator` parameter. ++ ++.. _de_dot_separator: ++ ++de_dot_separator ++^^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "_", "no", "none" ++ ++When processing labels and annotations, if the `de_dot` parameter is ++set to `"on"`, the key strings will have their `.` characters replaced ++with the string specified by the string value of this parameter. ++ ++.. _filenamerules: ++ ++filenamerules ++^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "SEE BELOW", "no", "none" ++ ++.. note:: ++ ++ This directive is not supported with liblognorm 2.0.2 and earlier. ++ ++When processing json-file logs, these are the lognorm rules to use to ++match the filename and extract metadata. The default value is:: ++ ++ rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%conta\ ++ iner_name:char-to:-%-%container_id:char-to:.%.log ++ ++.. note:: ++ ++ In the above rules, the slashes ``\`` ending each line indicate ++ line wrapping - they are not part of the rule. ++ ++There are two rules because the `container_hash` is optional. ++ ++.. _filenamerulebase: ++ ++filenamerulebase ++^^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "/etc/rsyslog.d/k8s_filename.rulebase", "no", "none" ++ ++When processing json-file logs, this is the rulebase used to ++match the filename and extract metadata. For the actual rules, see ++below `filenamerules`. ++ ++.. _containerrules: ++ ++containerrules ++^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "SEE BELOW", "no", "none" ++ ++.. note:: ++ ++ This directive is not supported with liblognorm 2.0.2 and earlier. ++ ++For journald logs, there must be a message property `CONTAINER_NAME` ++which has a value matching these rules specified by this parameter. ++The default value is:: ++ ++ rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:\ ++ _%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_u\ ++ sed_2:rest% ++ rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%na\ ++ mespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest% ++ ++.. note:: ++ ++ In the above rules, the slashes ``\`` ending each line indicate ++ line wrapping - they are not part of the rule. ++ ++There are two rules because the `container_hash` is optional. ++ ++.. _containerrulebase: ++ ++containerrulebase ++^^^^^^^^^^^^^^^^^ ++ ++.. csv-table:: ++ :header: "type", "default", "mandatory", "obsolete legacy directive" ++ :widths: auto ++ :class: parameter-table ++ ++ "word", "/etc/rsyslog.d/k8s_container_name.rulebase", "no", "none" ++ ++When processing json-file logs, this is the rulebase used to ++match the CONTAINER_NAME property value and extract metadata. For the ++actual rules, see `containerrules`. ++ ++Fields ++------ ++ ++These are the fields added from the metadata in the json-file filename, or from ++the `CONTAINER_NAME` and `CONTAINER_ID_FULL` fields from the `imjournal` input: ++ ++`$!kubernetes!namespace_name`, `$!kubernetes!pod_name`, ++`$!kubernetes!container_name`, `$!docker!id`, `$!kubernetes!master_url`. ++ ++If mmkubernetes can extract the above fields from the input, the following ++fields will always be present. If they are not present, mmkubernetes ++failed to look up the namespace or pod in Kubernetes: ++ ++`$!kubernetes!namespace_id`, `$!kubernetes!pod_id`, ++`$!kubernetes!creation_timestamp`, `$!kubernetes!host` ++ ++The following fields may be present, depending on how the namespace and pod are ++defined in Kubernetes, and depending on the value of the directive ++`annotation_match`: ++ ++`$!kubernetes!labels`, `$!kubernetes!annotations`, `$!kubernetes!namespace_labels`, ++`$!kubernetes!namespace_annotations` ++ ++More fields may be added in the future. ++ ++Example ++------- ++ ++Assuming you have an `imfile` input reading from docker json-file container ++logs managed by Kubernetes, with `addmetadata="on"` so that mmkubernetes can ++get the basic necessary Kubernetes metadata from the filename: ++ ++.. code-block:: none ++ ++ input(type="imfile" file="/var/log/containers/*.log" ++ tag="kubernetes" addmetadata="on") ++ ++and/or an `imjournal` input for docker journald container logs annotated by ++Kubernetes: ++ ++.. code-block:: none ++ ++ input(type="imjournal") ++ ++Then mmkubernetes can be used to annotate log records like this: ++ ++.. code-block:: none ++ ++ module(load="mmkubernetes") ++ ++ action(type="mmkubernetes") ++ ++After this, you should have log records with fields described in the `Fields` ++section above. ++ ++Credits ++------- ++ ++This work is based on ++https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter ++and has many of the same features. diff --git a/SOURCES/rsyslog-8.24.0-rhbz1459896-queues-defaults-doc.patch b/SOURCES/rsyslog-8.24.0-rhbz1459896-queues-defaults-doc.patch deleted file mode 100644 index e7bdc91..0000000 --- a/SOURCES/rsyslog-8.24.0-rhbz1459896-queues-defaults-doc.patch +++ /dev/null @@ -1,138 +0,0 @@ -From c8be9a713a57f07311560af50c24267b30bef21b Mon Sep 17 00:00:00 2001 -From: Jiri Vymazal -Date: Tue, 29 Aug 2017 16:32:15 +0200 -Subject: [PATCH] fixed queue default values - ---- - source/concepts/queues.rst | 7 +++---- - source/configuration/global/index.rst | 6 +++--- - source/configuration/global/options/rsconf1_mainmsgqueuesize.rst | 2 +- - source/rainerscript/queue_parameters.rst | 15 ++++++++++++--- - source/configuration/action/index.rst | 12 ++++++------ - 5 files changed, 24 insertions(+), 16 deletions(-) - -diff --git a/source/concepts/queues.rst b/source/concepts/queues.rst -index c71413c..9b41128 100644 ---- a/source/concepts/queues.rst -+++ b/source/concepts/queues.rst -@@ -273,10 +273,9 @@ actually needed. - The water marks can be set via the "*$QueueHighWatermark*\ " - and  "*$QueueLowWatermark*\ " configuration file directives. - Note that these are actual numbers, not percentages. Be sure they make --sense (also in respect to "*$QueueSize*\ "), as rsyslodg does --currently not perform any checks on the numbers provided. It is easy to --screw up the system here (yes, a feature enhancement request is filed --;)). -+sense (also in respect to "*$QueueSize*\ "). Rsyslodg does -+perform some checks on the numbers provided, and issues warning when -+numbers are "suspicious". - - Limiting the Queue Size - ----------------------- -diff --git a/source/configuration/global/index.rst b/source/configuration/global/index.rst -index 2738f21..a53ef23 100644 ---- a/source/configuration/global/index.rst -+++ b/source/configuration/global/index.rst -@@ -137,13 +137,13 @@ To understand queue parameters, read - - **$MainMsgQueueDequeueSlowdown** [number is timeout in - *micro*\ seconds (1000000us is 1sec!), default 0 (no delay). Simple - rate-limiting!] --- **$MainMsgQueueDiscardMark** [default 9750] -+- **$MainMsgQueueDiscardMark** [default 98000] - - **$MainMsgQueueDiscardSeverity** [either a textual or - numerical severity! default 4 (warning)] - - **$MainMsgQueueFileName** --- **$MainMsgQueueHighWaterMark** [default 8000] -+- **$MainMsgQueueHighWaterMark** [default 80000] - - **$MainMsgQueueImmediateShutdown** [on/**off**] --- **$MainMsgQueueLowWaterMark** [default 2000] -+- **$MainMsgQueueLowWaterMark** [default 20000] - - **$MainMsgQueueMaxFileSize** , default 1m - - **$MainMsgQueueTimeoutActionCompletion** [number is timeout in - ms (1000ms is 1sec!), default 1000, 0 means immediate!] -diff --git a/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst b/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst -index 050407c..3e902cf 100644 ---- a/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst -+++ b/source/configuration/global/options/rsconf1_mainmsgqueuesize.rst -@@ -3,7 +3,7 @@ $MainMsgQueueSize - - **Type:** global configuration directive - --**Default:** 10000 -+**Default:** 100000 - - **Description:** - -diff --git a/source/rainerscript/queue_parameters.rst b/source/rainerscript/queue_parameters.rst -index 4453721..3f2b7a2 100644 ---- a/source/rainerscript/queue_parameters.rst -+++ b/source/rainerscript/queue_parameters.rst -@@ -33,8 +33,14 @@ read the :doc:`queues <../concepts/queues>` documentation. - For more information on the current status of this restriction see - the `rsyslog FAQ: "lower bound for queue - sizes" `_. -+ -+ The default depends on queue type and, if you need -+ a specific value, please specify it. Otherwise rsyslog selects what -+ it consideres appropriate. For example, ruleset queues have a default -+ size of 50000 and action queues which are configured to be non-direct -+ have a size of 1000. - - **queue.dequeuebatchsize** number -- default 16 -+ default 128 - - **queue.maxdiskspace** number - The maximum size that all queue files together will use on disk. Note - that the actual size may be slightly larger than the configured max, -@@ -46,8 +47,9 @@ read the :doc:`queues <../concepts/queues>` documentation. - processing, because disk queue mode is very considerably slower than - in-memory queue mode. Going to disk should be reserved for cases - where an output action destination is offline for some period. -+ default 90% of queue size - - **queue.lowwatermark** number -- default 2000 -+ default 70% of queue size - - **queue.fulldelaymark** number - Number of messages when the queue should block delayable messages. - Messages are NO LONGER PROCESSED until the queue has sufficient space -@@ -59,9 +61,11 @@ read the :doc:`queues <../concepts/queues>` documentation. - out of space. Please note that if you use a DA queue, setting the - fulldelaymark BELOW the highwatermark makes the queue never activate - disk mode for delayable inputs. So this is probably not what you want. -+ default 97% of queue size - - **queue.lightdelaymark** number -+ default 70% of queue size - - **queue.discardmark** number -- default 9750 -+ default 80% of queue size - - **queue.discardseverity** number - \*numerical\* severity! default 8 (nothing discarded) - - **queue.checkpointinterval** number -diff --git a/source/configuration/action/index.rst b/source/configuration/action/index.rst -index 3e7cd24..9352866 100644 ---- a/source/configuration/action/index.rst -+++ b/source/configuration/action/index.rst -@@ -163,18 +163,18 @@ following action, only. The next and all other actions will be - in "direct" mode (no real queue) if not explicitely specified otherwise. - - - **$ActionQueueCheckpointInterval** --- **$ActionQueueDequeueBatchSize** [default 16] -+- **$ActionQueueDequeueBatchSize** [default 128] - - **$ActionQueueDequeueSlowdown** [number is timeout in - *micro*\ seconds (1000000us is 1sec!), default 0 (no delay). Simple - rate-limiting!] --- **$ActionQueueDiscardMark** [default 9750] --- **$ActionQueueDiscardSeverity** [\*numerical\* severity! default -- 4 (warning)] -+- **$ActionQueueDiscardMark** [default 80% of queue size] -+- **$ActionQueueDiscardSeverity** [\*numerical\* severity! default -+ 8 (nothing discarded)] - - **$ActionQueueFileName** --- **$ActionQueueHighWaterMark** [default 8000] -+- **$ActionQueueHighWaterMark** [default 90% of queue size] - - **$ActionQueueImmediateShutdown** [on/**off**] - - **$ActionQueueSize** --- **$ActionQueueLowWaterMark** [default 2000] -+- **$ActionQueueLowWaterMark** [default 70% of queue size] - - **$ActionQueueMaxFileSize** , default 1m - - **$ActionQueueTimeoutActionCompletion** [number is timeout in ms - (1000ms is 1sec!), default 1000, 0 means immediate!] diff --git a/SOURCES/rsyslog-8.24.0-rhbz1507028-recover_qi-doc.patch b/SOURCES/rsyslog-8.24.0-rhbz1507028-recover_qi-doc.patch deleted file mode 100644 index 42a69b1..0000000 --- a/SOURCES/rsyslog-8.24.0-rhbz1507028-recover_qi-doc.patch +++ /dev/null @@ -1,27 +0,0 @@ -From ff07a7cfc171dc2151cc8afe44776525d34a9e01 Mon Sep 17 00:00:00 2001 -From: jvymazal -Date: Tue, 3 Jan 2017 10:24:26 +0100 -Subject: [PATCH] Update queues.rst - -Update queues.rst ---- - source/concepts/queues.rst | 6 ++++++ - 1 file changed, 6 insertions(+) - -diff --git a/source/concepts/queues.rst b/source/concepts/queues.rst -index eb394e8..c71413c 100644 ---- a/source/concepts/queues.rst -+++ b/source/concepts/queues.rst -@@ -153,6 +153,12 @@ can be requested via "*QueueSyncQueueFiles on/off* with the - default being off. Activating this option has a performance penalty, so - it should not be turned on without reason. - -+If you happen to lose or otherwise need the housekeeping structures and -+have all yours queue chunks you can use perl script included in rsyslog -+package to generate it. -+Usage: recover_qi.pl -w *$WorkDirectory* -f QueueFileName -d 8 > QueueFileName.qi -+ -+ - In-Memory Queues - ~~~~~~~~~~~~~~~~ - diff --git a/SOURCES/rsyslog-8.24.0-rhbz1507145-omelastic-client-cert.patch b/SOURCES/rsyslog-8.24.0-rhbz1507145-omelastic-client-cert.patch new file mode 100644 index 0000000..84e9d0f --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1507145-omelastic-client-cert.patch @@ -0,0 +1,208 @@ +From 02772eb5f28b3c3a98f0d739b6210ca82d58f7ee Mon Sep 17 00:00:00 2001 +From: Rich Megginson +Date: Thu, 8 Feb 2018 18:13:13 -0700 +Subject: [PATCH] omelasticsearch - add support for CA cert, client cert auth + +This allows omelasticsearch to perform client cert based authentication +to Elasticsearch. +Add the following parameters: +`tls.cacert` - Full path and filename of the file containing the CA cert + for the CA that issued the Elasticsearch server(s) cert(s) +`tls.mycert` - Full path and filename of the file containing the client + cert used to authenticate to Elasticsearch +`tls.myprivkey` - Full path and filename of the file containing the client + key used to authenticate to Elasticsearch +--- + plugins/omelasticsearch/omelasticsearch.c | 79 ++++++++++++++++++++++++++++--- + 1 file changed, 73 insertions(+), 6 deletions(-) + +diff --git a/plugins/omelasticsearch/omelasticsearch.c b/plugins/omelasticsearch/omelasticsearch.c +index 97d8fb233..88bd5e16c 100644 +--- a/plugins/omelasticsearch/omelasticsearch.c ++++ b/plugins/omelasticsearch/omelasticsearch.c +@@ -110,6 +110,9 @@ typedef struct _instanceData { + size_t maxbytes; + sbool useHttps; + sbool allowUnsignedCerts; ++ uchar *caCertFile; ++ uchar *myCertFile; ++ uchar *myPrivKeyFile; + } instanceData; + + typedef struct wrkrInstanceData { +@@ -154,7 +157,10 @@ static struct cnfparamdescr actpdescr[] = { + { "template", eCmdHdlrGetWord, 0 }, + { "dynbulkid", eCmdHdlrBinary, 0 }, + { "bulkid", eCmdHdlrGetWord, 0 }, +- { "allowunsignedcerts", eCmdHdlrBinary, 0 } ++ { "allowunsignedcerts", eCmdHdlrBinary, 0 }, ++ { "tls.cacert", eCmdHdlrString, 0 }, ++ { "tls.mycert", eCmdHdlrString, 0 }, ++ { "tls.myprivkey", eCmdHdlrString, 0 } + }; + static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, +@@ -168,6 +174,9 @@ BEGINcreateInstance + CODESTARTcreateInstance + pData->fdErrFile = -1; + pthread_mutex_init(&pData->mutErrFile, NULL); ++ pData->caCertFile = NULL; ++ pData->myCertFile = NULL; ++ pData->myPrivKeyFile = NULL; + ENDcreateInstance + + BEGINcreateWrkrInstance +@@ -216,6 +225,9 @@ CODESTARTfreeInstance + free(pData->timeout); + free(pData->errorFile); + free(pData->bulkId); ++ free(pData->caCertFile); ++ free(pData->myCertFile); ++ free(pData->myPrivKeyFile); + ENDfreeInstance + + BEGINfreeWrkrInstance +@@ -270,6 +282,9 @@ CODESTARTdbgPrintInstInfo + dbgprintf("\tinterleaved=%d\n", pData->interleaved); + dbgprintf("\tdynbulkid=%d\n", pData->dynBulkId); + dbgprintf("\tbulkid='%s'\n", pData->bulkId); ++ dbgprintf("\ttls.cacert='%s'\n", pData->caCertFile); ++ dbgprintf("\ttls.mycert='%s'\n", pData->myCertFile); ++ dbgprintf("\ttls.myprivkey='%s'\n", pData->myPrivKeyFile); + ENDdbgPrintInstInfo + + +@@ -311,7 +326,7 @@ computeBaseUrl(const char*const serverParam, + r = useHttps ? es_addBuf(&urlBuf, SCHEME_HTTPS, sizeof(SCHEME_HTTPS)-1) : + es_addBuf(&urlBuf, SCHEME_HTTP, sizeof(SCHEME_HTTP)-1); + +- if (r == 0) r = es_addBuf(&urlBuf, serverParam, strlen(serverParam)); ++ if (r == 0) r = es_addBuf(&urlBuf, (char *)serverParam, strlen(serverParam)); + if (r == 0 && !strchr(host, ':')) { + snprintf(portBuf, sizeof(portBuf), ":%d", defaultPort); + r = es_addBuf(&urlBuf, portBuf, strlen(portBuf)); +@@ -1296,7 +1311,7 @@ finalize_it: + } + + static void +-curlCheckConnSetup(CURL *handle, HEADER *header, long timeout, sbool allowUnsignedCerts) ++curlCheckConnSetup(CURL *handle, HEADER *header, long timeout, sbool allowUnsignedCerts, wrkrInstanceData_t *pWrkrData) + { + curl_easy_setopt(handle, CURLOPT_HTTPHEADER, header); + curl_easy_setopt(handle, CURLOPT_NOBODY, TRUE); +@@ -1305,13 +1320,21 @@ curlCheckConnSetup(CURL *handle, HEADER *header, long timeout, sbool allowUnsign + + if(allowUnsignedCerts) + curl_easy_setopt(handle, CURLOPT_SSL_VERIFYPEER, FALSE); ++ if(pWrkrData->pData->caCertFile) ++ curl_easy_setopt(handle, CURLOPT_CAINFO, pWrkrData->pData->caCertFile); ++ if(pWrkrData->pData->myCertFile) ++ curl_easy_setopt(handle, CURLOPT_SSLCERT, pWrkrData->pData->myCertFile); ++ if(pWrkrData->pData->myPrivKeyFile) ++ curl_easy_setopt(handle, CURLOPT_SSLKEY, pWrkrData->pData->myPrivKeyFile); ++ /* uncomment for in-dept debuggung: ++ curl_easy_setopt(handle, CURLOPT_VERBOSE, TRUE); */ + + /* Only enable for debugging + curl_easy_setopt(curl, CURLOPT_VERBOSE, TRUE); */ + } + + static void +-curlPostSetup(CURL *handle, HEADER *header, uchar* authBuf) ++curlPostSetup(CURL *handle, HEADER *header, uchar* authBuf, wrkrInstanceData_t *pWrkrData) + { + curl_easy_setopt(handle, CURLOPT_HTTPHEADER, header); + curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, curlResult); +@@ -1322,6 +1345,12 @@ curlPostSetup(CURL *handle, HEADER *header, uchar* authBuf) + curl_easy_setopt(handle, CURLOPT_USERPWD, authBuf); + curl_easy_setopt(handle, CURLOPT_PROXYAUTH, CURLAUTH_ANY); + } ++ if(pWrkrData->pData->caCertFile) ++ curl_easy_setopt(handle, CURLOPT_CAINFO, pWrkrData->pData->caCertFile); ++ if(pWrkrData->pData->myCertFile) ++ curl_easy_setopt(handle, CURLOPT_SSLCERT, pWrkrData->pData->myCertFile); ++ if(pWrkrData->pData->myPrivKeyFile) ++ curl_easy_setopt(handle, CURLOPT_SSLKEY, pWrkrData->pData->myPrivKeyFile); + } + + static rsRetVal +@@ -1332,7 +1361,7 @@ curlSetup(wrkrInstanceData_t *pWrkrData, instanceData *pData) + if (pWrkrData->curlPostHandle == NULL) { + return RS_RET_OBJ_CREATION_FAILED; + } +- curlPostSetup(pWrkrData->curlPostHandle, pWrkrData->curlHeader, pData->authBuf); ++ curlPostSetup(pWrkrData->curlPostHandle, pWrkrData->curlHeader, pData->authBuf, pWrkrData); + + pWrkrData->curlCheckConnHandle = curl_easy_init(); + if (pWrkrData->curlCheckConnHandle == NULL) { +@@ -1341,7 +1370,7 @@ curlSetup(wrkrInstanceData_t *pWrkrData, instanceData *pData) + return RS_RET_OBJ_CREATION_FAILED; + } + curlCheckConnSetup(pWrkrData->curlCheckConnHandle, pWrkrData->curlHeader, +- pData->healthCheckTimeout, pData->allowUnsignedCerts); ++ pData->healthCheckTimeout, pData->allowUnsignedCerts, pWrkrData); + + return RS_RET_OK; + } +@@ -1372,6 +1401,9 @@ setInstParamDefaults(instanceData *pData) + pData->interleaved=0; + pData->dynBulkId= 0; + pData->bulkId = NULL; ++ pData->caCertFile = NULL; ++ pData->myCertFile = NULL; ++ pData->myPrivKeyFile = NULL; + } + + BEGINnewActInst +@@ -1380,6 +1412,8 @@ BEGINnewActInst + struct cnfarray* servers = NULL; + int i; + int iNumTpls; ++ FILE *fp; ++ char errStr[1024]; + CODESTARTnewActInst + if((pvals = nvlstGetParams(lst, &actpblk, NULL)) == NULL) { + ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); +@@ -1435,6 +1469,39 @@ CODESTARTnewActInst + pData->dynBulkId = pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "bulkid")) { + pData->bulkId = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); ++ } else if(!strcmp(actpblk.descr[i].name, "tls.cacert")) { ++ pData->caCertFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)pData->caCertFile, "r"); ++ if(fp == NULL) { ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ errmsg.LogError(0, RS_RET_NO_FILE_ACCESS, ++ "error: 'tls.cacert' file %s couldn't be accessed: %s\n", ++ pData->caCertFile, errStr); ++ } else { ++ fclose(fp); ++ } ++ } else if(!strcmp(actpblk.descr[i].name, "tls.mycert")) { ++ pData->myCertFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)pData->myCertFile, "r"); ++ if(fp == NULL) { ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ errmsg.LogError(0, RS_RET_NO_FILE_ACCESS, ++ "error: 'tls.mycert' file %s couldn't be accessed: %s\n", ++ pData->myCertFile, errStr); ++ } else { ++ fclose(fp); ++ } ++ } else if(!strcmp(actpblk.descr[i].name, "tls.myprivkey")) { ++ pData->myPrivKeyFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)pData->myPrivKeyFile, "r"); ++ if(fp == NULL) { ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ errmsg.LogError(0, RS_RET_NO_FILE_ACCESS, ++ "error: 'tls.myprivkey' file %s couldn't be accessed: %s\n", ++ pData->myPrivKeyFile, errStr); ++ } else { ++ fclose(fp); ++ } + } else { + dbgprintf("omelasticsearch: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); +-- +2.14.3 + diff --git a/SOURCES/rsyslog-8.24.0-rhbz1511485-deserialize-property-name.patch b/SOURCES/rsyslog-8.24.0-rhbz1511485-deserialize-property-name.patch new file mode 100644 index 0000000..2bf5f9e --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1511485-deserialize-property-name.patch @@ -0,0 +1,59 @@ +From c49e42f4f8381fc8e92579c41cefb2c85fe45929 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Tue, 7 Feb 2017 13:09:40 +0100 +Subject: [PATCH] core: fix sequence error in msg object deserializer + +Corruption of disk queue (or disk part of DA queue) always happens if +the "json" property (message variables) is present and "structured-data" +property is also present. This causes rsyslog to serialize to the +queue in wrong property sequence, which will lead to error -2308 on +deserialization. + +Seems to be a long-standing bug. Depending on version used, some or +all messages in disk queue may be lost. + +closes https://github.com/rsyslog/rsyslog/issues/1404 +--- + runtime/msg.c | 14 ++++++++------ + 1 file changed, 8 insertions(+), 6 deletions(-) + +diff --git a/runtime/msg.c b/runtime/msg.c +index 7cfeca843..cfa95517e 100644 +--- a/runtime/msg.c ++++ b/runtime/msg.c +@@ -1350,6 +1350,11 @@ MsgDeserialize(smsg_t * const pMsg, strm_t *pStrm) + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } ++ if(isProp("pszStrucData")) { ++ MsgSetStructuredData(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); ++ reinitVar(pVar); ++ CHKiRet(objDeserializeProperty(pVar, pStrm)); ++ } + if(isProp("json")) { + tokener = json_tokener_new(); + pMsg->json = json_tokener_parse_ex(tokener, (char*)rsCStrGetSzStrNoNULL(pVar->val.pStr), +@@ -1366,11 +1371,6 @@ MsgDeserialize(smsg_t * const pMsg, strm_t *pStrm) + reinitVar(pVar); + CHKiRet(objDeserializeProperty(pVar, pStrm)); + } +- if(isProp("pszStrucData")) { +- MsgSetStructuredData(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); +- reinitVar(pVar); +- CHKiRet(objDeserializeProperty(pVar, pStrm)); +- } + if(isProp("pCSAPPNAME")) { + MsgSetAPPNAME(pMsg, (char*) rsCStrGetSzStrNoNULL(pVar->val.pStr)); + reinitVar(pVar); +@@ -1401,8 +1401,10 @@ MsgDeserialize(smsg_t * const pMsg, strm_t *pStrm) + * but on the other hand it works decently AND we will probably replace + * the whole persisted format soon in any case. -- rgerhards, 2012-11-06 + */ +- if(!isProp("offMSG")) ++ if(!isProp("offMSG")) { ++ DBGPRINTF("error property: %s\n", rsCStrGetSzStrNoNULL(pVar->pcsName)); + ABORT_FINALIZE(RS_RET_DS_PROP_SEQ_ERR); ++ } + MsgSetMSGoffs(pMsg, pVar->val.num); + finalize_it: + if(pVar != NULL) diff --git a/SOURCES/rsyslog-8.24.0-rhbz1512551-caching-sockaddr.patch b/SOURCES/rsyslog-8.24.0-rhbz1512551-caching-sockaddr.patch new file mode 100644 index 0000000..cf36062 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1512551-caching-sockaddr.patch @@ -0,0 +1,70 @@ +From 5f828658a317c86095fc2b982801b58bf8b8ee6f Mon Sep 17 00:00:00 2001 +From: mosvald +Date: Mon, 4 Dec 2017 08:10:37 +0100 +Subject: [PATCH] cache sin_addr instead of the whole sockaddr structure + +--- + runtime/dnscache.c | 41 +++++++++++++++++++++++++++++++---------- + 1 file changed, 31 insertions(+), 10 deletions(-) + +diff --git a/runtime/dnscache.c b/runtime/dnscache.c +index 388a64f5e..d1d6e10a1 100644 +--- a/runtime/dnscache.c ++++ b/runtime/dnscache.c +@@ -79,22 +79,43 @@ static prop_t *staticErrValue; + static unsigned int + hash_from_key_fn(void *k) + { +- int len; +- uchar *rkey = (uchar*) k; /* we treat this as opaque bytes */ +- unsigned hashval = 1; +- +- len = SALEN((struct sockaddr*)k); +- while(len--) +- hashval = hashval * 33 + *rkey++; ++ int len = 0; ++ uchar *rkey; /* we treat this as opaque bytes */ ++ unsigned hashval = 1; ++ ++ switch (((struct sockaddr *)k)->sa_family) { ++ case AF_INET: ++ len = sizeof (struct in_addr); ++ rkey = (uchar*) &(((struct sockaddr_in *)k)->sin_addr); ++ break; ++ case AF_INET6: ++ len = sizeof (struct in6_addr); ++ rkey = (uchar*) &(((struct sockaddr_in6 *)k)->sin6_addr); ++ break; ++ } ++ while(len--) ++ hashval = hashval * 33 + *rkey++; + +- return hashval; ++ return hashval; + } + + static int + key_equals_fn(void *key1, void *key2) + { +- return (SALEN((struct sockaddr*)key1) == SALEN((struct sockaddr*) key2) +- && !memcmp(key1, key2, SALEN((struct sockaddr*) key1))); ++ int RetVal = 0; ++ ++ if (((struct sockaddr *)key1)->sa_family != ((struct sockaddr *)key2)->sa_family) ++ return 0; ++ switch (((struct sockaddr *)key1)->sa_family) { ++ case AF_INET: ++ RetVal = !memcmp(&((struct sockaddr_in *)key1)->sin_addr, &((struct sockaddr_in *)key2)->sin_addr, sizeof (struct in_addr)); ++ break; ++ case AF_INET6: ++ RetVal = !memcmp(&((struct sockaddr_in6 *)key1)->sin6_addr, &((struct sockaddr_in6 *)key2)->sin6_addr, sizeof (struct in6_addr)); ++ break; ++ } ++ ++ return RetVal; + } + + /* destruct a cache entry. +-- +2.14.3 + diff --git a/SOURCES/rsyslog-8.24.0-rhbz1531295-imfile-rewrite-with-symlink.patch b/SOURCES/rsyslog-8.24.0-rhbz1531295-imfile-rewrite-with-symlink.patch new file mode 100644 index 0000000..d0f5721 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1531295-imfile-rewrite-with-symlink.patch @@ -0,0 +1,3160 @@ +From: Jiri Vymazal +Date: Mon, 28 Jun 2018 15:07:55 +0100 +Subject: Imfile rewrite with symlink support + +This commit greatly refactors imfile internal workings. It changes the +handling of inotify, FEN, and polling modes. Mostly unchanged is the +processing of the way a file is read and state files are kept. + +This is about a 50% rewrite of the module. + +Polling, inotify, and FEN modes now use greatly unified code. Some +differences still exists and may be changed with further commits. The +internal handling of wildcards and file detection has been completely +re-written from scratch. For example, previously when multi-level +wildcards were used these were not reliably detected. The code also +now provides much of the same functionality in all modes, most importantly +wildcards are now also supported in polling mode. + +The refactoring sets ground for further enhancements and smaller +refactorings. This commit provides the same feature set that imfile +had previously. + +Some specific changes: +bugfix: imfile did not pick up all files when not present +at startup + +bugfix: directories only support "*" wildcard, no others + +bugfix: parameter "sortfiles" did only work in FEN mode + +provides the ability to dynamically add and remove files via +multi-level wildcards + +the state file name currently has been changed to inode number + +We change it to json and also change the way it is stored and loaded. +This sets base to additional improvements in imfile. + +When imfile rewrites state files, it does not truncate previous +content. If the new content is smaller than the existing one, the +existing part will not be overwritten, resulting in invalid json. +That in turn can lead to some other failures. + +This introduces symlink detection and following as well +as monitoring changes on them. + +stream/bugfix: memory leak on stream open if filename as already generated - +this can happen if imfile reads a state file. On each open, memory for the +file name can be lost. + +(cherry picked from commit a03dccf8484d621fe06cb2d11816fbe921751e54 - https://gitlab.cee.redhat.com/rsyslog/rsyslog) +--- + plugins/imfile/imfile.c | 2264 ++++++++++++++++++++++--------------------- + runtime/msg.c | 22 ++++++++++++++++++++++ + runtime/msg.h | 1 + + runtime/stream.c | 136 ++++++++++++++++++++------- + runtime/stream.h | 17 ++++++++++++++--- + 5 files changed, 1303 insertions(+), 1137 deletitions(-) + +diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c +index b0bc860bcd16beaecd67ce1b7c61991356ea5471..f8225d7068d8fc98edde7bbed194be1105b1696b 100644 +--- a/plugins/imfile/imfile.c ++++ b/plugins/imfile/imfile.c +@@ -35,6 +35,7 @@ + #include + #include + #include ++#include + #include + #ifdef HAVE_SYS_INOTIFY_H + #include +@@ -56,6 +57,7 @@ + #include "stringbuf.h" + #include "ruleset.h" + #include "ratelimit.h" ++#include "parserif.h" + + #include // TODO: fix via own module + +@@ -77,50 +81,19 @@ static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config para + + #define NUM_MULTISUB 1024 /* default max number of submits */ + #define DFLT_PollInterval 10 +- +-#define INIT_FILE_TAB_SIZE 4 /* default file table size - is extended as needed, use 2^x value */ +-#define INIT_FILE_IN_DIR_TAB_SIZE 1 /* initial size for "associated files tab" in directory table */ + #define INIT_WDMAP_TAB_SIZE 1 /* default wdMap table size - is extended as needed, use 2^x value */ +- + #define ADD_METADATA_UNSPECIFIED -1 ++#define CONST_LEN_CEE_COOKIE 5 ++#define CONST_CEE_COOKIE "@cee:" ++ ++/* If set to 1, fileTableDisplay will be compiled and used for debugging */ ++#define ULTRA_DEBUG 0 ++ ++/* Setting GLOB_BRACE to ZERO which disables support for GLOB_BRACE if not available on current platform */ ++#ifndef GLOB_BRACE ++ #define GLOB_BRACE 0 ++#endif + +-/* this structure is used in pure polling mode as well one of the support +- * structures for inotify. +- */ +-typedef struct lstn_s { +- struct lstn_s *next, *prev; +- struct lstn_s *masterLstn;/* if dynamic file (via wildcard), this points to the configured +- * master entry. For master entries, it is always NULL. Only +- * dynamic files can be deleted from the "files" list. */ +- uchar *pszFileName; +- uchar *pszDirName; +- uchar *pszBaseName; +- uchar *pszTag; +- size_t lenTag; +- uchar *pszStateFile; /* file in which state between runs is to be stored (dynamic if NULL) */ +- int readTimeout; +- int iFacility; +- int iSeverity; +- int maxLinesAtOnce; +- uint32_t trimLineOverBytes; +- int nRecords; /**< How many records did we process before persisting the stream? */ +- int iPersistStateInterval; /**< how often should state be persisted? (0=on close only) */ +- strm_t *pStrm; /* its stream (NULL if not assigned) */ +- sbool bRMStateOnDel; +- sbool hasWildcard; +- uint8_t readMode; /* which mode to use in ReadMulteLine call? */ +- uchar *startRegex; /* regex that signifies end of message (NULL if unset) */ +- regex_t end_preg; /* compiled version of startRegex */ +- uchar *prevLineSegment; /* previous line segment (in regex mode) */ +- sbool escapeLF; /* escape LF inside the MSG content? */ +- sbool reopenOnTruncate; +- sbool addMetadata; +- sbool addCeeTag; +- sbool freshStartTail; /* read from tail of file on fresh start? */ +- ruleset_t *pRuleset; /* ruleset to bind listener to (use system default if unspecified) */ +- ratelimit_t *ratelimiter; +- multi_submit_t multiSub; +-} lstn_t; + + static struct configSettings_s { + uchar *pszFileName; +@@ -138,9 +111,11 @@ static struct configSettings_s { + + struct instanceConf_s { + uchar *pszFileName; ++ uchar *pszFileName_forOldStateFile; /* we unfortunately needs this to read old state files */ + uchar *pszDirName; + uchar *pszFileBaseName; + uchar *pszTag; ++ size_t lenTag; + uchar *pszStateFile; + uchar *pszBindRuleset; + int nMultiSub; +@@ -151,11 +126,15 @@ struct instanceConf_s { + sbool bRMStateOnDel; + uint8_t readMode; + uchar *startRegex; ++ regex_t end_preg; /* compiled version of startRegex */ ++ sbool discardTruncatedMsg; ++ sbool msgDiscardingError; + sbool escapeLF; + sbool reopenOnTruncate; + sbool addCeeTag; + sbool addMetadata; + sbool freshStartTail; ++ sbool fileNotFoundError; + int maxLinesAtOnce; + uint32_t trimLineOverBytes; + ruleset_t *pBindRuleset; /* ruleset to bind listener to (use system default if unspecified) */ +@@ -163,9 +142,54 @@ struct instanceConf_s { + }; + + ++/* file system objects */ ++typedef struct fs_edge_s fs_edge_t; ++typedef struct fs_node_s fs_node_t; ++typedef struct act_obj_s act_obj_t; ++struct act_obj_s { ++ act_obj_t *prev; ++ act_obj_t *next; ++ fs_edge_t *edge; /* edge which this object belongs to */ ++ char *name; /* full path name of active object */ ++ char *basename; /* only basename */ //TODO: remove when refactoring rename support ++ char *source_name; /* if this object is target of a symlink, source_name is its name (else NULL) */ ++ //char *statefile; /* base name of state file (for move operations) */ ++ int wd; ++ time_t timeoutBase; /* what time to calculate the timeout against? */ ++ /* file dynamic data */ ++ int in_move; /* workaround for inotify move: if set, state file must not be deleted */ ++ ino_t ino; /* current inode nbr */ ++ strm_t *pStrm; /* its stream (NULL if not assigned) */ ++ int nRecords; /**< How many records did we process before persisting the stream? */ ++ ratelimit_t *ratelimiter; ++ multi_submit_t multiSub; ++ int is_symlink; ++}; ++struct fs_edge_s { ++ fs_node_t *parent; ++ fs_node_t *node; /* node this edge points to */ ++ fs_edge_t *next; ++ uchar *name; ++ uchar *path; ++ act_obj_t *active; ++ int is_file; ++ int ninst; /* nbr of instances in instarr */ ++ instanceConf_t **instarr; ++}; ++struct fs_node_s { ++ fs_edge_t *edges; ++ fs_node_t *root; ++}; ++ ++ + /* forward definitions */ +-static rsRetVal persistStrmState(lstn_t *pInfo); ++static rsRetVal persistStrmState(act_obj_t *); + static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal); ++static rsRetVal pollFile(act_obj_t *act); ++static int getBasename(uchar *const __restrict__ basen, uchar *const __restrict__ path); ++static void act_obj_unlink(act_obj_t *act); ++static uchar * getStateFileName(const act_obj_t *, uchar *, const size_t); ++static int getFullStateFileName(const uchar *const, uchar *const pszout, const size_t ilenout); + + + #define OPMODE_POLLING 0 +@@ -178,57 +200,23 @@ struct modConfData_s { + int readTimeout; + int timeoutGranularity; /* value in ms */ + instanceConf_t *root, *tail; +- lstn_t *pRootLstn; +- lstn_t *pTailLstn; ++ fs_node_t *conf_tree; + uint8_t opMode; + sbool configSetViaV2Method; ++ sbool sortFiles; ++ sbool normalizePath; /* normalize file system pathes (all start with root dir) */ + sbool haveReadTimeouts; /* use special processing if read timeouts exist */ ++ sbool bHadFileData; /* actually a global variable: ++ 1 - last call to pollFile() had data ++ 0 - last call to pollFile() had NO data ++ Must be manually reset to 0 if desired. Helper for ++ polling mode. ++ */ + }; + static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ + static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ + + #ifdef HAVE_INOTIFY_INIT +-/* support for inotify mode */ +- +-/* we need to track directories */ +-struct dirInfoFiles_s { /* associated files */ +- lstn_t *pLstn; +- int refcnt; /* due to inotify's async nature, we may have multiple +- * references to a single file inside our cache - e.g. when +- * inodes are removed, and the file name is re-created BUT another +- * process (like rsyslogd ;)) holds open the old inode. +- */ +-}; +-typedef struct dirInfoFiles_s dirInfoFiles_t; +- +-/* This structure is a dynamic table to track file entries */ +-struct fileTable_s { +- dirInfoFiles_t *listeners; +- int currMax; +- int allocMax; +-}; +-typedef struct fileTable_s fileTable_t; +- +-/* The dirs table (defined below) contains one entry for each directory that +- * is to be monitored. For each directory, it contains array which point to +- * the associated *active* files as well as *configured* files. Note that +- * the configured files may currently not exist, but will be processed +- * when they are created. +- */ +-struct dirInfo_s { +- uchar *dirName; +- fileTable_t active; /* associated active files */ +- fileTable_t configured; /* associated configured files */ +-}; +-typedef struct dirInfo_s dirInfo_t; +-static dirInfo_t *dirs = NULL; +-static int allocMaxDirs; +-static int currMaxDirs; +-/* the following two macros are used to select the correct file table */ +-#define ACTIVE_FILE 1 +-#define CONFIGURED_FILE 0 +- +- + /* We need to map watch descriptors to our actual objects. Unfortunately, the + * inotify API does not provide us with any cookie, so a simple O(1) algorithm + * cannot be done (what a shame...). We assume that maintaining the array is much +@@ -238,9 +226,7 @@ static int currMaxDirs; + */ + struct wd_map_s { + int wd; /* ascending sort key */ +- lstn_t *pLstn; /* NULL, if this is a dir entry, otherwise pointer into listener(file) table */ +- int dirIdx; /* index into dirs table, undefined if pLstn == NULL */ +- time_t timeoutBase; /* what time to calculate the timeout against? */ ++ act_obj_t *act; /* point to related active object */ + }; + typedef struct wd_map_s wd_map_t; + static wd_map_t *wdmap = NULL; +@@ -257,6 +243,8 @@ static struct cnfparamdescr modpdescr[] = { + { "pollinginterval", eCmdHdlrPositiveInt, 0 }, + { "readtimeout", eCmdHdlrPositiveInt, 0 }, + { "timeoutgranularity", eCmdHdlrPositiveInt, 0 }, ++ { "sortfiles", eCmdHdlrBinary, 0 }, ++ { "normalizepath", eCmdHdlrBinary, 0 }, + { "mode", eCmdHdlrGetWord, 0 } + }; + static struct cnfparamblk modpblk = +@@ -286,7 +274,8 @@ static struct cnfparamdescr inppdescr[] = { + { "addceetag", eCmdHdlrBinary, 0 }, + { "statefile", eCmdHdlrString, CNFPARAM_DEPRECATED }, + { "readtimeout", eCmdHdlrPositiveInt, 0 }, +- { "freshstarttail", eCmdHdlrBinary, 0} ++ { "freshstarttail", eCmdHdlrBinary, 0}, ++ { "filenotfounderror", eCmdHdlrBinary, 0} + }; + static struct cnfparamblk inppblk = + { CNFPARAMBLK_VERSION, +@@ -297,18 +286,106 @@ static struct cnfparamblk inppblk = + #include "im-helper.h" /* must be included AFTER the type definitions! */ + + +-#ifdef HAVE_INOTIFY_INIT +-/* support for inotify mode */ ++/* Support for "old cruft" state files will potentially become optional in the ++ * future (hopefully). To prepare so, we use conditional compilation with a ++ * fixed-true condition ;-) -- rgerhards, 2018-03-28 ++ * reason: https://github.com/rsyslog/rsyslog/issues/2231#issuecomment-376862280 ++ */ ++#define ENABLE_V1_STATE_FILE_FORMAT_SUPPORT 1 ++#ifdef ENABLE_V1_STATE_FILE_FORMAT_SUPPORT ++static uchar * ++OLD_getStateFileName(const instanceConf_t *const inst, ++ uchar *const __restrict__ buf, ++ const size_t lenbuf) ++{ ++ DBGPRINTF("OLD_getStateFileName trying '%s'\n", inst->pszFileName_forOldStateFile); ++ snprintf((char*)buf, lenbuf - 1, "imfile-state:%s", inst->pszFileName_forOldStateFile); ++ buf[lenbuf-1] = '\0'; /* be on the safe side... */ ++ uchar *p = buf; ++ for( ; *p ; ++p) { ++ if(*p == '/') ++ *p = '-'; ++ } ++ return buf; ++} + +-#if 0 /* enable if you need this for debugging */ ++/* try to open an old-style state file for given file. If the state file does not ++ * exist or cannot be read, an error is returned. ++ */ ++static rsRetVal ++OLD_openFileWithStateFile(act_obj_t *const act) ++{ ++ DEFiRet; ++ strm_t *psSF = NULL; ++ uchar pszSFNam[MAXFNAME]; ++ size_t lenSFNam; ++ struct stat stat_buf; ++ uchar statefile[MAXFNAME]; ++ const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? ++ ++ uchar *const statefn = OLD_getStateFileName(inst, statefile, sizeof(statefile)); ++ DBGPRINTF("OLD_openFileWithStateFile: trying to open state for '%s', state file '%s'\n", ++ act->name, statefn); ++ ++ /* Get full path and file name */ ++ lenSFNam = getFullStateFileName(statefn, pszSFNam, sizeof(pszSFNam)); ++ ++ /* check if the file exists */ ++ if(stat((char*) pszSFNam, &stat_buf) == -1) { ++ if(errno == ENOENT) { ++ DBGPRINTF("OLD_openFileWithStateFile: NO state file (%s) exists for '%s'\n", ++ pszSFNam, act->name); ++ ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); ++ } else { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ DBGPRINTF("OLD_openFileWithStateFile: error trying to access state " ++ "file for '%s':%s\n", act->name, errStr); ++ ABORT_FINALIZE(RS_RET_IO_ERROR); ++ } ++ } ++ ++ /* If we reach this point, we have a state file */ ++ ++ DBGPRINTF("old state file found - instantiating from it\n"); ++ CHKiRet(strm.Construct(&psSF)); ++ CHKiRet(strm.SettOperationsMode(psSF, STREAMMODE_READ)); ++ CHKiRet(strm.SetsType(psSF, STREAMTYPE_FILE_SINGLE)); ++ CHKiRet(strm.SetFName(psSF, pszSFNam, lenSFNam)); ++ CHKiRet(strm.SetFileNotFoundError(psSF, inst->fileNotFoundError)); ++ CHKiRet(strm.ConstructFinalize(psSF)); ++ ++ /* read back in the object */ ++ CHKiRet(obj.Deserialize(&act->pStrm, (uchar*) "strm", psSF, NULL, act)); ++ free(act->pStrm->pszFName); ++ CHKmalloc(act->pStrm->pszFName = ustrdup(act->name)); ++ ++ strm.CheckFileChange(act->pStrm); ++ CHKiRet(strm.SeekCurrOffs(act->pStrm)); ++ ++ /* we now persist the new state file and delete the old one, so we will ++ * never have to deal with the old one. */ ++ persistStrmState(act); ++ unlink((char*)pszSFNam); ++ ++finalize_it: ++ if(psSF != NULL) ++ strm.Destruct(&psSF); ++ RETiRet; ++} ++#endif /* #ifdef ENABLE_V1_STATE_FILE_FORMAT_SUPPORT */ ++ ++ ++#ifdef HAVE_INOTIFY_INIT ++#if ULTRA_DEBUG == 1 + static void +-dbg_wdmapPrint(char *msg) ++dbg_wdmapPrint(const char *msg) + { + int i; + DBGPRINTF("%s\n", msg); + for(i = 0 ; i < nWdmap ; ++i) +- DBGPRINTF("wdmap[%d]: wd: %d, file %d, dir %d\n", i, +- wdmap[i].wd, wdmap[i].fIdx, wdmap[i].dirIdx); ++ DBGPRINTF("wdmap[%d]: wd: %d, act %p, name: %s\n", ++ i, wdmap[i].wd, wdmap[i].act, wdmap[i].act->name); + } + #endif + +@@ -324,48 +401,10 @@ finalize_it: + RETiRet; + } + +-/* looks up a wdmap entry by dirIdx and returns it's index if found +- * or -1 if not found. +- */ +-static int +-wdmapLookupListner(lstn_t* pLstn) +-{ +- int i = 0; +- int wd = -1; +- /* Loop through */ +- for(i = 0 ; i < nWdmap; ++i) { +- if (wdmap[i].pLstn == pLstn) +- wd = wdmap[i].wd; +- } +- +- return wd; +-} +- +-/* compare function for bsearch() */ +-static int +-wdmap_cmp(const void *k, const void *a) +-{ +- int key = *((int*) k); +- wd_map_t *etry = (wd_map_t*) a; +- if(key < etry->wd) +- return -1; +- else if(key > etry->wd) +- return 1; +- else +- return 0; +-} +-/* looks up a wdmap entry and returns it's index if found +- * or -1 if not found. +- */ +-static wd_map_t * +-wdmapLookup(int wd) +-{ +- return bsearch(&wd, wdmap, nWdmap, sizeof(wd_map_t), wdmap_cmp); +-} + + /* note: we search backwards, as inotify tends to return increasing wd's */ + static rsRetVal +-wdmapAdd(int wd, const int dirIdx, lstn_t *const pLstn) ++wdmapAdd(int wd, act_obj_t *const act) + { + wd_map_t *newmap; + int newmapsize; +@@ -375,7 +414,7 @@ wdmapAdd(int wd, const int dirIdx, lstn_t *const pLstn) + for(i = nWdmap-1 ; i >= 0 && wdmap[i].wd > wd ; --i) + ; /* just scan */ + if(i >= 0 && wdmap[i].wd == wd) { +- DBGPRINTF("imfile: wd %d already in wdmap!\n", wd); ++ LogError(0, RS_RET_INTERNAL_ERROR, "imfile: wd %d already in wdmap!", wd); + ABORT_FINALIZE(RS_RET_FILE_ALREADY_IN_TABLE); + } + ++i; +@@ -392,17 +431,59 @@ wdmapAdd(int wd, const int dirIdx, lstn_t *const pLstn) + memmove(wdmap + i + 1, wdmap + i, sizeof(wd_map_t) * (nWdmap - i)); + } + wdmap[i].wd = wd; +- wdmap[i].dirIdx = dirIdx; +- wdmap[i].pLstn = pLstn; ++ wdmap[i].act = act; + ++nWdmap; +- DBGPRINTF("imfile: enter into wdmap[%d]: wd %d, dir %d, lstn %s:%s\n",i,wd,dirIdx, +- (pLstn == NULL) ? "DIRECTORY" : "FILE", +- (pLstn == NULL) ? dirs[dirIdx].dirName : pLstn->pszFileName); ++ DBGPRINTF("add wdmap[%d]: wd %d, act obj %p, path %s\n", i, wd, act, act->name); + + finalize_it: + RETiRet; + } + ++/* return wd or -1 on error */ ++static int ++in_setupWatch(act_obj_t *const act, const int is_file) ++{ ++ int wd = -1; ++ if(runModConf->opMode != OPMODE_INOTIFY) ++ goto done; ++ ++ wd = inotify_add_watch(ino_fd, act->name, ++ (is_file) ? IN_MODIFY|IN_DONT_FOLLOW : IN_CREATE|IN_DELETE|IN_MOVED_FROM|IN_MOVED_TO); ++ if(wd < 0) { /* There is high probability of selinux denial on top-level paths */ ++ if (errno != EACCES) ++ LogError(errno, RS_RET_IO_ERROR, "imfile: cannot watch object '%s'", act->name); ++ else ++ DBGPRINTF("Access denied when creating watch on '%s'\n", act->name); ++ goto done; ++ } ++ wdmapAdd(wd, act); ++ DBGPRINTF("in_setupWatch: watch %d added for %s(object %p)\n", wd, act->name, act); ++done: return wd; ++} ++ ++/* compare function for bsearch() */ ++static int ++wdmap_cmp(const void *k, const void *a) ++{ ++ int key = *((int*) k); ++ wd_map_t *etry = (wd_map_t*) a; ++ if(key < etry->wd) ++ return -1; ++ else if(key > etry->wd) ++ return 1; ++ else ++ return 0; ++} ++/* looks up a wdmap entry and returns it's index if found ++ * or -1 if not found. ++ */ ++static wd_map_t * ++wdmapLookup(int wd) ++{ ++ return bsearch(&wd, wdmap, nWdmap, sizeof(wd_map_t), wdmap_cmp); ++} ++ ++ + static rsRetVal + wdmapDel(const int wd) + { +@@ -427,46 +506,570 @@ finalize_it: + RETiRet; + } + +-#endif /* #if HAVE_INOTIFY_INIT */ ++#endif // #ifdef HAVE_INOTIFY_INIT ++ ++static void ++fen_setupWatch(act_obj_t *const __attribute__((unused)) act) ++{ ++ DBGPRINTF("fen_setupWatch: DUMMY CALLED - not on Solaris?\n"); ++} ++ ++static void ++fs_node_print(const fs_node_t *const node, const int level) ++{ ++ fs_edge_t *chld; ++ act_obj_t *act; ++ dbgprintf("node print[%2.2d]: %p edges:\n", level, node); ++ ++ for(chld = node->edges ; chld != NULL ; chld = chld->next) { ++ dbgprintf("node print[%2.2d]: child %p '%s' isFile %d, path: '%s'\n", ++ level, chld->node, chld->name, chld->is_file, chld->path); ++ for(int i = 0 ; i < chld->ninst ; ++i) { ++ dbgprintf("\tinst: %p\n", chld->instarr[i]); ++ } ++ for(act = chld->active ; act != NULL ; act = act->next) { ++ dbgprintf("\tact : %p\n", act); ++ dbgprintf("\tact : %p: name '%s', wd: %d\n", ++ act, act->name, act->wd); ++ } ++ } ++ for(chld = node->edges ; chld != NULL ; chld = chld->next) { ++ fs_node_print(chld->node, level+1); ++ } ++} ++ ++/* add a new file system object if it not yet exists, ignore call ++ * if it already does. ++ */ ++static rsRetVal ++act_obj_add(fs_edge_t *const edge, const char *const name, const int is_file, ++ const ino_t ino, const int is_symlink, const char *const source) ++{ ++ act_obj_t *act; ++ char basename[MAXFNAME]; ++ DEFiRet; ++ ++ DBGPRINTF("act_obj_add: edge %p, name '%s' (source '%s')\n", edge, name, source? source : "---"); ++ for(act = edge->active ; act != NULL ; act = act->next) { ++ if(!strcmp(act->name, name)) { ++ if (!source || !act->source_name || !strcmp(act->source_name, source)) { ++ DBGPRINTF("active object '%s' already exists in '%s' - no need to add\n", ++ name, edge->path); ++ FINALIZE; ++ } ++ } ++ } ++ DBGPRINTF("add new active object '%s' in '%s'\n", name, edge->path); ++ CHKmalloc(act = calloc(sizeof(act_obj_t), 1)); ++ CHKmalloc(act->name = strdup(name)); ++ if (-1 == getBasename((uchar*)basename, (uchar*)name)) { ++ CHKmalloc(act->basename = strdup(name)); /* assume basename is same as name */ ++ } else { ++ CHKmalloc(act->basename = strdup(basename)); ++ } ++ act->edge = edge; ++ act->ino = ino; ++ act->is_symlink = is_symlink; ++ if (source) { /* we are target of symlink */ ++ CHKmalloc(act->source_name = strdup(source)); ++ } else { ++ act->source_name = NULL; ++ } ++ #ifdef HAVE_INOTIFY_INIT ++ act->wd = in_setupWatch(act, is_file); ++ #endif ++ fen_setupWatch(act); ++ if(is_file && !is_symlink) { ++ const instanceConf_t *const inst = edge->instarr[0];// TODO: same file, multiple instances? ++ CHKiRet(ratelimitNew(&act->ratelimiter, "imfile", name)); ++ CHKmalloc(act->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(smsg_t *))); ++ act->multiSub.maxElem = inst->nMultiSub; ++ act->multiSub.nElem = 0; ++ pollFile(act); ++ } ++ ++ /* all well, add to active list */ ++ if(edge->active != NULL) { ++ edge->active->prev = act; ++ } ++ act->next = edge->active; ++ edge->active = act; ++//dbgprintf("printout of fs tree after act_obj_add for '%s'\n", name); ++//fs_node_print(runModConf->conf_tree, 0); ++//dbg_wdmapPrint("wdmap after act_obj_add"); ++finalize_it: ++ if(iRet != RS_RET_OK) { ++ if(act != NULL) { ++ free(act->name); ++ free(act); ++ } ++ } ++ RETiRet; ++} ++ ++ ++/* this walks an edges active list and detects and acts on any changes ++ * seen there. It does NOT detect newly appeared files, as they are not ++ * inside the active list! ++ */ ++static void ++detect_updates(fs_edge_t *const edge) ++{ ++ act_obj_t *act; ++ struct stat fileInfo; ++ int restart = 0; ++ ++ for(act = edge->active ; act != NULL ; ) { ++ DBGPRINTF("detect_updates checking active obj '%s'\n", act->name); ++ const int r = lstat(act->name, &fileInfo); ++ if(r == -1) { /* object gone away? */ ++ DBGPRINTF("object gone away, unlinking: '%s'\n", act->name); ++ act_obj_unlink(act); ++ restart = 1; ++ break; ++ } ++ // TODO: add inode check for change notification! ++ ++ /* Note: active nodes may get deleted, so we need to do the ++ * pointer advancement at the end of the for loop! ++ */ ++ act = act->next; ++ } ++ if (restart) ++ detect_updates(edge); ++} ++ ++ ++/* check if active files need to be processed. This is only needed in ++ * polling mode. ++ */ ++static void ++poll_active_files(fs_edge_t *const edge) ++{ ++ if( runModConf->opMode != OPMODE_POLLING ++ || !edge->is_file ++ || glbl.GetGlobalInputTermState() != 0) { ++ return; ++ } ++ ++ act_obj_t *act; ++ for(act = edge->active ; act != NULL ; act = act->next) { ++ fen_setupWatch(act); ++ DBGPRINTF("poll_active_files: polling '%s'\n", act->name); ++ pollFile(act); ++ } ++} ++ ++static rsRetVal ++process_symlink(fs_edge_t *const chld, const char *symlink) ++{ ++ DEFiRet; ++ char *target = NULL; ++ CHKmalloc(target = realpath(symlink, target)); ++ struct stat fileInfo; ++ if(lstat(target, &fileInfo) != 0) { ++ LogError(errno, RS_RET_ERR, "imfile: process_symlink cannot stat file '%s' - ignored", target); ++ FINALIZE; ++ } ++ const int is_file = (S_ISREG(fileInfo.st_mode)); ++ DBGPRINTF("process_symlink: found '%s', File: %d (config file: %d), symlink: %d\n", ++ target, is_file, chld->is_file, 0); ++ if (act_obj_add(chld, target, is_file, fileInfo.st_ino, 0, symlink) == RS_RET_OK) { ++ /* need to watch parent target as well for proper rotation support */ ++ uint idx = ustrlen(chld->active->name) - ustrlen(chld->active->basename); ++ if (idx) { /* basename is different from name */ ++ char parent[MAXFNAME]; ++ memcpy(parent, chld->active->name, idx-1); ++ parent[idx-1] = '\0'; ++ if(lstat(parent, &fileInfo) != 0) { ++ LogError(errno, RS_RET_ERR, ++ "imfile: process_symlink: cannot stat directory '%s' - ignored", parent); ++ FINALIZE; ++ } ++ DBGPRINTF("process_symlink: adding parent '%s' of target '%s'\n", parent, target); ++ act_obj_add(chld->parent->root->edges, parent, 0, fileInfo.st_ino, 0, NULL); ++ } ++ } ++ ++finalize_it: ++ free(target); ++ RETiRet; ++} ++ ++static void ++poll_tree(fs_edge_t *const chld) ++{ ++ struct stat fileInfo; ++ glob_t files; ++ int issymlink; ++ DBGPRINTF("poll_tree: chld %p, name '%s', path: %s\n", chld, chld->name, chld->path); ++ detect_updates(chld); ++ const int ret = glob((char*)chld->path, runModConf->sortFiles|GLOB_BRACE, NULL, &files); ++ DBGPRINTF("poll_tree: glob returned %d\n", ret); ++ if(ret == 0) { ++ DBGPRINTF("poll_tree: processing %d files\n", (int) files.gl_pathc); ++ for(unsigned i = 0 ; i < files.gl_pathc ; i++) { ++ if(glbl.GetGlobalInputTermState() != 0) { ++ goto done; ++ } ++ char *const file = files.gl_pathv[i]; ++ if(lstat(file, &fileInfo) != 0) { ++ LogError(errno, RS_RET_ERR, ++ "imfile: poll_tree cannot stat file '%s' - ignored", file); ++ continue; ++ } ++ ++ if (S_ISLNK(fileInfo.st_mode)) { ++ rsRetVal slink_ret = process_symlink(chld, file); ++ if (slink_ret != RS_RET_OK) { ++ continue; ++ } ++ issymlink = 1; ++ } else { ++ issymlink = 0; ++ } ++ const int is_file = (S_ISREG(fileInfo.st_mode) || issymlink); ++ DBGPRINTF("poll_tree: found '%s', File: %d (config file: %d), symlink: %d\n", ++ file, is_file, chld->is_file, issymlink); ++ if(!is_file && S_ISREG(fileInfo.st_mode)) { ++ LogMsg(0, RS_RET_ERR, LOG_WARNING, ++ "imfile: '%s' is neither a regular file, symlink, nor a " ++ "directory - ignored", file); ++ continue; ++ } ++ if(chld->is_file != is_file) { ++ LogMsg(0, RS_RET_ERR, LOG_WARNING, ++ "imfile: '%s' is %s but %s expected - ignored", ++ file, (is_file) ? "FILE" : "DIRECTORY", ++ (chld->is_file) ? "FILE" : "DIRECTORY"); ++ continue; ++ } ++ act_obj_add(chld, file, is_file, fileInfo.st_ino, issymlink, NULL); ++ } ++ globfree(&files); ++ } ++ ++ poll_active_files(chld); ++ ++done: return; ++} ++ ++#ifdef HAVE_INOTIFY_INIT // TODO: shouldn't we use that in polling as well? ++static void ++poll_timeouts(fs_edge_t *const edge) ++{ ++ if(edge->is_file) { ++ act_obj_t *act; ++ for(act = edge->active ; act != NULL ; act = act->next) { ++ if(strmReadMultiLine_isTimedOut(act->pStrm)) { ++ DBGPRINTF("timeout occured on %s\n", act->name); ++ pollFile(act); ++ } ++ } ++ } ++} ++#endif ++ ++ ++/* destruct a single act_obj object */ ++static void ++act_obj_destroy(act_obj_t *const act, const int is_deleted) ++{ ++ uchar *statefn; ++ uchar statefile[MAXFNAME]; ++ uchar toDel[MAXFNAME]; ++ ++ if(act == NULL) ++ return; ++ ++ DBGPRINTF("act_obj_destroy: act %p '%s', (source '%s'), wd %d, pStrm %p, is_deleted %d, in_move %d\n", ++ act, act->name, act->source_name? act->source_name : "---", act->wd, act->pStrm, is_deleted, act->in_move); ++ if(act->is_symlink && is_deleted) { ++ act_obj_t *target_act; ++ for(target_act = act->edge->active ; target_act != NULL ; target_act = target_act->next) { ++ if(target_act->source_name && !strcmp(target_act->source_name, act->name)) { ++ DBGPRINTF("act_obj_destroy: unlinking slink target %s of %s " ++ "symlink\n", target_act->name, act->name); ++ act_obj_unlink(target_act); ++ break; ++ } ++ } ++ } ++ if(act->ratelimiter != NULL) { ++ ratelimitDestruct(act->ratelimiter); ++ } ++ if(act->pStrm != NULL) { ++ const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? ++ pollFile(act); /* get any left-over data */ ++ if(inst->bRMStateOnDel) { ++ statefn = getStateFileName(act, statefile, sizeof(statefile)); ++ getFullStateFileName(statefn, toDel, sizeof(toDel)); ++ statefn = toDel; ++ } ++ persistStrmState(act); ++ strm.Destruct(&act->pStrm); ++ /* we delete state file after destruct in case strm obj initiated a write */ ++ if(is_deleted && !act->in_move && inst->bRMStateOnDel) { ++ DBGPRINTF("act_obj_destroy: deleting state file %s\n", statefn); ++ unlink((char*)statefn); ++ } ++ } ++ #ifdef HAVE_INOTIFY_INIT ++ if(act->wd != -1) { ++ wdmapDel(act->wd); ++ } ++ #endif ++ #if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) ++ if(act->pfinf != NULL) { ++ free(act->pfinf->fobj.fo_name); ++ free(act->pfinf); ++ } ++ #endif ++ free(act->basename); ++ free(act->source_name); ++ //free(act->statefile); ++ free(act->multiSub.ppMsgs); ++ #if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) ++ act->is_deleted = 1; ++ #else ++ free(act->name); ++ free(act); ++ #endif ++} ++ + ++/* destroy complete act list starting at given node */ ++static void ++act_obj_destroy_all(act_obj_t *act) ++{ ++ if(act == NULL) ++ return; ++ ++ DBGPRINTF("act_obj_destroy_all: act %p '%s', wd %d, pStrm %p\n", act, act->name, act->wd, act->pStrm); ++ while(act != NULL) { ++ act_obj_t *const toDel = act; ++ act = act->next; ++ act_obj_destroy(toDel, 0); ++ } ++} ++ ++#if 0 ++/* debug: find if ptr is still present in list */ ++static void ++chk_active(const act_obj_t *act, const act_obj_t *const deleted) ++{ ++ while(act != NULL) { ++ DBGPRINTF("chk_active %p vs %p\n", act, deleted); ++ if(act->prev == deleted) ++ DBGPRINTF("chk_active %p prev points to %p\n", act, deleted); ++ if(act->next == deleted) ++ DBGPRINTF("chk_active %p next points to %p\n", act, deleted); ++ act = act->next; ++ DBGPRINTF("chk_active next %p\n", act); ++ } ++} ++#endif ++ ++/* unlink act object from linked list and then ++ * destruct it. ++ */ ++static void ++act_obj_unlink(act_obj_t *act) ++{ ++ DBGPRINTF("act_obj_unlink %p: %s\n", act, act->name); ++ if(act->prev == NULL) { ++ act->edge->active = act->next; ++ } else { ++ act->prev->next = act->next; ++ } ++ if(act->next != NULL) { ++ act->next->prev = act->prev; ++ } ++ act_obj_destroy(act, 1); ++ act = NULL; ++//dbgprintf("printout of fs tree post unlink\n"); ++//fs_node_print(runModConf->conf_tree, 0); ++//dbg_wdmapPrint("wdmap after"); ++} ++ ++static void ++fs_node_destroy(fs_node_t *const node) ++{ ++ fs_edge_t *edge; ++ DBGPRINTF("node destroy: %p edges:\n", node); ++ ++ for(edge = node->edges ; edge != NULL ; ) { ++ fs_node_destroy(edge->node); ++ fs_edge_t *const toDel = edge; ++ edge = edge->next; ++ act_obj_destroy_all(toDel->active); ++ free(toDel->name); ++ free(toDel->path); ++ free(toDel->instarr); ++ free(toDel); ++ } ++ free(node); ++} ++ ++static void ++fs_node_walk(fs_node_t *const node, ++ void (*f_usr)(fs_edge_t*const)) ++{ ++ DBGPRINTF("node walk: %p edges:\n", node); ++ ++ fs_edge_t *edge; ++ for(edge = node->edges ; edge != NULL ; edge = edge->next) { ++ DBGPRINTF("node walk: child %p '%s'\n", edge->node, edge->name); ++ f_usr(edge); ++ fs_node_walk(edge->node, f_usr); ++ } ++} ++ ++ ++ ++/* add a file system object to config tree (or update existing node with new monitor) ++ */ ++static rsRetVal ++fs_node_add(fs_node_t *const node, fs_node_t *const source, ++ const uchar *const toFind, ++ const size_t pathIdx, ++ instanceConf_t *const inst) ++{ ++ DEFiRet; ++ fs_edge_t *newchld = NULL; ++ int i; ++ ++ DBGPRINTF("fs_node_add(%p, '%s') enter, idx %zd\n", ++ node, toFind+pathIdx, pathIdx); ++ assert(toFind[0] != '\0'); ++ for(i = pathIdx ; (toFind[i] != '\0') && (toFind[i] != '/') ; ++i) ++ /*JUST SKIP*/; ++ const int isFile = (toFind[i] == '\0') ? 1 : 0; ++ uchar ourPath[PATH_MAX]; ++ if(i == 0) { ++ ourPath[0] = '/'; ++ ourPath[1] = '\0'; ++ } else { ++ memcpy(ourPath, toFind, i); ++ ourPath[i] = '\0'; ++ } ++ const size_t nextPathIdx = i+1; ++ const size_t len = i - pathIdx; ++ uchar name[PATH_MAX]; ++ memcpy(name, toFind+pathIdx, len); ++ name[len] = '\0'; ++ DBGPRINTF("fs_node_add: name '%s'\n", name); node->root = source; ++ ++ fs_edge_t *chld; ++ for(chld = node->edges ; chld != NULL ; chld = chld->next) { ++ if(!ustrcmp(chld->name, name)) { ++ DBGPRINTF("fs_node_add(%p, '%s') found '%s'\n", chld->node, toFind, name); ++ /* add new instance */ ++ chld->ninst++; ++ CHKmalloc(chld->instarr = realloc(chld->instarr, sizeof(instanceConf_t*) * chld->ninst)); ++ chld->instarr[chld->ninst-1] = inst; ++ /* recurse */ ++ if(!isFile) { ++ CHKiRet(fs_node_add(chld->node, node, toFind, nextPathIdx, inst)); ++ } ++ FINALIZE; ++ } ++ } ++ ++ /* could not find node --> add it */ ++ DBGPRINTF("fs_node_add(%p, '%s') did not find '%s' - adding it\n", ++ node, toFind, name); ++ CHKmalloc(newchld = calloc(sizeof(fs_edge_t), 1)); ++ CHKmalloc(newchld->name = ustrdup(name)); ++ CHKmalloc(newchld->node = calloc(sizeof(fs_node_t), 1)); ++ CHKmalloc(newchld->path = ustrdup(ourPath)); ++ CHKmalloc(newchld->instarr = calloc(sizeof(instanceConf_t*), 1)); ++ newchld->instarr[0] = inst; ++ newchld->is_file = isFile; ++ newchld->ninst = 1; ++ newchld->parent = node; ++ ++ DBGPRINTF("fs_node_add(%p, '%s') returns %p\n", node, toFind, newchld->node); ++ ++ if(!isFile) { ++ CHKiRet(fs_node_add(newchld->node, node, toFind, nextPathIdx, inst)); ++ } ++ ++ /* link to list */ ++ newchld->next = node->edges; ++ node->edges = newchld; ++finalize_it: ++ if(iRet != RS_RET_OK) { ++ if(newchld != NULL) { ++ free(newchld->name); ++ free(newchld->node); ++ free(newchld->path); ++ free(newchld->instarr); ++ free(newchld); ++ } ++ } ++ RETiRet; ++} ++ ++/* Helper function to combine statefile and workdir ++ * This function is guranteed to work only on config data and DOES NOT ++ * open or otherwise modify disk file state. ++ */ ++static int ++getFullStateFileName(const uchar *const pszstatefile, uchar *const pszout, const size_t ilenout) ++{ ++ int lenout; ++ const uchar* pszworkdir; + +-/* this generates a state file name suitable for the current file. To avoid ++ /* Get Raw Workdir, if it is NULL we need to propper handle it */ ++ pszworkdir = glblGetWorkDirRaw(); ++ ++ /* Construct file name */ ++ lenout = snprintf((char*)pszout, ilenout, "%s/%s", ++ (char*) (pszworkdir == NULL ? "." : (char*) pszworkdir), (char*)pszstatefile); ++ ++ /* return out length */ ++ return lenout; ++} ++ ++ ++/* this generates a state file name suitable for the given file. To avoid + * malloc calls, it must be passed a buffer which should be MAXFNAME large. + * Note: the buffer is not necessarily populated ... always ONLY use the + * RETURN VALUE! ++ * This function is guranteed to work only on config data and DOES NOT ++ * open or otherwise modify disk file state. + */ + static uchar * +-getStateFileName(lstn_t *const __restrict__ pLstn, ++getStateFileName(const act_obj_t *const act, + uchar *const __restrict__ buf, + const size_t lenbuf) + { +- uchar *ret; +- if(pLstn->pszStateFile == NULL) { +- snprintf((char*)buf, lenbuf - 1, "imfile-state:%s", pLstn->pszFileName); +- buf[lenbuf-1] = '\0'; /* be on the safe side... */ +- uchar *p = buf; +- for( ; *p ; ++p) { +- if(*p == '/') +- *p = '-'; +- } +- ret = buf; +- } else { +- ret = pLstn->pszStateFile; +- } +- return ret; ++ DBGPRINTF("getStateFileName for '%s'\n", act->name); ++ snprintf((char*)buf, lenbuf - 1, "imfile-state:%lld", (long long) act->ino); ++ DBGPRINTF("getStateFileName: stat file name now is %s\n", buf); ++ return buf; + } + + + /* enqueue the read file line as a message. The provided string is +- * not freed - thuis must be done by the caller. ++ * not freed - this must be done by the caller. + */ +-static rsRetVal enqLine(lstn_t *const __restrict__ pLstn, +- cstr_t *const __restrict__ cstrLine) ++#define MAX_OFFSET_REPRESENTATION_NUM_BYTES 20 ++static rsRetVal ++enqLine(act_obj_t *const act, ++ cstr_t *const __restrict__ cstrLine, ++ const int64 strtOffs) + { + DEFiRet; ++ const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + smsg_t *pMsg; ++ uchar file_offset[MAX_OFFSET_REPRESENTATION_NUM_BYTES+1]; ++ const uchar *metadata_names[2] = {(uchar *)"filename",(uchar *)"fileoffset"} ; ++ const uchar *metadata_values[2] ; ++ const size_t msgLen = cstrLen(cstrLine); + +- if(rsCStrLen(cstrLine) == 0) { ++ if(msgLen == 0) { + /* we do not process empty lines */ + FINALIZE; + } +@@ -474,27 +1180,34 @@ static rsRetVal enqLine(lstn_t *const __restrict__ pLstn, + CHKiRet(msgConstruct(&pMsg)); + MsgSetFlowControlType(pMsg, eFLOWCTL_FULL_DELAY); + MsgSetInputName(pMsg, pInputName); +- if (pLstn->addCeeTag) { +- size_t msgLen = cstrLen(cstrLine); +- const char *const ceeToken = "@cee:"; +- size_t ceeMsgSize = msgLen + strlen(ceeToken) +1; ++ if(inst->addCeeTag) { ++ /* Make sure we account for terminating null byte */ ++ size_t ceeMsgSize = msgLen + CONST_LEN_CEE_COOKIE + 1; + char *ceeMsg; + CHKmalloc(ceeMsg = MALLOC(ceeMsgSize)); +- strcpy(ceeMsg, ceeToken); ++ strcpy(ceeMsg, CONST_CEE_COOKIE); + strcat(ceeMsg, (char*)rsCStrGetSzStrNoNULL(cstrLine)); + MsgSetRawMsg(pMsg, ceeMsg, ceeMsgSize); + free(ceeMsg); + } else { +- MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStrNoNULL(cstrLine), cstrLen(cstrLine)); ++ MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStrNoNULL(cstrLine), msgLen); + } + MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); +- MsgSetTAG(pMsg, pLstn->pszTag, pLstn->lenTag); +- msgSetPRI(pMsg, pLstn->iFacility | pLstn->iSeverity); +- MsgSetRuleset(pMsg, pLstn->pRuleset); +- if(pLstn->addMetadata) +- msgAddMetadata(pMsg, (uchar*)"filename", pLstn->pszFileName); +- ratelimitAddMsg(pLstn->ratelimiter, &pLstn->multiSub, pMsg); ++ MsgSetTAG(pMsg, inst->pszTag, inst->lenTag); ++ msgSetPRI(pMsg, inst->iFacility | inst->iSeverity); ++ MsgSetRuleset(pMsg, inst->pBindRuleset); ++ if(inst->addMetadata) { ++ if (act->source_name) { ++ metadata_values[0] = (const uchar*)act->source_name; ++ } else { ++ metadata_values[0] = (const uchar*)act->name; ++ } ++ snprintf((char *)file_offset, MAX_OFFSET_REPRESENTATION_NUM_BYTES+1, "%lld", strtOffs); ++ metadata_values[1] = file_offset; ++ msgAddMultiMetadata(pMsg, metadata_names, metadata_values, 2); ++ } ++ ratelimitAddMsg(act->ratelimiter, &act->multiSub, pMsg); + finalize_it: + RETiRet; + } +@@ -504,70 +1213,89 @@ finalize_it: + * exist or cannot be read, an error is returned. + */ + static rsRetVal +-openFileWithStateFile(lstn_t *const __restrict__ pLstn) ++openFileWithStateFile(act_obj_t *const act) + { + DEFiRet; +- strm_t *psSF = NULL; + uchar pszSFNam[MAXFNAME]; +- size_t lenSFNam; +- struct stat stat_buf; + uchar statefile[MAXFNAME]; ++ int fd = -1; ++ const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + +- uchar *const statefn = getStateFileName(pLstn, statefile, sizeof(statefile)); +- DBGPRINTF("imfile: trying to open state for '%s', state file '%s'\n", +- pLstn->pszFileName, statefn); +- /* Construct file name */ +- lenSFNam = snprintf((char*)pszSFNam, sizeof(pszSFNam), "%s/%s", +- (char*) glbl.GetWorkDir(), (char*)statefn); ++ uchar *const statefn = getStateFileName(act, statefile, sizeof(statefile)); ++ ++ getFullStateFileName(statefn, pszSFNam, sizeof(pszSFNam)); ++ DBGPRINTF("trying to open state for '%s', state file '%s'\n", act->name, pszSFNam); + + /* check if the file exists */ +- if(stat((char*) pszSFNam, &stat_buf) == -1) { ++ fd = open((char*)pszSFNam, O_CLOEXEC | O_NOCTTY | O_RDONLY, 0600); ++ if(fd < 0) { + if(errno == ENOENT) { +- DBGPRINTF("imfile: NO state file exists for '%s'\n", pLstn->pszFileName); +- ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); ++ DBGPRINTF("NO state file (%s) exists for '%s' - trying to see if " ++ "old-style file exists\n", pszSFNam, act->name); ++ CHKiRet(OLD_openFileWithStateFile(act)); ++ FINALIZE; + } else { +- char errStr[1024]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- DBGPRINTF("imfile: error trying to access state file for '%s':%s\n", +- pLstn->pszFileName, errStr); ++ LogError(errno, RS_RET_IO_ERROR, ++ "imfile error trying to access state file for '%s'", ++ act->name); + ABORT_FINALIZE(RS_RET_IO_ERROR); + } + } + +- /* If we reach this point, we have a state file */ ++ CHKiRet(strm.Construct(&act->pStrm)); + +- CHKiRet(strm.Construct(&psSF)); +- CHKiRet(strm.SettOperationsMode(psSF, STREAMMODE_READ)); +- CHKiRet(strm.SetsType(psSF, STREAMTYPE_FILE_SINGLE)); +- CHKiRet(strm.SetFName(psSF, pszSFNam, lenSFNam)); +- CHKiRet(strm.ConstructFinalize(psSF)); ++ struct json_object *jval; ++ struct json_object *json = fjson_object_from_fd(fd); ++ if(json == NULL) { ++ LogError(0, RS_RET_ERR, "imfile: error reading state file for '%s'", act->name); ++ } + +- /* read back in the object */ +- CHKiRet(obj.Deserialize(&pLstn->pStrm, (uchar*) "strm", psSF, NULL, pLstn)); +- DBGPRINTF("imfile: deserialized state file, state file base name '%s', " +- "configured base name '%s'\n", pLstn->pStrm->pszFName, +- pLstn->pszFileName); +- if(ustrcmp(pLstn->pStrm->pszFName, pLstn->pszFileName)) { +- errmsg.LogError(0, RS_RET_STATEFILE_WRONG_FNAME, "imfile: state file '%s' " +- "contains file name '%s', but is used for file '%s'. State " +- "file deleted, starting from begin of file.", +- pszSFNam, pLstn->pStrm->pszFName, pLstn->pszFileName); ++ /* we access some data items a bit dirty, as we need to refactor the whole ++ * thing in any case - TODO ++ */ ++ /* Note: we ignore filname property - it is just an aid to the user. Most ++ * importantly it *is wrong* after a file move! ++ */ ++ fjson_object_object_get_ex(json, "prev_was_nl", &jval); ++ act->pStrm->bPrevWasNL = fjson_object_get_int(jval); + +- unlink((char*)pszSFNam); +- ABORT_FINALIZE(RS_RET_STATEFILE_WRONG_FNAME); ++ fjson_object_object_get_ex(json, "curr_offs", &jval); ++ act->pStrm->iCurrOffs = fjson_object_get_int64(jval); ++ ++ fjson_object_object_get_ex(json, "strt_offs", &jval); ++ act->pStrm->strtOffs = fjson_object_get_int64(jval); ++ ++ fjson_object_object_get_ex(json, "prev_line_segment", &jval); ++ const uchar *const prev_line_segment = (const uchar*)fjson_object_get_string(jval); ++ if(jval != NULL) { ++ CHKiRet(rsCStrConstructFromszStr(&act->pStrm->prevLineSegment, prev_line_segment)); ++ cstrFinalize(act->pStrm->prevLineSegment); ++ uchar *ret = rsCStrGetSzStrNoNULL(act->pStrm->prevLineSegment); ++ DBGPRINTF("prev_line_segment present in state file 2, is: %s\n", ret); + } + +- strm.CheckFileChange(pLstn->pStrm); +- CHKiRet(strm.SeekCurrOffs(pLstn->pStrm)); ++ fjson_object_object_get_ex(json, "prev_msg_segment", &jval); ++ const uchar *const prev_msg_segment = (const uchar*)fjson_object_get_string(jval); ++ if(jval != NULL) { ++ CHKiRet(rsCStrConstructFromszStr(&act->pStrm->prevMsgSegment, prev_msg_segment)); ++ cstrFinalize(act->pStrm->prevMsgSegment); ++ uchar *ret = rsCStrGetSzStrNoNULL(act->pStrm->prevMsgSegment); ++ DBGPRINTF("prev_msg_segment present in state file 2, is: %s\n", ret); ++ } ++ fjson_object_put(json); + +- /* note: we do not delete the state file, so that the last position remains +- * known even in the case that rsyslogd aborts for some reason (like powerfail) +- */ ++ CHKiRet(strm.SetFName(act->pStrm, (uchar*)act->name, strlen(act->name))); ++ CHKiRet(strm.SettOperationsMode(act->pStrm, STREAMMODE_READ)); ++ CHKiRet(strm.SetsType(act->pStrm, STREAMTYPE_FILE_MONITOR)); ++ CHKiRet(strm.SetFileNotFoundError(act->pStrm, inst->fileNotFoundError)); ++ CHKiRet(strm.ConstructFinalize(act->pStrm)); + +-finalize_it: +- if(psSF != NULL) +- strm.Destruct(&psSF); ++ CHKiRet(strm.SeekCurrOffs(act->pStrm)); + ++finalize_it: ++ if(fd >= 0) { ++ close(fd); ++ } + RETiRet; + } + +@@ -576,30 +1304,32 @@ finalize_it: + * checked before calling it. + */ + static rsRetVal +-openFileWithoutStateFile(lstn_t *const __restrict__ pLstn) ++openFileWithoutStateFile(act_obj_t *const act) + { + DEFiRet; + struct stat stat_buf; + +- DBGPRINTF("imfile: clean startup withOUT state file for '%s'\n", pLstn->pszFileName); +- if(pLstn->pStrm != NULL) +- strm.Destruct(&pLstn->pStrm); +- CHKiRet(strm.Construct(&pLstn->pStrm)); +- CHKiRet(strm.SettOperationsMode(pLstn->pStrm, STREAMMODE_READ)); +- CHKiRet(strm.SetsType(pLstn->pStrm, STREAMTYPE_FILE_MONITOR)); +- CHKiRet(strm.SetFName(pLstn->pStrm, pLstn->pszFileName, strlen((char*) pLstn->pszFileName))); +- CHKiRet(strm.ConstructFinalize(pLstn->pStrm)); ++ const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? ++ ++ DBGPRINTF("clean startup withOUT state file for '%s'\n", act->name); ++ if(act->pStrm != NULL) ++ strm.Destruct(&act->pStrm); ++ CHKiRet(strm.Construct(&act->pStrm)); ++ CHKiRet(strm.SettOperationsMode(act->pStrm, STREAMMODE_READ)); ++ CHKiRet(strm.SetsType(act->pStrm, STREAMTYPE_FILE_MONITOR)); ++ CHKiRet(strm.SetFName(act->pStrm, (uchar*)act->name, strlen(act->name))); ++ CHKiRet(strm.SetFileNotFoundError(act->pStrm, inst->fileNotFoundError)); ++ CHKiRet(strm.ConstructFinalize(act->pStrm)); + + /* As a state file not exist, this is a fresh start. seek to file end + * when freshStartTail is on. + */ +- if(pLstn->freshStartTail){ +- if(stat((char*) pLstn->pszFileName, &stat_buf) != -1) { +- pLstn->pStrm->iCurrOffs = stat_buf.st_size; +- CHKiRet(strm.SeekCurrOffs(pLstn->pStrm)); ++ if(inst->freshStartTail){ ++ if(stat((char*) act->name, &stat_buf) != -1) { ++ act->pStrm->iCurrOffs = stat_buf.st_size; ++ CHKiRet(strm.SeekCurrOffs(act->pStrm)); + } + } +- strmSetReadTimeout(pLstn->pStrm, pLstn->readTimeout); + + finalize_it: + RETiRet; +@@ -608,17 +1338,18 @@ finalize_it: + * if so, reading it in. Processing continues from the last know location. + */ + static rsRetVal +-openFile(lstn_t *const __restrict__ pLstn) ++openFile(act_obj_t *const act) + { + DEFiRet; ++ const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + +- CHKiRet_Hdlr(openFileWithStateFile(pLstn)) { +- CHKiRet(openFileWithoutStateFile(pLstn)); ++ CHKiRet_Hdlr(openFileWithStateFile(act)) { ++ CHKiRet(openFileWithoutStateFile(act)); + } + +- DBGPRINTF("imfile: breopenOnTruncate %d for '%s'\n", +- pLstn->reopenOnTruncate, pLstn->pszFileName); +- CHKiRet(strm.SetbReopenOnTruncate(pLstn->pStrm, pLstn->reopenOnTruncate)); ++ DBGPRINTF("breopenOnTruncate %d for '%s'\n", inst->reopenOnTruncate, act->name); ++ CHKiRet(strm.SetbReopenOnTruncate(act->pStrm, inst->reopenOnTruncate)); ++ strmSetReadTimeout(act->pStrm, inst->readTimeout); + + finalize_it: + RETiRet; +@@ -638,58 +1369,72 @@ static void pollFileCancelCleanup(void *pArg) + } + + +-/* poll a file, need to check file rollover etc. open file if not open */ +-#if !defined(_AIX) +-#pragma GCC diagnostic ignored "-Wempty-body" +-#endif ++/* pollFile needs to be split due to the unfortunate pthread_cancel_push() macros. */ + static rsRetVal +-pollFile(lstn_t *pLstn, int *pbHadFileData) ++pollFileReal(act_obj_t *act, cstr_t **pCStr) + { +- cstr_t *pCStr = NULL; ++ int64 strtOffs; + DEFiRet; +- +- /* Note: we must do pthread_cleanup_push() immediately, because the POXIS macros +- * otherwise do not work if I include the _cleanup_pop() inside an if... -- rgerhards, 2008-08-14 +- */ +- pthread_cleanup_push(pollFileCancelCleanup, &pCStr); + int nProcessed = 0; +- if(pLstn->pStrm == NULL) { +- CHKiRet(openFile(pLstn)); /* open file */ ++ ++ DBGPRINTF("pollFileReal enter, pStrm %p, name '%s'\n", act->pStrm, act->name); ++ DBGPRINTF("pollFileReal enter, edge %p\n", act->edge); ++ DBGPRINTF("pollFileReal enter, edge->instarr %p\n", act->edge->instarr); ++ ++ instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? ++ ++ if(act->pStrm == NULL) { ++ CHKiRet(openFile(act)); /* open file */ + } + + /* loop below will be exited when strmReadLine() returns EOF */ + while(glbl.GetGlobalInputTermState() == 0) { +- if(pLstn->maxLinesAtOnce != 0 && nProcessed >= pLstn->maxLinesAtOnce) ++ if(inst->maxLinesAtOnce != 0 && nProcessed >= inst->maxLinesAtOnce) + break; +- if(pLstn->startRegex == NULL) { +- CHKiRet(strm.ReadLine(pLstn->pStrm, &pCStr, pLstn->readMode, pLstn->escapeLF, pLstn->trimLineOverBytes)); ++ if(inst->startRegex == NULL) { ++ CHKiRet(strm.ReadLine(act->pStrm, pCStr, inst->readMode, inst->escapeLF, ++ inst->trimLineOverBytes, &strtOffs)); + } else { +- CHKiRet(strmReadMultiLine(pLstn->pStrm, &pCStr, &pLstn->end_preg, pLstn->escapeLF)); ++ CHKiRet(strmReadMultiLine(act->pStrm, pCStr, &inst->end_preg, ++ inst->escapeLF, &strtOffs)); + } + ++nProcessed; +- if(pbHadFileData != NULL) +- *pbHadFileData = 1; /* this is just a flag, so set it and forget it */ +- CHKiRet(enqLine(pLstn, pCStr)); /* process line */ +- rsCStrDestruct(&pCStr); /* discard string (must be done by us!) */ +- if(pLstn->iPersistStateInterval > 0 && pLstn->nRecords++ >= pLstn->iPersistStateInterval) { +- persistStrmState(pLstn); +- pLstn->nRecords = 0; ++ runModConf->bHadFileData = 1; /* this is just a flag, so set it and forget it */ ++ CHKiRet(enqLine(act, *pCStr, strtOffs)); /* process line */ ++ rsCStrDestruct(pCStr); /* discard string (must be done by us!) */ ++ if(inst->iPersistStateInterval > 0 && ++act->nRecords >= inst->iPersistStateInterval) { ++ persistStrmState(act); ++ act->nRecords = 0; + } + } + + finalize_it: +- multiSubmitFlush(&pLstn->multiSub); +- pthread_cleanup_pop(0); ++ multiSubmitFlush(&act->multiSub); + +- if(pCStr != NULL) { +- rsCStrDestruct(&pCStr); ++ if(*pCStr != NULL) { ++ rsCStrDestruct(pCStr); + } + + RETiRet; + } +-#if !defined(_AIX) +-#pragma GCC diagnostic warning "-Wempty-body" +-#endif ++ ++/* poll a file, need to check file rollover etc. open file if not open */ ++static rsRetVal ++pollFile(act_obj_t *const act) ++{ ++ cstr_t *pCStr = NULL; ++ DEFiRet; ++ if (act->is_symlink) { ++ FINALIZE; /* no reason to poll symlink file */ ++ } ++ /* Note: we must do pthread_cleanup_push() immediately, because the POSIX macros ++ * otherwise do not work if I include the _cleanup_pop() inside an if... -- rgerhards, 2008-08-14 ++ */ ++ pthread_cleanup_push(pollFileCancelCleanup, &pCStr); ++ iRet = pollFileReal(act, &pCStr); ++ pthread_cleanup_pop(0); ++finalize_it: RETiRet; ++} + + + /* create input instance, set default parameters, and +@@ -722,6 +1467,7 @@ createInstance(instanceConf_t **pinst) + inst->addMetadata = ADD_METADATA_UNSPECIFIED; + inst->addCeeTag = 0; + inst->freshStartTail = 0; ++ inst->fileNotFoundError = 1; + inst->readTimeout = loadModConf->readTimeout; + + /* node created, let's add to config */ +@@ -767,19 +1513,11 @@ getBasename(uchar *const __restrict__ basen, uchar *const __restrict__ path) + } + + /* this function checks instance parameters and does some required pre-processing +- * (e.g. split filename in path and actual name) +- * Note: we do NOT use dirname()/basename() as they have portability problems. + */ + static rsRetVal +-checkInstance(instanceConf_t *inst) ++checkInstance(instanceConf_t *const inst) + { +- char dirn[MAXFNAME]; +- uchar basen[MAXFNAME]; +- int i; +- struct stat sb; +- int r; +- int eno; +- char errStr[512]; ++ uchar curr_wd[MAXFNAME]; + DEFiRet; + + /* this is primarily for the clang static analyzer, but also +@@ -788,36 +1526,37 @@ checkInstance(instanceConf_t *inst) + if(inst->pszFileName == NULL) + ABORT_FINALIZE(RS_RET_INTERNAL_ERROR); + +- i = getBasename(basen, inst->pszFileName); +- if (i == -1) { +- errmsg.LogError(0, RS_RET_CONFIG_ERROR, "imfile: file path '%s' does not include a basename component", +- inst->pszFileName); +- ABORT_FINALIZE(RS_RET_CONFIG_ERROR); +- } +- +- memcpy(dirn, inst->pszFileName, i); /* do not copy slash */ +- dirn[i] = '\0'; +- CHKmalloc(inst->pszFileBaseName = (uchar*) strdup((char*)basen)); +- CHKmalloc(inst->pszDirName = (uchar*) strdup(dirn)); +- +- if(dirn[0] == '\0') { +- dirn[0] = '/'; +- dirn[1] = '\0'; +- } +- r = stat(dirn, &sb); +- if(r != 0) { +- eno = errno; +- rs_strerror_r(eno, errStr, sizeof(errStr)); +- errmsg.LogError(0, RS_RET_CONFIG_ERROR, "imfile warning: directory '%s': %s", +- dirn, errStr); +- ABORT_FINALIZE(RS_RET_CONFIG_ERROR); +- } +- if(!S_ISDIR(sb.st_mode)) { +- errmsg.LogError(0, RS_RET_CONFIG_ERROR, "imfile warning: configured directory " +- "'%s' is NOT a directory", dirn); +- ABORT_FINALIZE(RS_RET_CONFIG_ERROR); ++ CHKmalloc(inst->pszFileName_forOldStateFile = ustrdup(inst->pszFileName)); ++ if(loadModConf->normalizePath) { ++ if(inst->pszFileName[0] == '.' && inst->pszFileName[1] == '/') { ++ DBGPRINTF("imfile: removing heading './' from name '%s'\n", inst->pszFileName); ++ memmove(inst->pszFileName, inst->pszFileName+2, ustrlen(inst->pszFileName) - 1); ++ } ++ ++ if(inst->pszFileName[0] != '/') { ++ if(getcwd((char*)curr_wd, MAXFNAME) == NULL || curr_wd[0] != '/') { ++ LogError(errno, RS_RET_ERR, "imfile: error querying current working " ++ "directory - can not continue with %s", inst->pszFileName); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ const size_t len_curr_wd = ustrlen(curr_wd); ++ if(len_curr_wd + ustrlen(inst->pszFileName) + 1 >= MAXFNAME) { ++ LogError(0, RS_RET_ERR, "imfile: length of configured file and current " ++ "working directory exceeds permitted size - ignoring %s", ++ inst->pszFileName); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ curr_wd[len_curr_wd] = '/'; ++ strcpy((char*)curr_wd+len_curr_wd+1, (char*)inst->pszFileName); ++ free(inst->pszFileName); ++ CHKmalloc(inst->pszFileName = ustrdup(curr_wd)); ++ } + } ++ dbgprintf("imfile: adding file monitor for '%s'\n", inst->pszFileName); + ++ if(inst->pszTag != NULL) { ++ inst->lenTag = ustrlen(inst->pszTag); ++ } + finalize_it: + RETiRet; + } +@@ -869,140 +1608,14 @@ addInstance(void __attribute__((unused)) *pVal, uchar *pNewVal) + inst->bRMStateOnDel = 0; + inst->readTimeout = loadModConf->readTimeout; + +- CHKiRet(checkInstance(inst)); +- +- /* reset legacy system */ +- cs.iPersistStateInterval = 0; +- resetConfigVariables(NULL, NULL); /* values are both dummies */ +- +-finalize_it: +- free(pNewVal); /* we do not need it, but we must free it! */ +- RETiRet; +-} +- +- +-/* This adds a new listener object to the bottom of the list, but +- * it does NOT initialize any data members except for the list +- * pointers themselves. +- */ +-static rsRetVal +-lstnAdd(lstn_t **newLstn) +-{ +- lstn_t *pLstn; +- DEFiRet; +- +- CHKmalloc(pLstn = (lstn_t*) MALLOC(sizeof(lstn_t))); +- if(runModConf->pRootLstn == NULL) { +- runModConf->pRootLstn = pLstn; +- pLstn->prev = NULL; +- } else { +- runModConf->pTailLstn->next = pLstn; +- pLstn->prev = runModConf->pTailLstn; +- } +- runModConf->pTailLstn = pLstn; +- pLstn->next = NULL; +- *newLstn = pLstn; +- +-finalize_it: +- RETiRet; +-} +- +-/* delete a listener object */ +-static void +-lstnDel(lstn_t *pLstn) +-{ +- DBGPRINTF("imfile: lstnDel called for %s\n", pLstn->pszFileName); +- if(pLstn->pStrm != NULL) { /* stream open? */ +- persistStrmState(pLstn); +- strm.Destruct(&(pLstn->pStrm)); +- } +- ratelimitDestruct(pLstn->ratelimiter); +- free(pLstn->multiSub.ppMsgs); +- free(pLstn->pszFileName); +- free(pLstn->pszTag); +- free(pLstn->pszStateFile); +- free(pLstn->pszBaseName); +- if(pLstn->startRegex != NULL) +- regfree(&pLstn->end_preg); +- +- if(pLstn == runModConf->pRootLstn) +- runModConf->pRootLstn = pLstn->next; +- if(pLstn == runModConf->pTailLstn) +- runModConf->pTailLstn = pLstn->prev; +- if(pLstn->next != NULL) +- pLstn->next->prev = pLstn->prev; +- if(pLstn->prev != NULL) +- pLstn->prev->next = pLstn->next; +- free(pLstn); +-} +- +-/* This function is called when a new listener shall be added. +- * It also does some late stage error checking on the config +- * and reports issues it finds. +- */ +-static rsRetVal +-addListner(instanceConf_t *inst) +-{ +- DEFiRet; +- lstn_t *pThis; +- sbool hasWildcard; +- +- hasWildcard = containsGlobWildcard((char*)inst->pszFileBaseName); +- if(hasWildcard) { +- if(runModConf->opMode == OPMODE_POLLING) { +- errmsg.LogError(0, RS_RET_IMFILE_WILDCARD, +- "imfile: The to-be-monitored file \"%s\" contains " +- "wildcards. This is not supported in " +- "polling mode.", inst->pszFileName); +- ABORT_FINALIZE(RS_RET_IMFILE_WILDCARD); +- } else if(inst->pszStateFile != NULL) { +- errmsg.LogError(0, RS_RET_IMFILE_WILDCARD, +- "imfile: warning: it looks like to-be-monitored " +- "file \"%s\" contains wildcards. This usually " +- "does not work well with specifying a state file.", +- inst->pszFileName); +- } +- } ++ CHKiRet(checkInstance(inst)); ++ ++ /* reset legacy system */ ++ cs.iPersistStateInterval = 0; ++ resetConfigVariables(NULL, NULL); /* values are both dummies */ + +- CHKiRet(lstnAdd(&pThis)); +- pThis->hasWildcard = hasWildcard; +- pThis->pszFileName = (uchar*) strdup((char*) inst->pszFileName); +- pThis->pszDirName = inst->pszDirName; /* use memory from inst! */ +- pThis->pszBaseName = (uchar*)strdup((char*)inst->pszFileBaseName); /* be consistent with expanded wildcards! */ +- pThis->pszTag = (uchar*) strdup((char*) inst->pszTag); +- pThis->lenTag = ustrlen(pThis->pszTag); +- pThis->pszStateFile = inst->pszStateFile == NULL ? NULL : (uchar*) strdup((char*) inst->pszStateFile); +- +- CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)inst->pszFileName)); +- CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(smsg_t *))); +- pThis->multiSub.maxElem = inst->nMultiSub; +- pThis->multiSub.nElem = 0; +- pThis->iSeverity = inst->iSeverity; +- pThis->iFacility = inst->iFacility; +- pThis->maxLinesAtOnce = inst->maxLinesAtOnce; +- pThis->trimLineOverBytes = inst->trimLineOverBytes; +- pThis->iPersistStateInterval = inst->iPersistStateInterval; +- pThis->readMode = inst->readMode; +- pThis->startRegex = inst->startRegex; /* no strdup, as it is read-only */ +- if(pThis->startRegex != NULL) +- if(regcomp(&pThis->end_preg, (char*)pThis->startRegex, REG_EXTENDED)) { +- DBGPRINTF("imfile: error regex compile\n"); +- ABORT_FINALIZE(RS_RET_ERR); +- } +- pThis->bRMStateOnDel = inst->bRMStateOnDel; +- pThis->escapeLF = inst->escapeLF; +- pThis->reopenOnTruncate = inst->reopenOnTruncate; +- pThis->addMetadata = (inst->addMetadata == ADD_METADATA_UNSPECIFIED) ? +- hasWildcard : inst->addMetadata; +- pThis->addCeeTag = inst->addCeeTag; +- pThis->readTimeout = inst->readTimeout; +- pThis->freshStartTail = inst->freshStartTail; +- pThis->pRuleset = inst->pBindRuleset; +- pThis->nRecords = 0; +- pThis->pStrm = NULL; +- pThis->prevLineSegment = NULL; +- pThis->masterLstn = NULL; /* we *are* a master! */ + finalize_it: ++ free(pNewVal); /* we do not need it, but we must free it! */ + RETiRet; + } + +@@ -1055,6 +1668,8 @@ CODESTARTnewInpInst + inst->addCeeTag = (sbool) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "freshstarttail")) { + inst->freshStartTail = (sbool) pvals[i].val.d.n; ++ } else if(!strcmp(inppblk.descr[i].name, "filenotfounderror")) { ++ inst->fileNotFoundError = (sbool) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "escapelf")) { + inst->escapeLF = (sbool) pvals[i].val.d.n; + } else if(!strcmp(inppblk.descr[i].name, "reopenontruncate")) { +@@ -1087,6 +1702,16 @@ CODESTARTnewInpInst + "at the same time --- remove one of them"); + ABORT_FINALIZE(RS_RET_PARAM_NOT_PERMITTED); + } ++ ++ if(inst->startRegex != NULL) { ++ const int errcode = regcomp(&inst->end_preg, (char*)inst->startRegex, REG_EXTENDED); ++ if(errcode != 0) { ++ char errbuff[512]; ++ regerror(errcode, &inst->end_preg, errbuff, sizeof(errbuff)); ++ parser_errmsg("imfile: error in regex expansion: %s", errbuff); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ } + if(inst->readTimeout != 0) + loadModConf->haveReadTimeouts = 1; + CHKiRet(checkInstance(inst)); +@@ -1106,6 +1731,10 @@ CODESTARTbeginCnfLoad + loadModConf->readTimeout = 0; /* default: no timeout */ + loadModConf->timeoutGranularity = 1000; /* default: 1 second */ + loadModConf->haveReadTimeouts = 0; /* default: no timeout */ ++ loadModConf->normalizePath = 1; ++ loadModConf->sortFiles = GLOB_NOSORT; ++ loadModConf->conf_tree = calloc(sizeof(fs_node_t), 1); ++ loadModConf->conf_tree->edges = NULL; + bLegacyCnfModGlobalsPermitted = 1; + /* init legacy config vars */ + cs.pszFileName = NULL; +@@ -1148,6 +1777,10 @@ CODESTARTsetModCnf + } else if(!strcmp(modpblk.descr[i].name, "timeoutgranularity")) { + /* note: we need ms, thus "* 1000" */ + loadModConf->timeoutGranularity = (int) pvals[i].val.d.n * 1000; ++ } else if(!strcmp(modpblk.descr[i].name, "sortfiles")) { ++ loadModConf->sortFiles = ((sbool) pvals[i].val.d.n) ? 0 : GLOB_NOSORT; ++ } else if(!strcmp(modpblk.descr[i].name, "normalizepath")) { ++ loadModConf->normalizePath = (sbool) pvals[i].val.d.n; + } else if(!strcmp(modpblk.descr[i].name, "mode")) { + if(!es_strconstcmp(pvals[i].val.d.estr, "polling")) + loadModConf->opMode = OPMODE_POLLING; +@@ -1217,19 +1850,31 @@ BEGINactivateCnf + instanceConf_t *inst; + CODESTARTactivateCnf + runModConf = pModConf; +- runModConf->pRootLstn = NULL, +- runModConf->pTailLstn = NULL; ++ if(runModConf->root == NULL) { ++ LogError(0, NO_ERRCODE, "imfile: no file monitors configured, " ++ "input not activated.\n"); ++ ABORT_FINALIZE(RS_RET_NO_RUN); ++ } + + for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { +- addListner(inst); ++ // TODO: provide switch to turn off this warning? ++ if(!containsGlobWildcard((char*)inst->pszFileName)) { ++ if(access((char*)inst->pszFileName, R_OK) != 0) { ++ LogError(errno, RS_RET_ERR, ++ "imfile: on startup file '%s' does not exist " ++ "but is configured in static file monitor - this " ++ "may indicate a misconfiguration. If the file " ++ "appears at a later time, it will automatically " ++ "be processed. Reason", inst->pszFileName); ++ } ++ } ++ fs_node_add(runModConf->conf_tree, NULL, inst->pszFileName, 0, inst); + } + +- /* if we could not set up any listeners, there is no point in running... */ +- if(runModConf->pRootLstn == 0) { +- errmsg.LogError(0, NO_ERRCODE, "imfile: no file monitors could be started, " +- "input not activated.\n"); +- ABORT_FINALIZE(RS_RET_NO_RUN); ++ if(Debug) { ++ fs_node_print(runModConf->conf_tree, 0); + } ++ + finalize_it: + ENDactivateCnf + +@@ -1237,14 +1882,20 @@ ENDactivateCnf + BEGINfreeCnf + instanceConf_t *inst, *del; + CODESTARTfreeCnf ++ fs_node_destroy(pModConf->conf_tree); ++ //move_list_destruct(pModConf); + for(inst = pModConf->root ; inst != NULL ; ) { ++ if(inst->startRegex != NULL) ++ regfree(&inst->end_preg); + free(inst->pszBindRuleset); + free(inst->pszFileName); +- free(inst->pszDirName); +- free(inst->pszFileBaseName); + free(inst->pszTag); + free(inst->pszStateFile); +- free(inst->startRegex); ++ free(inst->pszFileName_forOldStateFile); ++ if(inst->startRegex != NULL) { ++ regfree(&inst->end_preg); ++ free(inst->startRegex); ++ } + del = inst; + inst = inst->next; + free(del); +@@ -1252,45 +1903,25 @@ CODESTARTfreeCnf + ENDfreeCnf + + +-/* Monitor files in traditional polling mode. +- * +- * We go through all files and remember if at least one had data. If so, we do +- * another run (until no data was present in any file). Then we sleep for +- * PollInterval seconds and restart the whole process. This ensures that as +- * long as there is some data present, it will be processed at the fastest +- * possible pace - probably important for busy systmes. If we monitor just a +- * single file, the algorithm is slightly modified. In that case, the sleep +- * hapens immediately. The idea here is that if we have just one file, we +- * returned from the file processer because that file had no additional data. +- * So even if we found some lines, it is highly unlikely to find a new one +- * just now. Trying it would result in a performance-costly additional try +- * which in the very, very vast majority of cases will never find any new +- * lines. +- * On spamming the main queue: keep in mind that it will automatically rate-limit +- * ourselfes if we begin to overrun it. So we really do not need to care here. +- */ ++/* Monitor files in polling mode. */ + static rsRetVal + doPolling(void) + { +- int bHadFileData; /* were there at least one file with data during this run? */ + DEFiRet; + while(glbl.GetGlobalInputTermState() == 0) { ++ DBGPRINTF("doPolling: new poll run\n"); + do { +- lstn_t *pLstn; +- bHadFileData = 0; +- for(pLstn = runModConf->pRootLstn ; pLstn != NULL ; pLstn = pLstn->next) { +- if(glbl.GetGlobalInputTermState() == 1) +- break; /* terminate input! */ +- pollFile(pLstn, &bHadFileData); +- } +- } while(bHadFileData == 1 && glbl.GetGlobalInputTermState() == 0); +- /* warning: do...while()! */ ++ runModConf->bHadFileData = 0; ++ fs_node_walk(runModConf->conf_tree, poll_tree); ++ DBGPRINTF("doPolling: end poll walk, hadData %d\n", runModConf->bHadFileData); ++ } while(runModConf->bHadFileData); /* warning: do...while()! */ + + /* Note: the additional 10ns wait is vitally important. It guards rsyslog + * against totally hogging the CPU if the users selects a polling interval + * of 0 seconds. It doesn't hurt any other valid scenario. So do not remove. + * rgerhards, 2008-02-14 + */ ++ DBGPRINTF("doPolling: poll going to sleep\n"); + if(glbl.GetGlobalInputTermState() == 0) + srSleep(runModConf->iPollInterval, 10); + } +@@ -1298,631 +1929,122 @@ doPolling(void) + RETiRet; + } + ++#if defined(HAVE_INOTIFY_INIT) + +-#ifdef HAVE_INOTIFY_INIT +-static rsRetVal +-fileTableInit(fileTable_t *const __restrict__ tab, const int nelem) +-{ +- DEFiRet; +- CHKmalloc(tab->listeners = malloc(sizeof(dirInfoFiles_t) * nelem)); +- tab->allocMax = nelem; +- tab->currMax = 0; +-finalize_it: +- RETiRet; +-} +-/* uncomment if needed + static void +-fileTableDisplay(fileTable_t *tab) ++in_dbg_showEv(const struct inotify_event *ev) + { +- int f; +- uchar *baseName; +- DBGPRINTF("imfile: dirs.currMaxfiles %d\n", tab->currMax); +- for(f = 0 ; f < tab->currMax ; ++f) { +- baseName = tab->listeners[f].pLstn->pszBaseName; +- DBGPRINTF("imfile: TABLE %p CONTENTS, %d->%p:'%s'\n", tab, f, tab->listeners[f].pLstn, (char*)baseName); +- } +-} +-*/ +- +-static int +-fileTableSearch(fileTable_t *const __restrict__ tab, uchar *const __restrict__ fn) +-{ +- int f; +- uchar *baseName = NULL; +- /* UNCOMMENT FOR DEBUG fileTableDisplay(tab); */ +- for(f = 0 ; f < tab->currMax ; ++f) { +- baseName = tab->listeners[f].pLstn->pszBaseName; +- if(!fnmatch((char*)baseName, (char*)fn, FNM_PATHNAME | FNM_PERIOD)) +- break; /* found */ +- } +- if(f == tab->currMax) +- f = -1; +- DBGPRINTF("imfile: fileTableSearch file '%s' - '%s', found:%d\n", fn, baseName, f); +- return f; +-} +- +-static int +-fileTableSearchNoWildcard(fileTable_t *const __restrict__ tab, uchar *const __restrict__ fn) +-{ +- int f; +- uchar *baseName = NULL; +- /* UNCOMMENT FOR DEBUG fileTableDisplay(tab); */ +- for(f = 0 ; f < tab->currMax ; ++f) { +- baseName = tab->listeners[f].pLstn->pszBaseName; +- if (strcmp((const char*)baseName, (const char*)fn) == 0) +- break; /* found */ +- } +- if(f == tab->currMax) +- f = -1; +- DBGPRINTF("imfile: fileTableSearchNoWildcard file '%s' - '%s', found:%d\n", fn, baseName, f); +- return f; +-} +- +-/* add file to file table */ +-static rsRetVal +-fileTableAddFile(fileTable_t *const __restrict__ tab, lstn_t *const __restrict__ pLstn) +-{ +- int j; +- DEFiRet; +- /* UNCOMMENT FOR DEBUG fileTableDisplay(tab); */ +- for(j = 0 ; j < tab->currMax && tab->listeners[j].pLstn != pLstn ; ++j) +- ; /* just scan */ +- if(j < tab->currMax) { +- ++tab->listeners[j].refcnt; +- DBGPRINTF("imfile: file '%s' already registered, refcnt now %d\n", +- pLstn->pszFileName, tab->listeners[j].refcnt); +- FINALIZE; ++ if(ev->mask & IN_IGNORED) { ++ dbgprintf("INOTIFY event: watch was REMOVED\n"); + } +- +- if(tab->currMax == tab->allocMax) { +- const int newMax = 2 * tab->allocMax; +- dirInfoFiles_t *newListenerTab = realloc(tab->listeners, newMax * sizeof(dirInfoFiles_t)); +- if(newListenerTab == NULL) { +- errmsg.LogError(0, RS_RET_OUT_OF_MEMORY, +- "cannot alloc memory to map directory/file relationship " +- "for '%s' - ignoring", pLstn->pszFileName); +- ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); +- } +- tab->listeners = newListenerTab; +- tab->allocMax = newMax; +- DBGPRINTF("imfile: increased dir table to %d entries\n", allocMaxDirs); ++ if(ev->mask & IN_MODIFY) { ++ dbgprintf("INOTIFY event: watch was MODIFID\n"); + } +- +- tab->listeners[tab->currMax].pLstn = pLstn; +- tab->listeners[tab->currMax].refcnt = 1; +- tab->currMax++; +-finalize_it: +- RETiRet; +-} +- +-/* delete a file from file table */ +-static rsRetVal +-fileTableDelFile(fileTable_t *const __restrict__ tab, lstn_t *const __restrict__ pLstn) +-{ +- int j; +- DEFiRet; +- +- for(j = 0 ; j < tab->currMax && tab->listeners[j].pLstn != pLstn ; ++j) +- ; /* just scan */ +- if(j == tab->currMax) { +- DBGPRINTF("imfile: no association for file '%s'\n", pLstn->pszFileName); +- FINALIZE; ++ if(ev->mask & IN_ACCESS) { ++ dbgprintf("INOTIFY event: watch IN_ACCESS\n"); + } +- tab->listeners[j].refcnt--; +- if(tab->listeners[j].refcnt == 0) { +- /* we remove that entry (but we never shrink the table) */ +- if(j < tab->currMax - 1) { +- /* entry in middle - need to move others */ +- memmove(tab->listeners+j, tab->listeners+j+1, +- (tab->currMax -j-1) * sizeof(dirInfoFiles_t)); +- } +- --tab->currMax; ++ if(ev->mask & IN_ATTRIB) { ++ dbgprintf("INOTIFY event: watch IN_ATTRIB\n"); + } +-finalize_it: +- RETiRet; +-} +-/* add entry to dirs array */ +-static rsRetVal +-dirsAdd(uchar *dirName) +-{ +- int newMax; +- dirInfo_t *newDirTab; +- DEFiRet; +- +- if(currMaxDirs == allocMaxDirs) { +- newMax = 2 * allocMaxDirs; +- newDirTab = realloc(dirs, newMax * sizeof(dirInfo_t)); +- if(newDirTab == NULL) { +- errmsg.LogError(0, RS_RET_OUT_OF_MEMORY, +- "cannot alloc memory to monitor directory '%s' - ignoring", +- dirName); +- ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); +- } +- dirs = newDirTab; +- allocMaxDirs = newMax; +- DBGPRINTF("imfile: increased dir table to %d entries\n", allocMaxDirs); ++ if(ev->mask & IN_CLOSE_WRITE) { ++ dbgprintf("INOTIFY event: watch IN_CLOSE_WRITE\n"); + } +- +- /* if we reach this point, there is space in the file table for the new entry */ +- dirs[currMaxDirs].dirName = dirName; +- CHKiRet(fileTableInit(&dirs[currMaxDirs].active, INIT_FILE_IN_DIR_TAB_SIZE)); +- CHKiRet(fileTableInit(&dirs[currMaxDirs].configured, INIT_FILE_IN_DIR_TAB_SIZE)); +- +- ++currMaxDirs; +- DBGPRINTF("imfile: added to dirs table: '%s'\n", dirName); +-finalize_it: +- RETiRet; +-} +- +- +-/* checks if a dir name is already inside the dirs array. If so, returns +- * its index. If not present, -1 is returned. +- */ +-static int +-dirsFindDir(uchar *dir) +-{ +- int i; +- +- for(i = 0 ; i < currMaxDirs && ustrcmp(dir, dirs[i].dirName) ; ++i) +- ; /* just scan, all done in for() */ +- if(i == currMaxDirs) +- i = -1; +- return i; +-} +- +-static rsRetVal +-dirsInit(void) +-{ +- instanceConf_t *inst; +- DEFiRet; +- +- free(dirs); +- CHKmalloc(dirs = malloc(sizeof(dirInfo_t) * INIT_FILE_TAB_SIZE)); +- allocMaxDirs = INIT_FILE_TAB_SIZE; +- currMaxDirs = 0; +- +- for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { +- if(dirsFindDir(inst->pszDirName) == -1) +- dirsAdd(inst->pszDirName); ++ if(ev->mask & IN_CLOSE_NOWRITE) { ++ dbgprintf("INOTIFY event: watch IN_CLOSE_NOWRITE\n"); + } +- +-finalize_it: +- RETiRet; +-} +- +-/* add file to directory (create association) +- * fIdx is index into file table, all other information is pulled from that table. +- * bActive is 1 if the file is to be added to active set, else zero +- */ +-static rsRetVal +-dirsAddFile(lstn_t *__restrict__ pLstn, const int bActive) +-{ +- int dirIdx; +- dirInfo_t *dir; +- DEFiRet; +- +- dirIdx = dirsFindDir(pLstn->pszDirName); +- if(dirIdx == -1) { +- errmsg.LogError(0, RS_RET_INTERNAL_ERROR, "imfile: could not find " +- "directory '%s' in dirs array - ignoring", +- pLstn->pszDirName); +- FINALIZE; ++ if(ev->mask & IN_CREATE) { ++ dbgprintf("INOTIFY event: file was CREATED: %s\n", ev->name); + } +- +- dir = dirs + dirIdx; +- CHKiRet(fileTableAddFile((bActive ? &dir->active : &dir->configured), pLstn)); +- DBGPRINTF("imfile: associated file [%s] to directory %d[%s], Active = %d\n", +- pLstn->pszFileName, dirIdx, dir->dirName, bActive); +- /* UNCOMMENT FOR DEBUG fileTableDisplay(bActive ? &dir->active : &dir->configured); */ +-finalize_it: +- RETiRet; +-} +- +- +-static void +-in_setupDirWatch(const int dirIdx) +-{ +- int wd; +- wd = inotify_add_watch(ino_fd, (char*)dirs[dirIdx].dirName, IN_CREATE|IN_DELETE|IN_MOVED_FROM); +- if(wd < 0) { +- DBGPRINTF("imfile: could not create dir watch for '%s'\n", +- dirs[dirIdx].dirName); +- goto done; ++ if(ev->mask & IN_DELETE) { ++ dbgprintf("INOTIFY event: watch IN_DELETE\n"); + } +- wdmapAdd(wd, dirIdx, NULL); +- DBGPRINTF("imfile: watch %d added for dir %s\n", wd, dirs[dirIdx].dirName); +-done: return; +-} +- +-/* Setup a new file watch for a known active file. It must already have +- * been entered into the correct tables. +- * Note: we need to try to read this file, as it may already contain data this +- * needs to be processed, and we won't get an event for that as notifications +- * happen only for things after the watch has been activated. +- * Note: newFileName is NULL for configured files, and non-NULL for dynamically +- * detected files (e.g. wildcards!) +- */ +-static void +-startLstnFile(lstn_t *const __restrict__ pLstn) +-{ +- rsRetVal localRet; +- const int wd = inotify_add_watch(ino_fd, (char*)pLstn->pszFileName, IN_MODIFY); +- if(wd < 0) { +- char errStr[512]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- DBGPRINTF("imfile: could not create file table entry for '%s' - " +- "not processing it now: %s\n", +- pLstn->pszFileName, errStr); +- goto done; ++ if(ev->mask & IN_DELETE_SELF) { ++ dbgprintf("INOTIFY event: watch IN_DELETE_SELF\n"); + } +- if((localRet = wdmapAdd(wd, -1, pLstn)) != RS_RET_OK) { +- DBGPRINTF("imfile: error %d adding file to wdmap, ignoring\n", localRet); +- goto done; ++ if(ev->mask & IN_MOVE_SELF) { ++ dbgprintf("INOTIFY event: watch IN_MOVE_SELF\n"); + } +- DBGPRINTF("imfile: watch %d added for file %s\n", wd, pLstn->pszFileName); +- dirsAddFile(pLstn, ACTIVE_FILE); +- pollFile(pLstn, NULL); +-done: return; +-} +- +-/* Duplicate an existing listener. This is called when a new file is to +- * be monitored due to wildcard detection. Returns the new pLstn in +- * the ppExisting parameter. +- */ +-static rsRetVal +-lstnDup(lstn_t **ppExisting, uchar *const __restrict__ newname) +-{ +- DEFiRet; +- lstn_t *const existing = *ppExisting; +- lstn_t *pThis; +- +- CHKiRet(lstnAdd(&pThis)); +- pThis->pszDirName = existing->pszDirName; /* read-only */ +- pThis->pszBaseName = (uchar*)strdup((char*)newname); +- if(asprintf((char**)&pThis->pszFileName, "%s/%s", (char*)pThis->pszDirName, (char*)newname) == -1) { +- DBGPRINTF("imfile/lstnDup: asprintf failed, malfunction can happen\n"); +- ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); +- } +- pThis->pszTag = (uchar*) strdup((char*) existing->pszTag); +- pThis->lenTag = ustrlen(pThis->pszTag); +- pThis->pszStateFile = existing->pszStateFile == NULL ? NULL : (uchar*) strdup((char*) existing->pszStateFile); +- +- CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)pThis->pszFileName)); +- pThis->multiSub.maxElem = existing->multiSub.maxElem; +- pThis->multiSub.nElem = 0; +- CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(pThis->multiSub.maxElem * sizeof(smsg_t*))); +- pThis->iSeverity = existing->iSeverity; +- pThis->iFacility = existing->iFacility; +- pThis->maxLinesAtOnce = existing->maxLinesAtOnce; +- pThis->trimLineOverBytes = existing->trimLineOverBytes; +- pThis->iPersistStateInterval = existing->iPersistStateInterval; +- pThis->readMode = existing->readMode; +- pThis->startRegex = existing->startRegex; /* no strdup, as it is read-only */ +- if(pThis->startRegex != NULL) // TODO: make this a single function with better error handling +- if(regcomp(&pThis->end_preg, (char*)pThis->startRegex, REG_EXTENDED)) { +- DBGPRINTF("imfile: error regex compile\n"); +- ABORT_FINALIZE(RS_RET_ERR); +- } +- pThis->bRMStateOnDel = existing->bRMStateOnDel; +- pThis->hasWildcard = existing->hasWildcard; +- pThis->escapeLF = existing->escapeLF; +- pThis->reopenOnTruncate = existing->reopenOnTruncate; +- pThis->addMetadata = existing->addMetadata; +- pThis->addCeeTag = existing->addCeeTag; +- pThis->readTimeout = existing->readTimeout; +- pThis->freshStartTail = existing->freshStartTail; +- pThis->pRuleset = existing->pRuleset; +- pThis->nRecords = 0; +- pThis->pStrm = NULL; +- pThis->prevLineSegment = NULL; +- pThis->masterLstn = existing; +- *ppExisting = pThis; +-finalize_it: +- RETiRet; +-} +- +-/* Setup a new file watch for dynamically discovered files (via wildcards). +- * Note: we need to try to read this file, as it may already contain data this +- * needs to be processed, and we won't get an event for that as notifications +- * happen only for things after the watch has been activated. +- */ +-static void +-in_setupFileWatchDynamic(lstn_t *pLstn, uchar *const __restrict__ newBaseName) +-{ +- char fullfn[MAXFNAME]; +- struct stat fileInfo; +- snprintf(fullfn, MAXFNAME, "%s/%s", pLstn->pszDirName, newBaseName); +- if(stat(fullfn, &fileInfo) != 0) { +- char errStr[1024]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- DBGPRINTF("imfile: ignoring file '%s' cannot stat(): %s\n", +- fullfn, errStr); +- goto done; ++ if(ev->mask & IN_MOVED_FROM) { ++ dbgprintf("INOTIFY event: watch IN_MOVED_FROM, cookie %u, name '%s'\n", ev->cookie, ev->name); + } +- +- if(S_ISDIR(fileInfo.st_mode)) { +- DBGPRINTF("imfile: ignoring directory '%s'\n", fullfn); +- goto done; ++ if(ev->mask & IN_MOVED_TO) { ++ dbgprintf("INOTIFY event: watch IN_MOVED_TO, cookie %u, name '%s'\n", ev->cookie, ev->name); + } +- +- if(lstnDup(&pLstn, newBaseName) != RS_RET_OK) +- goto done; +- +- startLstnFile(pLstn); +-done: return; +-} +- +-/* Setup a new file watch for static (configured) files. +- * Note: we need to try to read this file, as it may already contain data this +- * needs to be processed, and we won't get an event for that as notifications +- * happen only for things after the watch has been activated. +- */ +-static void +-in_setupFileWatchStatic(lstn_t *pLstn) +-{ +- DBGPRINTF("imfile: adding file '%s' to configured table\n", +- pLstn->pszFileName); +- dirsAddFile(pLstn, CONFIGURED_FILE); +- +- if(pLstn->hasWildcard) { +- DBGPRINTF("imfile: file '%s' has wildcard, doing initial " +- "expansion\n", pLstn->pszFileName); +- glob_t files; +- const int ret = glob((char*)pLstn->pszFileName, +- GLOB_MARK|GLOB_NOSORT|GLOB_BRACE, NULL, &files); +- if(ret == 0) { +- for(unsigned i = 0 ; i < files.gl_pathc ; i++) { +- uchar basen[MAXFNAME]; +- uchar *const file = (uchar*)files.gl_pathv[i]; +- if(file[strlen((char*)file)-1] == '/') +- continue;/* we cannot process subdirs! */ +- getBasename(basen, file); +- in_setupFileWatchDynamic(pLstn, basen); +- } +- globfree(&files); +- } +- } else { +- /* Duplicate static object as well, otherwise the configobject could be deleted later! */ +- if(lstnDup(&pLstn, pLstn->pszBaseName) != RS_RET_OK) { +- DBGPRINTF("imfile: in_setupFileWatchStatic failed to duplicate listener for '%s'\n", pLstn->pszFileName); +- goto done; +- } +- startLstnFile(pLstn); ++ if(ev->mask & IN_OPEN) { ++ dbgprintf("INOTIFY event: watch IN_OPEN\n"); + } +-done: return; +-} +- +-/* setup our initial set of watches, based on user config */ +-static void +-in_setupInitialWatches(void) +-{ +- int i; +- for(i = 0 ; i < currMaxDirs ; ++i) { +- in_setupDirWatch(i); +- } +- lstn_t *pLstn; +- for(pLstn = runModConf->pRootLstn ; pLstn != NULL ; pLstn = pLstn->next) { +- if(pLstn->masterLstn == NULL) { +- /* we process only static (master) entries */ +- in_setupFileWatchStatic(pLstn); +- } ++ if(ev->mask & IN_ISDIR) { ++ dbgprintf("INOTIFY event: watch IN_ISDIR\n"); + } + } + +-static void +-in_dbg_showEv(struct inotify_event *ev) +-{ +- if(ev->mask & IN_IGNORED) { +- DBGPRINTF("INOTIFY event: watch was REMOVED\n"); +- } else if(ev->mask & IN_MODIFY) { +- DBGPRINTF("INOTIFY event: watch was MODIFID\n"); +- } else if(ev->mask & IN_ACCESS) { +- DBGPRINTF("INOTIFY event: watch IN_ACCESS\n"); +- } else if(ev->mask & IN_ATTRIB) { +- DBGPRINTF("INOTIFY event: watch IN_ATTRIB\n"); +- } else if(ev->mask & IN_CLOSE_WRITE) { +- DBGPRINTF("INOTIFY event: watch IN_CLOSE_WRITE\n"); +- } else if(ev->mask & IN_CLOSE_NOWRITE) { +- DBGPRINTF("INOTIFY event: watch IN_CLOSE_NOWRITE\n"); +- } else if(ev->mask & IN_CREATE) { +- DBGPRINTF("INOTIFY event: file was CREATED: %s\n", ev->name); +- } else if(ev->mask & IN_DELETE) { +- DBGPRINTF("INOTIFY event: watch IN_DELETE\n"); +- } else if(ev->mask & IN_DELETE_SELF) { +- DBGPRINTF("INOTIFY event: watch IN_DELETE_SELF\n"); +- } else if(ev->mask & IN_MOVE_SELF) { +- DBGPRINTF("INOTIFY event: watch IN_MOVE_SELF\n"); +- } else if(ev->mask & IN_MOVED_FROM) { +- DBGPRINTF("INOTIFY event: watch IN_MOVED_FROM\n"); +- } else if(ev->mask & IN_MOVED_TO) { +- DBGPRINTF("INOTIFY event: watch IN_MOVED_TO\n"); +- } else if(ev->mask & IN_OPEN) { +- DBGPRINTF("INOTIFY event: watch IN_OPEN\n"); +- } else if(ev->mask & IN_ISDIR) { +- DBGPRINTF("INOTIFY event: watch IN_ISDIR\n"); +- } else { +- DBGPRINTF("INOTIFY event: unknown mask code %8.8x\n", ev->mask); +- } +-} +- + +-/* inotify told us that a file's wd was closed. We now need to remove +- * the file from our internal structures. Remember that a different inode +- * with the same name may already be in processing. +- */ + static void +-in_removeFile(const int dirIdx, +- lstn_t *const __restrict__ pLstn) ++in_handleFileEvent(struct inotify_event *ev, const wd_map_t *const etry) + { +- uchar statefile[MAXFNAME]; +- uchar toDel[MAXFNAME]; +- int bDoRMState; +- int wd; +- uchar *statefn; +- DBGPRINTF("imfile: remove listener '%s', dirIdx %d\n", +- pLstn->pszFileName, dirIdx); +- if(pLstn->bRMStateOnDel) { +- statefn = getStateFileName(pLstn, statefile, sizeof(statefile)); +- snprintf((char*)toDel, sizeof(toDel), "%s/%s", +- glbl.GetWorkDir(), (char*)statefn); +- bDoRMState = 1; ++ if(ev->mask & IN_MODIFY) { ++ DBGPRINTF("fs_node_notify_file_update: act->name '%s'\n", etry->act->name); ++ pollFile(etry->act); + } else { +- bDoRMState = 0; +- } +- pollFile(pLstn, NULL); /* one final try to gather data */ +- /* delete listener data */ +- DBGPRINTF("imfile: DELETING listener data for '%s' - '%s'\n", pLstn->pszBaseName, pLstn->pszFileName); +- lstnDel(pLstn); +- fileTableDelFile(&dirs[dirIdx].active, pLstn); +- if(bDoRMState) { +- DBGPRINTF("imfile: unlinking '%s'\n", toDel); +- if(unlink((char*)toDel) != 0) { +- char errStr[1024]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- errmsg.LogError(0, RS_RET_ERR, "imfile: could not remove state " +- "file \"%s\": %s", toDel, errStr); +- } ++ DBGPRINTF("got non-expected inotify event:\n"); ++ in_dbg_showEv(ev); + } +- wd = wdmapLookupListner(pLstn); +- wdmapDel(wd); + } + +-static void +-in_handleDirEventCREATE(struct inotify_event *ev, const int dirIdx) +-{ +- lstn_t *pLstn; +- int ftIdx; +- ftIdx = fileTableSearch(&dirs[dirIdx].active, (uchar*)ev->name); +- if(ftIdx >= 0) { +- pLstn = dirs[dirIdx].active.listeners[ftIdx].pLstn; +- } else { +- DBGPRINTF("imfile: file '%s' not active in dir '%s'\n", +- ev->name, dirs[dirIdx].dirName); +- ftIdx = fileTableSearch(&dirs[dirIdx].configured, (uchar*)ev->name); +- if(ftIdx == -1) { +- DBGPRINTF("imfile: file '%s' not associated with dir '%s'\n", +- ev->name, dirs[dirIdx].dirName); +- goto done; +- } +- pLstn = dirs[dirIdx].configured.listeners[ftIdx].pLstn; +- } +- DBGPRINTF("imfile: file '%s' associated with dir '%s'\n", ev->name, dirs[dirIdx].dirName); +- in_setupFileWatchDynamic(pLstn, (uchar*)ev->name); +-done: return; +-} + +-/* note: we need to care only for active files in the DELETE case. +- * Two reasons: a) if this is a configured file, it should be active +- * b) if not for some reason, there still is nothing we can do against +- * it, and trying to process a *deleted* file really makes no sense +- * (remeber we don't have it open, so it actually *is gone*). ++/* workaround for IN_MOVED: walk active list and prevent state file deletion of ++ * IN_MOVED_IN active object ++ * TODO: replace by a more generic solution. + */ + static void +-in_handleDirEventDELETE(struct inotify_event *const ev, const int dirIdx) +-{ +- const int ftIdx = fileTableSearch(&dirs[dirIdx].active, (uchar*)ev->name); +- if(ftIdx == -1) { +- DBGPRINTF("imfile: deleted file '%s' not active in dir '%s'\n", +- ev->name, dirs[dirIdx].dirName); +- goto done; +- } +- DBGPRINTF("imfile: imfile delete processing for '%s'\n", +- dirs[dirIdx].active.listeners[ftIdx].pLstn->pszFileName); +- in_removeFile(dirIdx, dirs[dirIdx].active.listeners[ftIdx].pLstn); +-done: return; +-} +- +-static void +-in_handleDirEvent(struct inotify_event *const ev, const int dirIdx) ++flag_in_move(fs_edge_t *const edge, const char *name_moved) + { +- DBGPRINTF("imfile: handle dir event for %s\n", dirs[dirIdx].dirName); +- if((ev->mask & IN_CREATE)) { +- in_handleDirEventCREATE(ev, dirIdx); +- } else if((ev->mask & IN_DELETE)) { +- in_handleDirEventDELETE(ev, dirIdx); +- } else { +- DBGPRINTF("imfile: got non-expected inotify event:\n"); +- in_dbg_showEv(ev); +- } +-} ++ act_obj_t *act; + +- +-static void +-in_handleFileEvent(struct inotify_event *ev, const wd_map_t *const etry) +-{ +- if(ev->mask & IN_MODIFY) { +- pollFile(etry->pLstn, NULL); +- } else { +- DBGPRINTF("imfile: got non-expected inotify event:\n"); +- in_dbg_showEv(ev); ++ for(act = edge->active ; act != NULL ; act = act->next) { ++ DBGPRINTF("checking active object %s\n", act->basename); ++ if(!strcmp(act->basename, name_moved)){ ++ DBGPRINTF("found file\n"); ++ act->in_move = 1; ++ break; ++ } else { ++ DBGPRINTF("name check fails, '%s' != '%s'\n", act->basename, name_moved); ++ } + } + } + + static void + in_processEvent(struct inotify_event *ev) + { +- wd_map_t *etry; +- lstn_t *pLstn; +- int iRet; +- int ftIdx; +- int wd; +- + if(ev->mask & IN_IGNORED) { +- goto done; +- } else if(ev->mask & IN_MOVED_FROM) { +- /* Find wd entry and remove it */ +- etry = wdmapLookup(ev->wd); +- if(etry != NULL) { +- ftIdx = fileTableSearchNoWildcard(&dirs[etry->dirIdx].active, (uchar*)ev->name); +- DBGPRINTF("imfile: IN_MOVED_FROM Event (ftIdx=%d, name=%s)\n", ftIdx, ev->name); +- if(ftIdx >= 0) { +- /* Find listener and wd table index*/ +- pLstn = dirs[etry->dirIdx].active.listeners[ftIdx].pLstn; +- wd = wdmapLookupListner(pLstn); +- +- /* Remove file from inotify watch */ +- iRet = inotify_rm_watch(ino_fd, wd); /* Note this will TRIGGER IN_IGNORED Event! */ +- if (iRet != 0) { +- DBGPRINTF("imfile: inotify_rm_watch error %d (ftIdx=%d, wd=%d, name=%s)\n", errno, ftIdx, wd, ev->name); +- } else { +- DBGPRINTF("imfile: inotify_rm_watch successfully removed file from watch (ftIdx=%d, wd=%d, name=%s)\n", ftIdx, wd, ev->name); +- } +- in_removeFile(etry->dirIdx, pLstn); +- DBGPRINTF("imfile: IN_MOVED_FROM Event file removed file (wd=%d, name=%s)\n", wd, ev->name); +- } +- } ++ DBGPRINTF("imfile: got IN_IGNORED event\n"); + goto done; + } +- etry = wdmapLookup(ev->wd); ++ ++ DBGPRINTF("in_processEvent process Event %x for %s\n", ev->mask, ev->name); ++ const wd_map_t *const etry = wdmapLookup(ev->wd); + if(etry == NULL) { +- DBGPRINTF("imfile: could not lookup wd %d\n", ev->wd); ++ LogMsg(0, RS_RET_INTERNAL_ERROR, LOG_WARNING, "imfile: internal error? " ++ "inotify provided watch descriptor %d which we could not find " ++ "in our tables - ignored", ev->wd); + goto done; + } +- if(etry->pLstn == NULL) { /* directory? */ +- in_handleDirEvent(ev, etry->dirIdx); ++ DBGPRINTF("in_processEvent process Event %x is_file %d, act->name '%s'\n", ++ ev->mask, etry->act->edge->is_file, etry->act->name); ++ ++ if((ev->mask & IN_MOVED_FROM)) { ++ flag_in_move(etry->act->edge->node->edges, ev->name); ++ } ++ if(ev->mask & (IN_MOVED_FROM | IN_MOVED_TO)) { ++ fs_node_walk(etry->act->edge->node, poll_tree); ++ } else if(etry->act->edge->is_file && !(etry->act->is_symlink)) { ++ in_handleFileEvent(ev, etry); // esentially poll_file()! + } else { +- in_handleFileEvent(ev, etry); ++ fs_node_walk(etry->act->edge->node, poll_tree); + } + done: return; + } + +-static void +-in_do_timeout_processing(void) +-{ +- int i; +- DBGPRINTF("imfile: readTimeouts are configured, checking if some apply\n"); +- +- for(i = 0 ; i < nWdmap ; ++i) { +- dbgprintf("imfile: wdmap %d, plstn %p\n", i, wdmap[i].pLstn); +- lstn_t *const pLstn = wdmap[i].pLstn; +- if(pLstn != NULL && strmReadMultiLine_isTimedOut(pLstn->pStrm)) { +- dbgprintf("imfile: wdmap %d, timeout occured\n", i); +- pollFile(pLstn, NULL); +- } +- } +- +-} +- + + /* Monitor files in inotify mode */ + #if !defined(_AIX) +@@ -1940,14 +2062,16 @@ do_inotify(void) + DEFiRet; + + CHKiRet(wdmapInit()); +- CHKiRet(dirsInit()); + ino_fd = inotify_init(); +- if(ino_fd < 0) { +- errmsg.LogError(1, RS_RET_INOTIFY_INIT_FAILED, "imfile: Init inotify instance failed "); +- return RS_RET_INOTIFY_INIT_FAILED; +- } +- DBGPRINTF("imfile: inotify fd %d\n", ino_fd); +- in_setupInitialWatches(); ++ if(ino_fd < 0) { ++ LogError(errno, RS_RET_INOTIFY_INIT_FAILED, "imfile: Init inotify " ++ "instance failed "); ++ return RS_RET_INOTIFY_INIT_FAILED; ++ } ++ DBGPRINTF("inotify fd %d\n", ino_fd); ++ ++ /* do watch initialization */ ++ fs_node_walk(runModConf->conf_tree, poll_tree); + + while(glbl.GetGlobalInputTermState() == 0) { + if(runModConf->haveReadTimeouts) { +@@ -1959,7 +2083,8 @@ do_inotify(void) + r = poll(&pollfd, 1, runModConf->timeoutGranularity); + } while(r == -1 && errno == EINTR); + if(r == 0) { +- in_do_timeout_processing(); ++ DBGPRINTF("readTimeouts are configured, checking if some apply\n"); ++ fs_node_walk(runModConf->conf_tree, poll_timeouts); + continue; + } else if (r == -1) { + char errStr[1024]; +@@ -2035,49 +2160,96 @@ CODESTARTwillRun + CHKiRet(prop.Construct(&pInputName)); + CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("imfile"), sizeof("imfile") - 1)); + CHKiRet(prop.ConstructFinalize(pInputName)); +- + finalize_it: + ENDwillRun + ++// TODO: refactor this into a generically-usable "atomic file creation" utility for ++// all kinds of "state files" ++static rsRetVal ++atomicWriteStateFile(const char *fn, const char *content) ++{ ++ DEFiRet; ++ const int fd = open(fn, O_CLOEXEC | O_NOCTTY | O_WRONLY | O_CREAT | O_TRUNC, 0600); ++ if(fd < 0) { ++ LogError(errno, RS_RET_IO_ERROR, "imfile: cannot open state file '%s' for " ++ "persisting file state - some data will probably be duplicated " ++ "on next startup", fn); ++ ABORT_FINALIZE(RS_RET_IO_ERROR); ++ } ++ ++ const size_t toWrite = strlen(content); ++ const ssize_t w = write(fd, content, toWrite); ++ if(w != (ssize_t) toWrite) { ++ LogError(errno, RS_RET_IO_ERROR, "imfile: partial write to state file '%s' " ++ "this may cause trouble in the future. We will try to delete the " ++ "state file, as this provides most consistent state", fn); ++ unlink(fn); ++ ABORT_FINALIZE(RS_RET_IO_ERROR); ++ } ++ ++finalize_it: ++ if(fd >= 0) { ++ close(fd); ++ } ++ RETiRet; ++} ++ ++ + /* This function persists information for a specific file being monitored. + * To do so, it simply persists the stream object. We do NOT abort on error + * iRet as that makes matters worse (at least we can try persisting the others...). + * rgerhards, 2008-02-13 + */ + static rsRetVal +-persistStrmState(lstn_t *pLstn) ++persistStrmState(act_obj_t *const act) + { + DEFiRet; +- strm_t *psSF = NULL; /* state file (stream) */ +- size_t lenDir; + uchar statefile[MAXFNAME]; ++ uchar statefname[MAXFNAME]; ++ ++ uchar *const statefn = getStateFileName(act, statefile, sizeof(statefile)); ++ getFullStateFileName(statefn, statefname, sizeof(statefname)); ++ DBGPRINTF("persisting state for '%s', state file '%s'\n", act->name, statefname); ++ ++ struct json_object *jval = NULL; ++ struct json_object *json = NULL; ++ CHKmalloc(json = json_object_new_object()); ++ jval = json_object_new_string((char*) act->name); ++ json_object_object_add(json, "filename", jval); ++ jval = json_object_new_int(strmGetPrevWasNL(act->pStrm)); ++ json_object_object_add(json, "prev_was_nl", jval); ++ ++ /* we access some data items a bit dirty, as we need to refactor the whole ++ * thing in any case - TODO ++ */ ++ jval = json_object_new_int64(act->pStrm->iCurrOffs); ++ json_object_object_add(json, "curr_offs", jval); ++ jval = json_object_new_int64(act->pStrm->strtOffs); ++ json_object_object_add(json, "strt_offs", jval); + +- uchar *const statefn = getStateFileName(pLstn, statefile, sizeof(statefile)); +- DBGPRINTF("imfile: persisting state for '%s' to file '%s'\n", +- pLstn->pszFileName, statefn); +- CHKiRet(strm.Construct(&psSF)); +- lenDir = ustrlen(glbl.GetWorkDir()); +- if(lenDir > 0) +- CHKiRet(strm.SetDir(psSF, glbl.GetWorkDir(), lenDir)); +- CHKiRet(strm.SettOperationsMode(psSF, STREAMMODE_WRITE_TRUNC)); +- CHKiRet(strm.SetsType(psSF, STREAMTYPE_FILE_SINGLE)); +- CHKiRet(strm.SetFName(psSF, statefn, strlen((char*) statefn))); +- CHKiRet(strm.ConstructFinalize(psSF)); ++ const uchar *const prevLineSegment = strmGetPrevLineSegment(act->pStrm); ++ if(prevLineSegment != NULL) { ++ jval = json_object_new_string((const char*) prevLineSegment); ++ json_object_object_add(json, "prev_line_segment", jval); ++ } + +- CHKiRet(strm.Serialize(pLstn->pStrm, psSF)); +- CHKiRet(strm.Flush(psSF)); ++ const uchar *const prevMsgSegment = strmGetPrevMsgSegment(act->pStrm); ++ if(prevMsgSegment != NULL) { ++ jval = json_object_new_string((const char*) prevMsgSegment); ++ json_object_object_add(json, "prev_msg_segment", jval); ++ } + +- CHKiRet(strm.Destruct(&psSF)); ++ const char *jstr = json_object_to_json_string_ext(json, JSON_C_TO_STRING_SPACED); + +-finalize_it: +- if(psSF != NULL) +- strm.Destruct(&psSF); ++ CHKiRet(atomicWriteStateFile((const char*)statefname, jstr)); ++ json_object_put(json); + ++finalize_it: + if(iRet != RS_RET_OK) { + errmsg.LogError(0, iRet, "imfile: could not persist state " + "file %s - data may be repeated on next " + "startup. Is WorkDirectory set?", +- statefn); ++ statefname); + } + + RETiRet; +@@ -2089,11 +2261,6 @@ finalize_it: + */ + BEGINafterRun + CODESTARTafterRun +- while(runModConf->pRootLstn != NULL) { +- /* Note: lstnDel() reasociates root! */ +- lstnDel(runModConf->pRootLstn); +- } +- + if(pInputName != NULL) + prop.Destruct(&pInputName); + ENDafterRun +@@ -2118,12 +2285,6 @@ CODESTARTmodExit + objRelease(prop, CORE_COMPONENT); + objRelease(ruleset, CORE_COMPONENT); + #ifdef HAVE_INOTIFY_INIT +- /* we use these vars only in inotify mode */ +- if(dirs != NULL) { +- free(dirs->active.listeners); +- free(dirs->configured.listeners); +- free(dirs); +- } + free(wdmap); + #endif + ENDmodExit +diff --git a/runtime/msg.c b/runtime/msg.c +index a885d2368bbaeea90a6e92dc0d569d169b1dd2e5..f45d6175283097974023905fc072508a18a8270a 100644 +--- a/runtime/msg.c ++++ b/runtime/msg.c +@@ -4890,6 +4890,28 @@ finalize_it: + RETiRet; + } + ++rsRetVal ++msgAddMultiMetadata(smsg_t *const __restrict__ pMsg, ++ const uchar ** __restrict__ metaname, ++ const uchar ** __restrict__ metaval, ++ const int count) ++{ ++ DEFiRet; ++ int i = 0 ; ++ struct json_object *const json = json_object_new_object(); ++ CHKmalloc(json); ++ for ( i = 0 ; i < count ; i++ ) { ++ struct json_object *const jval = json_object_new_string((char*)metaval[i]); ++ if(jval == NULL) { ++ json_object_put(json); ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ } ++ json_object_object_add(json, (const char *const)metaname[i], jval); ++ } ++ iRet = msgAddJSON(pMsg, (uchar*)"!metadata", json, 0, 0); ++finalize_it: ++ RETiRet; ++} + + static struct json_object * + jsonDeepCopy(struct json_object *src) +diff --git a/runtime/msg.h b/runtime/msg.h +index 6521e19b28b013f0d06e357bdb0f33a94dab638b..0e92da43398156f4871b2e567a242cb089f67a08 100644 +--- a/runtime/msg.h ++++ b/runtime/msg.h +@@ -195,6 +195,7 @@ int getPRIi(const smsg_t * const pM); + void getRawMsg(smsg_t *pM, uchar **pBuf, int *piLen); + rsRetVal msgAddJSON(smsg_t *pM, uchar *name, struct json_object *json, int force_reset, int sharedReference); + rsRetVal msgAddMetadata(smsg_t *msg, uchar *metaname, uchar *metaval); ++rsRetVal msgAddMultiMetadata(smsg_t *msg, const uchar **metaname, const uchar **metaval, const int count); + rsRetVal MsgGetSeverity(smsg_t *pThis, int *piSeverity); + rsRetVal MsgDeserialize(smsg_t *pMsg, strm_t *pStrm); + rsRetVal MsgSetPropsViaJSON(smsg_t *__restrict__ const pMsg, const uchar *__restrict__ const json); +diff --git a/runtime/stream.c b/runtime/stream.c +index 701144c0e39d6fbcf9dd63fe60421e1dcd6f01c6..fb1ff11d1890bbaee107658dd3568c2bc67c223d 100644 +--- a/runtime/stream.c ++++ b/runtime/stream.c +@@ -91,6 +91,41 @@ static rsRetVal strmSeekCurrOffs(strm_t *pThis); + + /* methods */ + ++/* note: this may return NULL if not line segment is currently set */ ++// TODO: due to the cstrFinalize() this is not totally clean, albeit for our ++// current use case it does not hurt -- refactor! rgerhards, 2018-03-27 ++const uchar * ++strmGetPrevLineSegment(strm_t *const pThis) ++{ ++ const uchar *ret = NULL; ++ if(pThis->prevLineSegment != NULL) { ++ cstrFinalize(pThis->prevLineSegment); ++ ret = rsCStrGetSzStrNoNULL(pThis->prevLineSegment); ++ } ++ return ret; ++} ++/* note: this may return NULL if not line segment is currently set */ ++// TODO: due to the cstrFinalize() this is not totally clean, albeit for our ++// current use case it does not hurt -- refactor! rgerhards, 2018-03-27 ++const uchar * ++strmGetPrevMsgSegment(strm_t *const pThis) ++{ ++ const uchar *ret = NULL; ++ if(pThis->prevMsgSegment != NULL) { ++ cstrFinalize(pThis->prevMsgSegment); ++ ret = rsCStrGetSzStrNoNULL(pThis->prevMsgSegment); ++ } ++ return ret; ++} ++ ++ ++int ++strmGetPrevWasNL(const strm_t *const pThis) ++{ ++ return pThis->bPrevWasNL; ++} ++ ++ + /* output (current) file name for debug log purposes. Falls back to various + * levels of impreciseness if more precise name is not known. + */ +@@ -242,17 +277,18 @@ doPhysOpen(strm_t *pThis) + } + + pThis->fd = open((char*)pThis->pszCurrFName, iFlags | O_LARGEFILE, pThis->tOpenMode); ++ const int errno_save = errno; /* dbgprintf can mangle it! */ + DBGPRINTF("file '%s' opened as #%d with mode %d\n", pThis->pszCurrFName, + pThis->fd, (int) pThis->tOpenMode); + if(pThis->fd == -1) { +- char errStr[1024]; +- int err = errno; +- rs_strerror_r(err, errStr, sizeof(errStr)); +- DBGOPRINT((obj_t*) pThis, "open error %d, file '%s': %s\n", errno, pThis->pszCurrFName, errStr); +- if(err == ENOENT) +- ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); +- else +- ABORT_FINALIZE(RS_RET_FILE_OPEN_ERROR); ++ const rsRetVal errcode = (errno_save == ENOENT) ++ ? RS_RET_FILE_NOT_FOUND : RS_RET_FILE_OPEN_ERROR; ++ if(pThis->fileNotFoundError) { ++ LogError(errno_save, errcode, "file '%s': open error", pThis->pszCurrFName); ++ } else { ++ DBGPRINTF("file '%s': open error", pThis->pszCurrFName); ++ } ++ ABORT_FINALIZE(errcode); + } + + if(pThis->tOperationsMode == STREAMMODE_READ) { +@@ -344,6 +380,8 @@ static rsRetVal strmOpenFile(strm_t *pThis) + + if(pThis->fd != -1) + ABORT_FINALIZE(RS_RET_OK); ++ ++ free(pThis->pszCurrFName); + pThis->pszCurrFName = NULL; /* used to prevent mem leak in case of error */ + + if(pThis->pszFName == NULL) +@@ -733,11 +771,11 @@ static rsRetVal strmUnreadChar(strm_t *pThis, uchar c) + * a line, but following lines that are indented are part of the same log entry + */ + static rsRetVal +-strmReadLine(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF, uint32_t trimLineOverBytes) ++strmReadLine(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF, ++ uint32_t trimLineOverBytes, int64 *const strtOffs) + { + uchar c; + uchar finished; +- rsRetVal readCharRet; + DEFiRet; + + ASSERT(pThis != NULL); +@@ -756,12 +794,7 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF, uint + if(mode == 0) { + while(c != '\n') { + CHKiRet(cstrAppendChar(*ppCStr, c)); +- readCharRet = strmReadChar(pThis, &c); +- if((readCharRet == RS_RET_TIMED_OUT) || +- (readCharRet == RS_RET_EOF) ) { /* end reached without \n? */ +- CHKiRet(rsCStrConstructFromCStr(&pThis->prevLineSegment, *ppCStr)); +- } +- CHKiRet(readCharRet); ++ CHKiRet(strmReadChar(pThis, &c)); + } + if (trimLineOverBytes > 0 && (uint32_t) cstrLen(*ppCStr) > trimLineOverBytes) { + /* Truncate long line at trimLineOverBytes position */ +@@ -850,12 +883,19 @@ strmReadLine(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF, uint + } + + finalize_it: +- if(iRet != RS_RET_OK && *ppCStr != NULL) { +- if(cstrLen(*ppCStr) > 0) { +- /* we may have an empty string in an unsuccsfull poll or after restart! */ +- rsCStrConstructFromCStr(&pThis->prevLineSegment, *ppCStr); ++ if(iRet == RS_RET_OK) { ++ if(strtOffs != NULL) { ++ *strtOffs = pThis->strtOffs; ++ } ++ pThis->strtOffs = pThis->iCurrOffs; /* we are at begin of next line */ ++ } else { ++ if(*ppCStr != NULL) { ++ if(cstrLen(*ppCStr) > 0) { ++ /* we may have an empty string in an unsuccesfull poll or after restart! */ ++ rsCStrConstructFromCStr(&pThis->prevLineSegment, *ppCStr); ++ } ++ cstrDestruct(ppCStr); + } +- cstrDestruct(ppCStr); + } + + RETiRet; +@@ -882,7 +922,8 @@ strmReadMultiLine_isTimedOut(const strm_t *const __restrict__ pThis) + * added 2015-05-12 rgerhards + */ + rsRetVal +-strmReadMultiLine(strm_t *pThis, cstr_t **ppCStr, regex_t *preg, const sbool bEscapeLF) ++strmReadMultiLine(strm_t *pThis, cstr_t **ppCStr, regex_t *preg, const sbool bEscapeLF, ++ int64 *const strtOffs) + { + uchar c; + uchar finished = 0; +@@ -946,16 +987,24 @@ strmReadMultiLine(strm_t *pThis, cstr_t **ppCStr, regex_t *preg, const sbool bEs + } while(finished == 0); + + finalize_it: +- if( pThis->readTimeout +- && (iRet != RS_RET_OK) +- && (pThis->prevMsgSegment != NULL) +- && (tCurr > pThis->lastRead + pThis->readTimeout)) { +- CHKiRet(rsCStrConstructFromCStr(ppCStr, pThis->prevMsgSegment)); +- cstrDestruct(&pThis->prevMsgSegment); +- pThis->lastRead = tCurr; +- dbgprintf("stream: generated msg based on timeout: %s\n", cstrGetSzStrNoNULL(*ppCStr)); +- FINALIZE; +- iRet = RS_RET_OK; ++ *strtOffs = pThis->strtOffs; ++ if(thisLine != NULL) { ++ cstrDestruct(&thisLine); ++ } ++ if(iRet == RS_RET_OK) { ++ pThis->strtOffs = pThis->iCurrOffs; /* we are at begin of next line */ ++ } else { ++ if( pThis->readTimeout ++ && (pThis->prevMsgSegment != NULL) ++ && (tCurr > pThis->lastRead + pThis->readTimeout)) { ++ CHKiRet(rsCStrConstructFromCStr(ppCStr, pThis->prevMsgSegment)); ++ cstrDestruct(&pThis->prevMsgSegment); ++ pThis->lastRead = tCurr; ++ pThis->strtOffs = pThis->iCurrOffs; /* we are at begin of next line */ ++ dbgprintf("stream: generated msg based on timeout: %s\n", cstrGetSzStrNoNULL(*ppCStr)); ++ FINALIZE; ++ iRet = RS_RET_OK; ++ } + } + RETiRet; + } +@@ -974,7 +1023,10 @@ BEGINobjConstruct(strm) /* be sure to specify the object type also in END macro! + pThis->pszSizeLimitCmd = NULL; + pThis->prevLineSegment = NULL; + pThis->prevMsgSegment = NULL; ++ pThis->strtOffs = 0; ++ pThis->ignoringMsg = 0; + pThis->bPrevWasNL = 0; ++ pThis->fileNotFoundError = 1; + ENDobjConstruct(strm) + + +@@ -1686,7 +1738,7 @@ static rsRetVal strmSeek(strm_t *pThis, off64_t offs) + DBGPRINTF("strmSeek: error %lld seeking to offset %lld\n", i, (long long) offs); + ABORT_FINALIZE(RS_RET_IO_ERROR); + } +- pThis->iCurrOffs = offs; /* we are now at *this* offset */ ++ pThis->strtOffs = pThis->iCurrOffs = offs; /* we are now at *this* offset */ + pThis->iBufPtr = 0; /* buffer invalidated */ + + finalize_it: +@@ -1738,7 +1790,7 @@ strmMultiFileSeek(strm_t *pThis, unsigned int FNum, off64_t offs, off64_t *bytes + } else { + *bytesDel = 0; + } +- pThis->iCurrOffs = offs; ++ pThis->strtOffs = pThis->iCurrOffs = offs; + + finalize_it: + RETiRet; +@@ -1763,7 +1815,7 @@ static rsRetVal strmSeekCurrOffs(strm_t *pThis) + + /* As the cryprov may use CBC or similiar things, we need to read skip data */ + targetOffs = pThis->iCurrOffs; +- pThis->iCurrOffs = 0; ++ pThis->strtOffs = pThis->iCurrOffs = 0; + DBGOPRINT((obj_t*) pThis, "encrypted, doing skip read of %lld bytes\n", + (long long) targetOffs); + while(targetOffs != pThis->iCurrOffs) { +@@ -1935,6 +1987,12 @@ static rsRetVal strmSetiMaxFiles(strm_t *pThis, int iNewVal) + return RS_RET_OK; + } + ++static rsRetVal strmSetFileNotFoundError(strm_t *pThis, int pFileNotFoundError) ++{ ++ pThis->fileNotFoundError = pFileNotFoundError; ++ return RS_RET_OK; ++} ++ + + /* set the stream's file prefix + * The passed-in string is duplicated. So if the caller does not need +@@ -2076,6 +2134,9 @@ static rsRetVal strmSerialize(strm_t *pThis, strm_t *pStrm) + l = pThis->inode; + objSerializeSCALAR_VAR(pStrm, inode, INT64, l); + ++ l = pThis->strtOffs; ++ objSerializeSCALAR_VAR(pStrm, strtOffs, INT64, l); ++ + if(pThis->prevLineSegment != NULL) { + cstrFinalize(pThis->prevLineSegment); + objSerializePTR(pStrm, prevLineSegment, CSTR); +@@ -2188,8 +2249,12 @@ static rsRetVal strmSetProperty(strm_t *pThis, var_t *pProp) + pThis->iCurrOffs = pProp->val.num; + } else if(isProp("inode")) { + pThis->inode = (ino_t) pProp->val.num; ++ } else if(isProp("strtOffs")) { ++ pThis->strtOffs = pProp->val.num; + } else if(isProp("iMaxFileSize")) { + CHKiRet(strmSetiMaxFileSize(pThis, pProp->val.num)); ++ } else if(isProp("fileNotFoundError")) { ++ CHKiRet(strmSetFileNotFoundError(pThis, pProp->val.num)); + } else if(isProp("iMaxFiles")) { + CHKiRet(strmSetiMaxFiles(pThis, pProp->val.num)); + } else if(isProp("iFileNumDigits")) { +@@ -2253,6 +2318,7 @@ CODESTARTobjQueryInterface(strm) + pIf->WriteChar = strmWriteChar; + pIf->WriteLong = strmWriteLong; + pIf->SetFName = strmSetFName; ++ pIf->SetFileNotFoundError = strmSetFileNotFoundError; + pIf->SetDir = strmSetDir; + pIf->Flush = strmFlush; + pIf->RecordBegin = strmRecordBegin; +diff --git a/runtime/stream.h b/runtime/stream.h +index 1eee34979db34620b82e6351111864645187b035..bcb81a14f60f9effa52fffa42d18d66c484ae86d 100644 +--- a/runtime/stream.h ++++ b/runtime/stream.h +@@ -159,6 +159,10 @@ typedef struct strm_s { + sbool bIsTTY; /* is this a tty file? */ + cstr_t *prevLineSegment; /* for ReadLine, previous, unprocessed part of file */ + cstr_t *prevMsgSegment; /* for ReadMultiLine, previous, yet unprocessed part of msg */ ++ int64 strtOffs; /* start offset in file for current line/msg */ ++ int fileNotFoundError; ++ int noRepeatedErrorOutput; /* if a file is missing the Error is only given once */ ++ int ignoringMsg; + } strm_t; + + +@@ -174,6 +178,7 @@ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ + rsRetVal (*Write)(strm_t *const pThis, const uchar *const pBuf, size_t lenBuf); + rsRetVal (*WriteChar)(strm_t *pThis, uchar c); + rsRetVal (*WriteLong)(strm_t *pThis, long i); ++ rsRetVal (*SetFileNotFoundError)(strm_t *pThis, int pFileNotFoundError); + rsRetVal (*SetFName)(strm_t *pThis, uchar *pszPrefix, size_t iLenPrefix); + rsRetVal (*SetDir)(strm_t *pThis, uchar *pszDir, size_t iLenDir); + rsRetVal (*Flush)(strm_t *pThis); +@@ -198,7 +203,8 @@ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ + INTERFACEpropSetMeth(strm, iFlushInterval, int); + INTERFACEpropSetMeth(strm, pszSizeLimitCmd, uchar*); + /* v6 added */ +- rsRetVal (*ReadLine)(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF, uint32_t trimLineOverBytes); ++ rsRetVal (*ReadLine)(strm_t *pThis, cstr_t **ppCStr, uint8_t mode, sbool bEscapeLF, ++ uint32_t trimLineOverBytes, int64 *const strtOffs); + /* v7 added 2012-09-14 */ + INTERFACEpropSetMeth(strm, bVeryReliableZip, int); + /* v8 added 2013-03-21 */ +@@ -207,19 +213,24 @@ BEGINinterface(strm) /* name must also be changed in ENDinterface macro! */ + INTERFACEpropSetMeth(strm, cryprov, cryprov_if_t*); + INTERFACEpropSetMeth(strm, cryprovData, void*); + ENDinterface(strm) +-#define strmCURR_IF_VERSION 12 /* increment whenever you change the interface structure! */ ++#define strmCURR_IF_VERSION 13 /* increment whenever you change the interface structure! */ + /* V10, 2013-09-10: added new parameter bEscapeLF, changed mode to uint8_t (rgerhards) */ + /* V11, 2015-12-03: added new parameter bReopenOnTruncate */ + /* V12, 2015-12-11: added new parameter trimLineOverBytes, changed mode to uint32_t */ ++/* V13, 2017-09-06: added new parameter strtoffs to ReadLine() */ + + #define strmGetCurrFileNum(pStrm) ((pStrm)->iCurrFNum) + + /* prototypes */ + PROTOTYPEObjClassInit(strm); + rsRetVal strmMultiFileSeek(strm_t *pThis, unsigned int fileNum, off64_t offs, off64_t *bytesDel); +-rsRetVal strmReadMultiLine(strm_t *pThis, cstr_t **ppCStr, regex_t *preg, sbool bEscapeLF); ++rsRetVal strmReadMultiLine(strm_t *pThis, cstr_t **ppCStr, regex_t *preg, ++ sbool bEscapeLF, int64 *const strtOffs); + int strmReadMultiLine_isTimedOut(const strm_t *const __restrict__ pThis); + void strmDebugOutBuf(const strm_t *const pThis); + void strmSetReadTimeout(strm_t *const __restrict__ pThis, const int val); ++const uchar * strmGetPrevLineSegment(strm_t *const pThis); ++const uchar * strmGetPrevMsgSegment(strm_t *const pThis); ++int strmGetPrevWasNL(const strm_t *const pThis); + + #endif /* #ifndef STREAM_H_INCLUDED */ + diff --git a/SOURCES/rsyslog-8.24.0-rhbz1538372-imjournal-duplicates.patch b/SOURCES/rsyslog-8.24.0-rhbz1538372-imjournal-duplicates.patch new file mode 100644 index 0000000..1788d6c --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1538372-imjournal-duplicates.patch @@ -0,0 +1,370 @@ +From: Jiri Vymazal +Date: Wed, 25 Jul 2018 15:05:01 -0500 + +modification and merge of below patches for RHEL consumers, +also modified journal invalidate/rotation handling to keep possibility +to continue after switch of persistent journal +original: +%From 3bede5ba768975c8b6fe3d1f3e11075910f52fdd Mon Sep 17 00:00:00 2001 +%From: Jiri Vymazal +%Date: Wed, 7 Mar 2018 11:57:29 +0100 +%Subject: [PATCH] Fetching cursor on readJournal() and simplified pollJournal() +% +%Fetching journal cursor in persistJournal could cause us to save +%invalid cursor leading to duplicating messages further on, now we are +%saving it on each readJournal() where we now that the state is good. +%This result in simplyfing persisJournalState() a bit as well. +% +%pollJournal() is now cleaner and faster, correctly handles INVALIDATE +%status from journald and is able to continue polling after journal +%flush. Also reduced POLL_TIMEOUT a bit as it caused rsyslog to exit +%with error in corner cases for some ppc when left at full second. +plus +% +%From a99f9b4b42d261c384aee09306fc421df2cca7a5 Mon Sep 17 00:00:00 2001 +%From: Peter Portante +%Date: Wed, 24 Jan 2018 19:34:41 -0500 +%Subject: [PATCH] Proposed fix for handling journal correctly +% +%The fix is to immediately setup the inotify file descriptor via +%`sd_journal_get_fd()` right after a journal open, and then +%periodically call `sd_journal_process()` to give the client API +%library a chance to detect deleted journal files on disk that need to +%be closed so they can be properly erased by the file system. +% +%We remove the open/close dance and simplify that code as a result. +% +%Fixes issue #2436. +and also: +%From 27f96c84d34ee000fbb5d45b00233f2ec3cf2d8a Mon Sep 17 00:00:00 2001 +%From: Rainer Gerhards +%Date: Tue, 24 Oct 2017 16:14:13 +0200 +%Subject: [PATCH] imjournal bugfix: do not disable itself on error +% +%If some functions calls inside the main loop failed, imjournal exited +%with an error code, actually disabling all logging from the journal. +%This was probably never intended. +% +%This patch makes imjournal recover the situation instead. +% +%closes https://github.com/rsyslog/rsyslog/issues/1895 +--- + plugins/imjournal/imjournal.c | 211 ++++++++++++++++++++++-------------------- + 1 file changed, 110 insertions(+), 102 deletions(-) + +--- a/plugins/imjournal/imjournal.c ++++ b/plugins/imjournal/imjournal.c +@@ -80,6 +80,7 @@ static struct configSettings_s { + int iDfltFacility; + int bUseJnlPID; + char *dfltTag; ++ int bWorkAroundJournalBug; + } cs; + + static rsRetVal facilityHdlr(uchar **pp, void *pVal); +@@ -95,6 +96,7 @@ static struct cnfparamdescr modpdescr[] = { + { "defaultfacility", eCmdHdlrString, 0 }, + { "usepidfromsystem", eCmdHdlrBinary, 0 }, + { "defaulttag", eCmdHdlrGetWord, 0 }, ++ { "workaroundjournalbug", eCmdHdlrBinary, 0 } + }; + static struct cnfparamblk modpblk = + { CNFPARAMBLK_VERSION, +@@ -114,6 +114,10 @@ /* module-global parameters */ + static const char *pid_field_name; /* read-only after startup */ + static ratelimit_t *ratelimiter = NULL; + static sd_journal *j; ++static int j_inotify_fd; ++static char *last_cursor = NULL; ++ ++#define J_PROCESS_PERIOD 1024 /* Call sd_journal_process() every 1,024 records */ + + static rsRetVal persistJournalState(void); + static rsRetVal loadJournalState(void); +@@ -123,6 +127,14 @@ openJournal(sd_journal** jj) + + if (sd_journal_open(jj, SD_JOURNAL_LOCAL_ONLY) < 0) + iRet = RS_RET_IO_ERROR; ++ int r; ++ ++ if ((r = sd_journal_get_fd(j)) < 0) { ++ errmsg.LogError(-r, RS_RET_IO_ERROR, "imjournal: sd_journal_get_fd() failed"); ++ iRet = RS_RET_IO_ERROR; ++ } else { ++ j_inotify_fd = r; ++ } + RETiRet; + } + +@@ -132,6 +144,7 @@ closeJournal(sd_journal** jj) + persistJournalState(); + } + sd_journal_close(*jj); ++ j_inotify_fd = 0; + } + + +@@ -262,6 +275,7 @@ readjournal(void) + char *message = NULL; + char *sys_iden = NULL; + char *sys_iden_help = NULL; ++ char *c = NULL; + + const void *get; + const void *pidget; +@@ -433,6 +437,15 @@ readjournal(void) + tv.tv_usec = timestamp % 1000000; + } + ++ if (cs.bWorkAroundJournalBug) { ++ /* save journal cursor (at this point we can be sure it is valid) */ ++ sd_journal_get_cursor(j, &c); ++ if (c) { ++ free(last_cursor); ++ last_cursor = c; ++ } ++ } ++ + /* submit message */ + enqMsg((uchar *)message, (uchar *) sys_iden_help, facility, severity, &tv, json, 0); + +@@ -413,44 +433,49 @@ persistJournalState (void) + DEFiRet; + FILE *sf; /* state file */ + char tmp_sf[MAXFNAME]; +- char *cursor; + int ret = 0; + +- /* On success, sd_journal_get_cursor() returns 1 in systemd +- 197 or older and 0 in systemd 198 or newer */ +- if ((ret = sd_journal_get_cursor(j, &cursor)) >= 0) { +- /* we create a temporary name by adding a ".tmp" +- * suffix to the end of our state file's name +- */ +- snprintf(tmp_sf, sizeof(tmp_sf), "%s.tmp", cs.stateFile); +- if ((sf = fopen(tmp_sf, "wb")) != NULL) { +- if (fprintf(sf, "%s", cursor) < 0) { +- iRet = RS_RET_IO_ERROR; +- } +- fclose(sf); +- free(cursor); +- /* change the name of the file to the configured one */ +- if (iRet == RS_RET_OK && rename(tmp_sf, cs.stateFile) == -1) { +- char errStr[256]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- iRet = RS_RET_IO_ERROR; +- errmsg.LogError(0, iRet, "rename() failed: " +- "'%s', new path: '%s'\n", errStr, cs.stateFile); +- } ++ if (cs.bWorkAroundJournalBug) { ++ if (!last_cursor) ++ ABORT_FINALIZE(RS_RET_OK); + +- } else { +- char errStr[256]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- errmsg.LogError(0, RS_RET_FOPEN_FAILURE, "fopen() failed: " +- "'%s', path: '%s'\n", errStr, tmp_sf); +- iRet = RS_RET_FOPEN_FAILURE; +- } +- } else { ++ } else if ((ret = sd_journal_get_cursor(j, &last_cursor)) < 0) { + char errStr[256]; + rs_strerror_r(-(ret), errStr, sizeof(errStr)); + errmsg.LogError(0, RS_RET_ERR, "sd_journal_get_cursor() failed: '%s'\n", errStr); +- iRet = RS_RET_ERR; ++ ABORT_FINALIZE(RS_RET_ERR); + } ++ /* we create a temporary name by adding a ".tmp" ++ * suffix to the end of our state file's name ++ */ ++ snprintf(tmp_sf, sizeof(tmp_sf), "%s.tmp", cs.stateFile); ++ ++ sf = fopen(tmp_sf, "wb"); ++ if (!sf) { ++ errmsg.LogError(errno, RS_RET_FOPEN_FAILURE, "imjournal: fopen() failed for path: '%s'", tmp_sf); ++ ABORT_FINALIZE(RS_RET_FOPEN_FAILURE); ++ } ++ ++ ret = fputs(last_cursor, sf); ++ if (ret < 0) { ++ errmsg.LogError(errno, RS_RET_IO_ERROR, "imjournal: failed to save cursor to: '%s'", tmp_sf); ++ ret = fclose(sf); ++ ABORT_FINALIZE(RS_RET_IO_ERROR); ++ } ++ ++ ret = fclose(sf); ++ if (ret < 0) { ++ errmsg.LogError(errno, RS_RET_IO_ERROR, "imjournal: fclose() failed for path: '%s'", tmp_sf); ++ ABORT_FINALIZE(RS_RET_IO_ERROR); ++ } ++ ++ ret = rename(tmp_sf, cs.stateFile); ++ if (ret < 0) { ++ errmsg.LogError(errno, iRet, "imjournal: rename() failed for new path: '%s'", cs.stateFile); ++ ABORT_FINALIZE(RS_RET_IO_ERROR); ++ } ++ ++finalize_it: + RETiRet; + } + +@@ -473,64 +473,26 @@ + * except for the special handling of EINTR. + */ + +-#define POLL_TIMEOUT 1000 /* timeout for poll is 1s */ ++#define POLL_TIMEOUT 900000 /* timeout for poll is 900ms */ + + static rsRetVal + pollJournal(void) + { + DEFiRet; +- struct pollfd pollfd; +- int pr = 0; +- int jr = 0; +- +- pollfd.fd = sd_journal_get_fd(j); +- pollfd.events = sd_journal_get_events(j); +- pr = poll(&pollfd, 1, POLL_TIMEOUT); +- if (pr == -1) { +- if (errno == EINTR) { +- /* EINTR is also received during termination +- * so return now to check the term state. +- */ +- ABORT_FINALIZE(RS_RET_OK); +- } else { +- char errStr[256]; +- +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- errmsg.LogError(0, RS_RET_ERR, +- "poll() failed: '%s'", errStr); +- ABORT_FINALIZE(RS_RET_ERR); +- } +- } ++ int r; + ++ r = sd_journal_wait(j, POLL_TIMEOUT); + +- jr = sd_journal_process(j); +- +- if (pr == 1 && jr == SD_JOURNAL_INVALIDATE) { +- /* do not persist stateFile sd_journal_get_cursor will fail! */ +- char* tmp = cs.stateFile; +- cs.stateFile = NULL; ++ if (r == SD_JOURNAL_INVALIDATE) { + closeJournal(&j); +- cs.stateFile = tmp; + + iRet = openJournal(&j); +- if (iRet != RS_RET_OK) { +- char errStr[256]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- errmsg.LogError(0, RS_RET_IO_ERROR, +- "sd_journal_open() failed: '%s'", errStr); ++ if (iRet != RS_RET_OK) + ABORT_FINALIZE(RS_RET_ERR); +- } + +- if(cs.stateFile != NULL){ ++ if (cs.stateFile) + iRet = loadJournalState(); +- } +- LogMsg(0, RS_RET_OK, LOG_NOTICE, "imjournal: journal reloaded..."); +- } else if (jr < 0) { +- char errStr[256]; +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- errmsg.LogError(0, RS_RET_ERR, +- "sd_journal_process() failed: '%s'", errStr); +- ABORT_FINALIZE(RS_RET_ERR); ++ errmsg.LogMsg(0, RS_RET_OK, LOG_NOTICE, "imjournal: journal reloaded..."); + } + + finalize_it: +@@ -631,8 +612,17 @@ loadJournalState(void) + RETiRet; + } + ++static void ++tryRecover(void) { ++ errmsg.LogMsg(0, RS_RET_OK, LOG_INFO, "imjournal: trying to recover from unexpected " ++ "journal error"); ++ closeJournal(&j); ++ srSleep(10, 0); // do not hammer machine with too-frequent retries ++ openJournal(&j); ++} ++ + BEGINrunInput +- int count = 0; ++ uint64_t count = 0; + CODESTARTrunInput + CHKiRet(ratelimitNew(&ratelimiter, "imjournal", NULL)); + dbgprintf("imjournal: ratelimiting burst %d, interval %d\n", cs.ratelimitBurst, +@@ -665,26 +655,38 @@ CODESTARTrunInput + + r = sd_journal_next(j); + if (r < 0) { +- char errStr[256]; +- +- rs_strerror_r(errno, errStr, sizeof(errStr)); +- errmsg.LogError(0, RS_RET_ERR, +- "sd_journal_next() failed: '%s'", errStr); +- ABORT_FINALIZE(RS_RET_ERR); ++ tryRecover(); ++ continue; + } + + if (r == 0) { + /* No new messages, wait for activity. */ +- CHKiRet(pollJournal()); ++ if (pollJournal() != RS_RET_OK) { ++ tryRecover(); ++ } + continue; + } + +- CHKiRet(readjournal()); ++ if (readjournal() != RS_RET_OK) { ++ tryRecover(); ++ continue; ++ } ++ ++ count++; ++ ++ if ((count % J_PROCESS_PERIOD) == 0) { ++ /* Give the journal a periodic chance to detect rotated journal files to be cleaned up. */ ++ r = sd_journal_process(j); ++ if (r < 0) { ++ errmsg.LogError(-r, RS_RET_ERR, "imjournal: sd_journal_process() failed"); ++ tryRecover(); ++ continue; ++ } ++ } ++ + if (cs.stateFile) { /* can't persist without a state file */ + /* TODO: This could use some finer metric. */ +- count++; +- if (count == cs.iPersistStateInterval) { +- count = 0; ++ if ((count % cs.iPersistStateInterval) == 0) { + persistJournalState(); + } + } +@@ -901,6 +909,8 @@ CODESTARTsetModCnf + cs.bUseJnlPID = (int) pvals[i].val.d.n; + } else if (!strcmp(modpblk.descr[i].name, "defaulttag")) { + cs.dfltTag = (char *)es_str2cstr(pvals[i].val.d.estr, NULL); ++ } else if (!strcmp(modpblk.descr[i].name, "workaroundjournalbug")) { ++ cs.bWorkAroundJournalBug = (int) pvals[i].val.d.n; + } else { + dbgprintf("imjournal: program error, non-handled " + "param '%s' in beginCnfLoad\n", modpblk.descr[i].name); +@@ -961,6 +971,8 @@ CODEmodInit_QueryRegCFSLineHdlr + NULL, &cs.bUseJnlPID, STD_LOADABLE_MODULE_ID)); + CHKiRet(omsdRegCFSLineHdlr((uchar *)"imjournaldefaulttag", 0, eCmdHdlrGetWord, + NULL, &cs.dfltTag, STD_LOADABLE_MODULE_ID)); ++ CHKiRet(omsdRegCFSLineHdlr((uchar *)"workaroundjournalbug", 0, eCmdHdlrBinary, ++ NULL, &cs.bWorkAroundJournalBug, STD_LOADABLE_MODULE_ID)); + ENDmodInit + /* vim:set ai: + */ diff --git a/SOURCES/rsyslog-8.24.0-rhbz1539193-mmkubernetes-new-plugin.patch b/SOURCES/rsyslog-8.24.0-rhbz1539193-mmkubernetes-new-plugin.patch new file mode 100644 index 0000000..f37557c --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1539193-mmkubernetes-new-plugin.patch @@ -0,0 +1,1659 @@ +From: Jiri Vymazal +Date: Mon, 28 Jun 2018 12:07:55 +0100 +Subject: Kubernetes Metadata plugin - mmkubernetes + +This plugin is used to annotate records logged by Kubernetes containers. +It will add the namespace uuid, pod uuid, pod and namespace labels and +annotations, and other metadata associated with the pod and namespace. +It will work with either log files in `/var/log/containers/*.log` or +with journald entries with `CONTAINER_NAME` and `CONTAINER_ID_FULL`. + +For usage and configuration see syslog-doc + +*Credits* + +This work is based on https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter +and has many of the same features. + +(cherry picked from commit a6264bf8f91975c9bc0fc602dcdc6881486f1579) +(cherry picked from commit b8e68366422052dca9e0a9409baa410f20ae88c8) + +(cherry picked from commit 77886e21292d8220f93b3404236da0e8f7159255) +(cherry picked from commit e4d1c7b3832eedc8a1545c2ee6bf022f545d0c76) +(cherry picked from commit 3d9f820642b0edc78da0b5bed818590dcd31fa9c) +(cherry picked from commit 1d49aac5cb101704486bfb065fac362ca69f06bc) +(cherry picked from commit fc2ad45f78dd666b8c9e706ad88c17aaff146d2d) +(cherry picked from commit 8cf87f64f6c74a4544112ec7fddc5bf4d43319a7) +--- + Makefile.am | 5 + + configure.ac | 35 + + contrib/mmkubernetes/Makefile.am | 6 + + contrib/mmkubernetes/k8s_container_name.rulebase | 3 + + contrib/mmkubernetes/k8s_filename.rulebase | 2 + + contrib/mmkubernetes/mmkubernetes.c | 1491 +++++++++++++++++++++++ + contrib/mmkubernetes/sample.conf | 7 + + 7 files changed, 1549 insertions(+) + create mode 100644 contrib/mmkubernetes/Makefile.am + create mode 100644 contrib/mmkubernetes/k8s_container_name.rulebase + create mode 100644 contrib/mmkubernetes/k8s_filename.rulebase + create mode 100644 contrib/mmkubernetes/mmkubernetes.c + create mode 100644 contrib/mmkubernetes/sample.conf + +diff --git a/Makefile.am b/Makefile.am +index a276ef9ea..b58ebaf93 100644 +--- a/Makefile.am ++++ b/Makefile.am +@@ -275,6 +275,11 @@ if ENABLE_OMTCL + SUBDIRS += contrib/omtcl + endif + ++# mmkubernetes ++if ENABLE_MMKUBERNETES ++SUBDIRS += contrib/mmkubernetes ++endif ++ + # tests are added as last element, because tests may need different + # modules that need to be generated first + SUBDIRS += tests +diff --git a/configure.ac b/configure.ac +index a9411f4be..c664222b9 100644 +--- a/configure.ac ++++ b/configure.ac +@@ -1889,6 +1889,39 @@ AM_CONDITIONAL(ENABLE_OMTCL, test x$enable_omtcl = xyes) + + # END TCL SUPPORT + ++# mmkubernetes - Kubernetes metadata support ++ ++AC_ARG_ENABLE(mmkubernetes, ++ [AS_HELP_STRING([--enable-mmkubernetes], ++ [Enable compilation of the mmkubernetes module @<:@default=no@:>@])], ++ [case "${enableval}" in ++ yes) enable_mmkubernetes="yes" ;; ++ no) enable_mmkubernetes="no" ;; ++ *) AC_MSG_ERROR(bad value ${enableval} for --enable-mmkubernetes) ;; ++ esac], ++ [enable_mmkubernetes=no] ++) ++if test "x$enable_mmkubernetes" = "xyes"; then ++ PKG_CHECK_MODULES([CURL], [libcurl]) ++ PKG_CHECK_MODULES(LIBLOGNORM, lognorm >= 2.0.3) ++ ++ save_CFLAGS="$CFLAGS" ++ save_LIBS="$LIBS" ++ ++ CFLAGS="$CFLAGS $LIBLOGNORM_CFLAGS" ++ LIBS="$LIBS $LIBLOGNORM_LIBS" ++ ++ AC_CHECK_FUNC([ln_loadSamplesFromString], ++ [AC_DEFINE([HAVE_LOADSAMPLESFROMSTRING], [1], [Define if ln_loadSamplesFromString exists.])], ++ [AC_DEFINE([NO_LOADSAMPLESFROMSTRING], [1], [Define if ln_loadSamplesFromString does not exist.])]) ++ ++ CFLAGS="$save_CFLAGS" ++ LIBS="$save_LIBS" ++fi ++AM_CONDITIONAL(ENABLE_MMKUBERNETES, test x$enable_mmkubernetes = xyes) ++ ++# END Kubernetes metadata support ++ + # man pages + AC_CHECKING([if required man pages already exist]) + have_to_generate_man_pages="no" +@@ -2016,6 +2035,7 @@ AC_CONFIG_FILES([Makefile \ + contrib/omhttpfs/Makefile \ + contrib/omamqp1/Makefile \ + contrib/omtcl/Makefile \ ++ contrib/mmkubernetes/Makefile \ + tests/Makefile]) + AC_OUTPUT + +@@ -2090,6 +2110,7 @@ echo " mmrfc5424addhmac enabled: $enable_mmrfc5424addhmac" + echo " mmpstrucdata enabled: $enable_mmpstrucdata" + echo " mmsequence enabled: $enable_mmsequence" + echo " mmdblookup enabled: $enable_mmdblookup" ++echo " mmkubernetes enabled: $enable_mmkubernetes" + echo + echo "---{ database support }---" + echo " MySql support enabled: $enable_mysql" +diff --git a/contrib/mmkubernetes/Makefile.am b/contrib/mmkubernetes/Makefile.am +new file mode 100644 +index 000000000..3dcc235a6 +--- /dev/null ++++ b/contrib/mmkubernetes/Makefile.am +@@ -0,0 +1,6 @@ ++pkglib_LTLIBRARIES = mmkubernetes.la ++ ++mmkubernetes_la_SOURCES = mmkubernetes.c ++mmkubernetes_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) $(CURL_CFLAGS) $(LIBLOGNORM_CFLAGS) ++mmkubernetes_la_LDFLAGS = -module -avoid-version ++mmkubernetes_la_LIBADD = $(CURL_LIBS) $(LIBLOGNORM_LIBS) +diff --git a/contrib/mmkubernetes/k8s_container_name.rulebase b/contrib/mmkubernetes/k8s_container_name.rulebase +new file mode 100644 +index 000000000..35fbb317c +--- /dev/null ++++ b/contrib/mmkubernetes/k8s_container_name.rulebase +@@ -0,0 +1,3 @@ ++version=2 ++rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest% ++rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest% +diff --git a/contrib/mmkubernetes/k8s_filename.rulebase b/contrib/mmkubernetes/k8s_filename.rulebase +new file mode 100644 +index 000000000..24c0d9138 +--- /dev/null ++++ b/contrib/mmkubernetes/k8s_filename.rulebase +@@ -0,0 +1,2 @@ ++version=2 ++rule=:/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name_and_id:char-to:.%.log +diff --git a/contrib/mmkubernetes/mmkubernetes.c b/contrib/mmkubernetes/mmkubernetes.c +new file mode 100644 +index 000000000..5012c54f6 +--- /dev/null ++++ b/contrib/mmkubernetes/mmkubernetes.c +@@ -0,0 +1,1491 @@ ++/* mmkubernetes.c ++ * This is a message modification module. It uses metadata obtained ++ * from the message to query Kubernetes and obtain additional metadata ++ * relating to the container instance. ++ * ++ * Inspired by: ++ * https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter ++ * ++ * NOTE: read comments in module-template.h for details on the calling interface! ++ * ++ * Copyright 2016 Red Hat Inc. ++ * ++ * This file is part of rsyslog. ++ * ++ * Licensed under the Apache License, Version 2.0 (the "License"); ++ * you may not use this file except in compliance with the License. ++ * You may obtain a copy of the License at ++ * ++ * http://www.apache.org/licenses/LICENSE-2.0 ++ * -or- ++ * see COPYING.ASL20 in the source distribution ++ * ++ * Unless required by applicable law or agreed to in writing, software ++ * distributed under the License is distributed on an "AS IS" BASIS, ++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ++ * See the License for the specific language governing permissions and ++ * limitations under the License. ++ */ ++ ++/* needed for asprintf */ ++#ifndef _GNU_SOURCE ++# define _GNU_SOURCE ++#endif ++ ++#include "config.h" ++#include "rsyslog.h" ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include "conf.h" ++#include "syslogd-types.h" ++#include "module-template.h" ++#include "errmsg.h" ++#include "regexp.h" ++#include "hashtable.h" ++#include "srUtils.h" ++ ++/* static data */ ++MODULE_TYPE_OUTPUT /* this is technically an output plugin */ ++MODULE_TYPE_KEEP /* releasing the module would cause a leak through libcurl */ ++MODULE_CNFNAME("mmkubernetes") ++DEF_OMOD_STATIC_DATA ++DEFobjCurrIf(errmsg) ++DEFobjCurrIf(regexp) ++ ++#define HAVE_LOADSAMPLESFROMSTRING 1 ++#if defined(NO_LOADSAMPLESFROMSTRING) ++#undef HAVE_LOADSAMPLESFROMSTRING ++#endif ++/* original from fluentd plugin: ++ * 'var\.log\.containers\.(?[a-z0-9]([-a-z0-9]*[a-z0-9])?\ ++ * (\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_\ ++ * (?.+)-(?[a-z0-9]{64})\.log$' ++ * this is for _tag_ match, not actual filename match - in_tail turns filename ++ * into a fluentd tag ++ */ ++#define DFLT_FILENAME_LNRULES "rule=:/var/log/containers/%pod_name:char-to:_%_"\ ++ "%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log" ++#define DFLT_FILENAME_RULEBASE "/etc/rsyslog.d/k8s_filename.rulebase" ++/* original from fluentd plugin: ++ * '^(?[^_]+)_(?[^\._]+)\ ++ * (\.(?[^_]+))?_(?[^_]+)_\ ++ * (?[^_]+)_[^_]+_[^_]+$' ++ */ ++#define DFLT_CONTAINER_LNRULES "rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%."\ ++ "%container_hash:char-to:_%_"\ ++ "%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%\n"\ ++ "rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_"\ ++ "%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%" ++#define DFLT_CONTAINER_RULEBASE "/etc/rsyslog.d/k8s_container_name.rulebase" ++#define DFLT_SRCMD_PATH "$!metadata!filename" ++#define DFLT_DSTMD_PATH "$!" ++#define DFLT_DE_DOT 1 /* true */ ++#define DFLT_DE_DOT_SEPARATOR "_" ++#define DFLT_CONTAINER_NAME "$!CONTAINER_NAME" /* name of variable holding CONTAINER_NAME value */ ++#define DFLT_CONTAINER_ID_FULL "$!CONTAINER_ID_FULL" /* name of variable holding CONTAINER_ID_FULL value */ ++#define DFLT_KUBERNETES_URL "https://kubernetes.default.svc.cluster.local:443" ++ ++static struct cache_s { ++ const uchar *kbUrl; ++ struct hashtable *mdHt; ++ struct hashtable *nsHt; ++ pthread_mutex_t *cacheMtx; ++} **caches; ++ ++typedef struct { ++ int nmemb; ++ uchar **patterns; ++ regex_t *regexps; ++} annotation_match_t; ++ ++/* module configuration data */ ++struct modConfData_s { ++ rsconf_t *pConf; /* our overall config object */ ++ uchar *kubernetesUrl; /* scheme, host, port, and optional path prefix for Kubernetes API lookups */ ++ uchar *srcMetadataPath; /* where to get data for kubernetes queries */ ++ uchar *dstMetadataPath; /* where to put metadata obtained from kubernetes */ ++ uchar *caCertFile; /* File holding the CA cert (+optional chain) of CA that issued the Kubernetes server cert */ ++ sbool allowUnsignedCerts; /* For testing/debugging - do not check for CA certs (CURLOPT_SSL_VERIFYPEER FALSE) */ ++ uchar *token; /* The token value to use to authenticate to Kubernetes - takes precedence over tokenFile */ ++ uchar *tokenFile; /* The file whose contents is the token value to use to authenticate to Kubernetes */ ++ sbool de_dot; /* If true (default), convert '.' characters in labels & annotations to de_dot_separator */ ++ uchar *de_dot_separator; /* separator character (default '_') to use for de_dotting */ ++ size_t de_dot_separator_len; /* length of separator character */ ++ annotation_match_t annotation_match; /* annotation keys must match these to be included in record */ ++ char *fnRules; /* lognorm rules for container log filename match */ ++ uchar *fnRulebase; /* lognorm rulebase filename for container log filename match */ ++ char *contRules; /* lognorm rules for CONTAINER_NAME value match */ ++ uchar *contRulebase; /* lognorm rulebase filename for CONTAINER_NAME value match */ ++}; ++ ++/* action (instance) configuration data */ ++typedef struct _instanceData { ++ uchar *kubernetesUrl; /* scheme, host, port, and optional path prefix for Kubernetes API lookups */ ++ msgPropDescr_t *srcMetadataDescr; /* where to get data for kubernetes queries */ ++ uchar *dstMetadataPath; /* where to put metadata obtained from kubernetes */ ++ uchar *caCertFile; /* File holding the CA cert (+optional chain) of CA that issued the Kubernetes server cert */ ++ sbool allowUnsignedCerts; /* For testing/debugging - do not check for CA certs (CURLOPT_SSL_VERIFYPEER FALSE) */ ++ uchar *token; /* The token value to use to authenticate to Kubernetes - takes precedence over tokenFile */ ++ uchar *tokenFile; /* The file whose contents is the token value to use to authenticate to Kubernetes */ ++ sbool de_dot; /* If true (default), convert '.' characters in labels & annotations to de_dot_separator */ ++ uchar *de_dot_separator; /* separator character (default '_') to use for de_dotting */ ++ size_t de_dot_separator_len; /* length of separator character */ ++ annotation_match_t annotation_match; /* annotation keys must match these to be included in record */ ++ char *fnRules; /* lognorm rules for container log filename match */ ++ uchar *fnRulebase; /* lognorm rulebase filename for container log filename match */ ++ ln_ctx fnCtxln; /**< context to be used for liblognorm */ ++ char *contRules; /* lognorm rules for CONTAINER_NAME value match */ ++ uchar *contRulebase; /* lognorm rulebase filename for CONTAINER_NAME value match */ ++ ln_ctx contCtxln; /**< context to be used for liblognorm */ ++ msgPropDescr_t *contNameDescr; /* CONTAINER_NAME field */ ++ msgPropDescr_t *contIdFullDescr; /* CONTAINER_ID_FULL field */ ++ struct cache_s *cache; ++} instanceData; ++ ++typedef struct wrkrInstanceData { ++ instanceData *pData; ++ CURL *curlCtx; ++ struct curl_slist *curlHdr; ++ char *curlRply; ++ size_t curlRplyLen; ++} wrkrInstanceData_t; ++ ++/* module parameters (v6 config format) */ ++static struct cnfparamdescr modpdescr[] = { ++ { "kubernetesurl", eCmdHdlrString, 0 }, ++ { "srcmetadatapath", eCmdHdlrString, 0 }, ++ { "dstmetadatapath", eCmdHdlrString, 0 }, ++ { "tls.cacert", eCmdHdlrString, 0 }, ++ { "allowunsignedcerts", eCmdHdlrBinary, 0 }, ++ { "token", eCmdHdlrString, 0 }, ++ { "tokenfile", eCmdHdlrString, 0 }, ++ { "annotation_match", eCmdHdlrArray, 0 }, ++ { "de_dot", eCmdHdlrBinary, 0 }, ++ { "de_dot_separator", eCmdHdlrString, 0 }, ++ { "filenamerulebase", eCmdHdlrString, 0 }, ++ { "containerrulebase", eCmdHdlrString, 0 } ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ , ++ { "filenamerules", eCmdHdlrArray, 0 }, ++ { "containerrules", eCmdHdlrArray, 0 } ++#endif ++}; ++static struct cnfparamblk modpblk = { ++ CNFPARAMBLK_VERSION, ++ sizeof(modpdescr)/sizeof(struct cnfparamdescr), ++ modpdescr ++}; ++ ++/* action (instance) parameters (v6 config format) */ ++static struct cnfparamdescr actpdescr[] = { ++ { "kubernetesurl", eCmdHdlrString, 0 }, ++ { "srcmetadatapath", eCmdHdlrString, 0 }, ++ { "dstmetadatapath", eCmdHdlrString, 0 }, ++ { "tls.cacert", eCmdHdlrString, 0 }, ++ { "allowunsignedcerts", eCmdHdlrBinary, 0 }, ++ { "token", eCmdHdlrString, 0 }, ++ { "tokenfile", eCmdHdlrString, 0 }, ++ { "annotation_match", eCmdHdlrArray, 0 }, ++ { "de_dot", eCmdHdlrBinary, 0 }, ++ { "de_dot_separator", eCmdHdlrString, 0 }, ++ { "filenamerulebase", eCmdHdlrString, 0 }, ++ { "containerrulebase", eCmdHdlrString, 0 } ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ , ++ { "filenamerules", eCmdHdlrArray, 0 }, ++ { "containerrules", eCmdHdlrArray, 0 } ++#endif ++}; ++static struct cnfparamblk actpblk = ++ { CNFPARAMBLK_VERSION, ++ sizeof(actpdescr)/sizeof(struct cnfparamdescr), ++ actpdescr ++ }; ++ ++static modConfData_t *loadModConf = NULL; /* modConf ptr to use for the current load process */ ++static modConfData_t *runModConf = NULL; /* modConf ptr to use for the current exec process */ ++ ++static void free_annotationmatch(annotation_match_t *match) { ++ if (match) { ++ for(int ii = 0 ; ii < match->nmemb; ++ii) { ++ if (match->patterns) ++ free(match->patterns[ii]); ++ if (match->regexps) ++ regexp.regfree(&match->regexps[ii]); ++ } ++ free(match->patterns); ++ match->patterns = NULL; ++ free(match->regexps); ++ match->regexps = NULL; ++ match->nmemb = 0; ++ } ++} ++ ++static int init_annotationmatch(annotation_match_t *match, struct cnfarray *ar) { ++ DEFiRet; ++ ++ match->nmemb = ar->nmemb; ++ CHKmalloc(match->patterns = calloc(sizeof(uchar*), match->nmemb)); ++ CHKmalloc(match->regexps = calloc(sizeof(regex_t), match->nmemb)); ++ for(int jj = 0; jj < ar->nmemb; ++jj) { ++ int rexret = 0; ++ match->patterns[jj] = (uchar*)es_str2cstr(ar->arr[jj], NULL); ++ rexret = regexp.regcomp(&match->regexps[jj], ++ (char *)match->patterns[jj], REG_EXTENDED|REG_NOSUB); ++ if (0 != rexret) { ++ char errMsg[512]; ++ regexp.regerror(rexret, &match->regexps[jj], errMsg, sizeof(errMsg)); ++ iRet = RS_RET_CONFIG_ERROR; ++ errmsg.LogError(0, iRet, ++ "error: could not compile annotation_match string [%s]" ++ " into an extended regexp - %d: %s\n", ++ match->patterns[jj], rexret, errMsg); ++ break; ++ } ++ } ++finalize_it: ++ if (iRet) ++ free_annotationmatch(match); ++ RETiRet; ++} ++ ++static int copy_annotationmatch(annotation_match_t *src, annotation_match_t *dest) { ++ DEFiRet; ++ ++ dest->nmemb = src->nmemb; ++ CHKmalloc(dest->patterns = malloc(sizeof(uchar*) * dest->nmemb)); ++ CHKmalloc(dest->regexps = calloc(sizeof(regex_t), dest->nmemb)); ++ for(int jj = 0 ; jj < src->nmemb ; ++jj) { ++ CHKmalloc(dest->patterns[jj] = (uchar*)strdup((char *)src->patterns[jj])); ++ /* assumes was already successfully compiled */ ++ regexp.regcomp(&dest->regexps[jj], (char *)dest->patterns[jj], REG_EXTENDED|REG_NOSUB); ++ } ++finalize_it: ++ if (iRet) ++ free_annotationmatch(dest); ++ RETiRet; ++} ++ ++/* takes a hash of annotations and returns another json object hash containing only the ++ * keys that match - this logic is taken directly from fluent-plugin-kubernetes_metadata_filter ++ * except that we do not add the key multiple times to the object to be returned ++ */ ++static struct json_object *match_annotations(annotation_match_t *match, ++ struct json_object *annotations) { ++ struct json_object *ret = NULL; ++ ++ for (int jj = 0; jj < match->nmemb; ++jj) { ++ struct json_object_iterator it = json_object_iter_begin(annotations); ++ struct json_object_iterator itEnd = json_object_iter_end(annotations); ++ for (;!json_object_iter_equal(&it, &itEnd); json_object_iter_next(&it)) { ++ const char *const key = json_object_iter_peek_name(&it); ++ if (!ret || !fjson_object_object_get_ex(ret, key, NULL)) { ++ if (!regexp.regexec(&match->regexps[jj], key, 0, NULL, 0)) { ++ if (!ret) { ++ ret = json_object_new_object(); ++ } ++ json_object_object_add(ret, key, ++ json_object_get(json_object_iter_peek_value(&it))); ++ } ++ } ++ } ++ } ++ return ret; ++} ++ ++/* This will take a hash of labels or annotations and will de_dot the keys. ++ * It will return a brand new hash. AFAICT, there is no safe way to ++ * iterate over the hash while modifying it in place. ++ */ ++static struct json_object *de_dot_json_object(struct json_object *jobj, ++ const char *delim, size_t delim_len) { ++ struct json_object *ret = NULL; ++ struct json_object_iterator it = json_object_iter_begin(jobj); ++ struct json_object_iterator itEnd = json_object_iter_end(jobj); ++ es_str_t *new_es_key = NULL; ++ DEFiRet; ++ ++ ret = json_object_new_object(); ++ while (!json_object_iter_equal(&it, &itEnd)) { ++ const char *const key = json_object_iter_peek_name(&it); ++ const char *cc = strstr(key, "."); ++ if (NULL == cc) { ++ json_object_object_add(ret, key, ++ json_object_get(json_object_iter_peek_value(&it))); ++ } else { ++ char *new_key = NULL; ++ const char *prevcc = key; ++ new_es_key = es_newStrFromCStr(key, (es_size_t)(cc-prevcc)); ++ while (cc) { ++ if (es_addBuf(&new_es_key, (char *)delim, (es_size_t)delim_len)) ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ cc += 1; /* one past . */ ++ prevcc = cc; /* beginning of next substring */ ++ if ((cc = strstr(prevcc, ".")) || (cc = strchr(prevcc, '\0'))) { ++ if (es_addBuf(&new_es_key, (char *)prevcc, (es_size_t)(cc-prevcc))) ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ if (!*cc) ++ cc = NULL; /* EOS - done */ ++ } ++ } ++ new_key = es_str2cstr(new_es_key, NULL); ++ es_deleteStr(new_es_key); ++ new_es_key = NULL; ++ json_object_object_add(ret, new_key, ++ json_object_get(json_object_iter_peek_value(&it))); ++ free(new_key); ++ } ++ json_object_iter_next(&it); ++ } ++finalize_it: ++ if (iRet != RS_RET_OK) { ++ json_object_put(ret); ++ ret = NULL; ++ } ++ if (new_es_key) ++ es_deleteStr(new_es_key); ++ return ret; ++} ++ ++/* given a "metadata" object field, do ++ * - make sure "annotations" field has only the matching keys ++ * - de_dot the "labels" and "annotations" fields keys ++ * This modifies the jMetadata object in place ++ */ ++static void parse_labels_annotations(struct json_object *jMetadata, ++ annotation_match_t *match, sbool de_dot, ++ const char *delim, size_t delim_len) { ++ struct json_object *jo = NULL; ++ ++ if (fjson_object_object_get_ex(jMetadata, "annotations", &jo)) { ++ if ((jo = match_annotations(match, jo))) ++ json_object_object_add(jMetadata, "annotations", jo); ++ else ++ json_object_object_del(jMetadata, "annotations"); ++ } ++ /* dedot labels and annotations */ ++ if (de_dot) { ++ struct json_object *jo2 = NULL; ++ if (fjson_object_object_get_ex(jMetadata, "annotations", &jo)) { ++ if ((jo2 = de_dot_json_object(jo, delim, delim_len))) { ++ json_object_object_add(jMetadata, "annotations", jo2); ++ } ++ } ++ if (fjson_object_object_get_ex(jMetadata, "labels", &jo)) { ++ if ((jo2 = de_dot_json_object(jo, delim, delim_len))) { ++ json_object_object_add(jMetadata, "labels", jo2); ++ } ++ } ++ } ++} ++ ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++static int array_to_rules(struct cnfarray *ar, char **rules) { ++ DEFiRet; ++ es_str_t *tmpstr = NULL; ++ es_size_t size = 0; ++ ++ if (rules == NULL) ++ FINALIZE; ++ *rules = NULL; ++ if (!ar->nmemb) ++ FINALIZE; ++ for (int jj = 0; jj < ar->nmemb; jj++) ++ size += es_strlen(ar->arr[jj]); ++ if (!size) ++ FINALIZE; ++ CHKmalloc(tmpstr = es_newStr(size)); ++ CHKiRet((es_addStr(&tmpstr, ar->arr[0]))); ++ CHKiRet((es_addBufConstcstr(&tmpstr, "\n"))); ++ for(int jj=1; jj < ar->nmemb; ++jj) { ++ CHKiRet((es_addStr(&tmpstr, ar->arr[jj]))); ++ CHKiRet((es_addBufConstcstr(&tmpstr, "\n"))); ++ } ++ CHKiRet((es_addBufConstcstr(&tmpstr, "\0"))); ++ CHKmalloc(*rules = es_str2cstr(tmpstr, NULL)); ++finalize_it: ++ if (tmpstr) { ++ es_deleteStr(tmpstr); ++ } ++ if (iRet != RS_RET_OK) { ++ free(*rules); ++ *rules = NULL; ++ } ++ RETiRet; ++} ++#endif ++ ++/* callback for liblognorm error messages */ ++static void ++errCallBack(void __attribute__((unused)) *cookie, const char *msg, ++ size_t __attribute__((unused)) lenMsg) ++{ ++ errmsg.LogError(0, RS_RET_ERR_LIBLOGNORM, "liblognorm error: %s", msg); ++} ++ ++static rsRetVal ++set_lnctx(ln_ctx *ctxln, char *instRules, uchar *instRulebase, char *modRules, uchar *modRulebase) ++{ ++ DEFiRet; ++ if (ctxln == NULL) ++ FINALIZE; ++ CHKmalloc(*ctxln = ln_initCtx()); ++ ln_setErrMsgCB(*ctxln, errCallBack, NULL); ++ if(instRules) { ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ if(ln_loadSamplesFromString(*ctxln, instRules) !=0) { ++ errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: normalization rules '%s' " ++ "could not be loaded", instRules); ++ ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); ++ } ++#else ++ (void)instRules; ++#endif ++ } else if(instRulebase) { ++ if(ln_loadSamples(*ctxln, (char*) instRulebase) != 0) { ++ errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: normalization rulebase '%s' " ++ "could not be loaded", instRulebase); ++ ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); ++ } ++ } else if(modRules) { ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ if(ln_loadSamplesFromString(*ctxln, modRules) !=0) { ++ errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: normalization rules '%s' " ++ "could not be loaded", modRules); ++ ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); ++ } ++#else ++ (void)modRules; ++#endif ++ } else if(modRulebase) { ++ if(ln_loadSamples(*ctxln, (char*) modRulebase) != 0) { ++ errmsg.LogError(0, RS_RET_NO_RULEBASE, "error: normalization rulebase '%s' " ++ "could not be loaded", modRulebase); ++ ABORT_FINALIZE(RS_RET_ERR_LIBLOGNORM_SAMPDB_LOAD); ++ } ++ } ++finalize_it: ++ if (iRet != RS_RET_OK){ ++ ln_exitCtx(*ctxln); ++ *ctxln = NULL; ++ } ++ RETiRet; ++} ++ ++BEGINbeginCnfLoad ++CODESTARTbeginCnfLoad ++ loadModConf = pModConf; ++ pModConf->pConf = pConf; ++ENDbeginCnfLoad ++ ++ ++BEGINsetModCnf ++ struct cnfparamvals *pvals = NULL; ++ int i; ++ FILE *fp; ++ int ret; ++CODESTARTsetModCnf ++ pvals = nvlstGetParams(lst, &modpblk, NULL); ++ if(pvals == NULL) { ++ errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "mmkubernetes: " ++ "error processing module config parameters [module(...)]"); ++ ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); ++ } ++ ++ if(Debug) { ++ dbgprintf("module (global) param blk for mmkubernetes:\n"); ++ cnfparamsPrint(&modpblk, pvals); ++ } ++ ++ loadModConf->de_dot = DFLT_DE_DOT; ++ for(i = 0 ; i < modpblk.nParams ; ++i) { ++ if(!pvals[i].bUsed) { ++ continue; ++ } else if(!strcmp(modpblk.descr[i].name, "kubernetesurl")) { ++ free(loadModConf->kubernetesUrl); ++ loadModConf->kubernetesUrl = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ } else if(!strcmp(modpblk.descr[i].name, "srcmetadatapath")) { ++ free(loadModConf->srcMetadataPath); ++ loadModConf->srcMetadataPath = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ /* todo: sanitize the path */ ++ } else if(!strcmp(modpblk.descr[i].name, "dstmetadatapath")) { ++ free(loadModConf->dstMetadataPath); ++ loadModConf->dstMetadataPath = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ /* todo: sanitize the path */ ++ } else if(!strcmp(modpblk.descr[i].name, "tls.cacert")) { ++ free(loadModConf->caCertFile); ++ loadModConf->caCertFile = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)loadModConf->caCertFile, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: certificate file %s couldn't be accessed: %s\n", ++ loadModConf->caCertFile, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++ } else if(!strcmp(modpblk.descr[i].name, "allowunsignedcerts")) { ++ loadModConf->allowUnsignedCerts = pvals[i].val.d.n; ++ } else if(!strcmp(modpblk.descr[i].name, "token")) { ++ free(loadModConf->token); ++ loadModConf->token = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ } else if(!strcmp(modpblk.descr[i].name, "tokenfile")) { ++ free(loadModConf->tokenFile); ++ loadModConf->tokenFile = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)loadModConf->tokenFile, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: token file %s couldn't be accessed: %s\n", ++ loadModConf->tokenFile, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++ } else if(!strcmp(modpblk.descr[i].name, "annotation_match")) { ++ free_annotationmatch(&loadModConf->annotation_match); ++ if ((ret = init_annotationmatch(&loadModConf->annotation_match, pvals[i].val.d.ar))) ++ ABORT_FINALIZE(ret); ++ } else if(!strcmp(modpblk.descr[i].name, "de_dot")) { ++ loadModConf->de_dot = pvals[i].val.d.n; ++ } else if(!strcmp(modpblk.descr[i].name, "de_dot_separator")) { ++ free(loadModConf->de_dot_separator); ++ loadModConf->de_dot_separator = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ } else if(!strcmp(modpblk.descr[i].name, "filenamerules")) { ++ free(loadModConf->fnRules); ++ CHKiRet((array_to_rules(pvals[i].val.d.ar, &loadModConf->fnRules))); ++#endif ++ } else if(!strcmp(modpblk.descr[i].name, "filenamerulebase")) { ++ free(loadModConf->fnRulebase); ++ loadModConf->fnRulebase = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)loadModConf->fnRulebase, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: filenamerulebase file %s couldn't be accessed: %s\n", ++ loadModConf->fnRulebase, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ } else if(!strcmp(modpblk.descr[i].name, "containerrules")) { ++ free(loadModConf->contRules); ++ CHKiRet((array_to_rules(pvals[i].val.d.ar, &loadModConf->contRules))); ++#endif ++ } else if(!strcmp(modpblk.descr[i].name, "containerrulebase")) { ++ free(loadModConf->contRulebase); ++ loadModConf->contRulebase = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)loadModConf->contRulebase, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: containerrulebase file %s couldn't be accessed: %s\n", ++ loadModConf->contRulebase, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++ } else { ++ dbgprintf("mmkubernetes: program error, non-handled " ++ "param '%s' in module() block\n", modpblk.descr[i].name); ++ /* todo: error message? */ ++ } ++ } ++ ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ if (loadModConf->fnRules && loadModConf->fnRulebase) { ++ errmsg.LogError(0, RS_RET_CONFIG_ERROR, ++ "mmkubernetes: only 1 of filenamerules or filenamerulebase may be used"); ++ ABORT_FINALIZE(RS_RET_CONFIG_ERROR); ++ } ++ if (loadModConf->contRules && loadModConf->contRulebase) { ++ errmsg.LogError(0, RS_RET_CONFIG_ERROR, ++ "mmkubernetes: only 1 of containerrules or containerrulebase may be used"); ++ ABORT_FINALIZE(RS_RET_CONFIG_ERROR); ++ } ++#endif ++ ++ /* set defaults */ ++ if(loadModConf->srcMetadataPath == NULL) ++ loadModConf->srcMetadataPath = (uchar *) strdup(DFLT_SRCMD_PATH); ++ if(loadModConf->dstMetadataPath == NULL) ++ loadModConf->dstMetadataPath = (uchar *) strdup(DFLT_DSTMD_PATH); ++ if(loadModConf->de_dot_separator == NULL) ++ loadModConf->de_dot_separator = (uchar *) strdup(DFLT_DE_DOT_SEPARATOR); ++ if(loadModConf->de_dot_separator) ++ loadModConf->de_dot_separator_len = strlen((const char *)loadModConf->de_dot_separator); ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ if (loadModConf->fnRules == NULL && loadModConf->fnRulebase == NULL) ++ loadModConf->fnRules = strdup(DFLT_FILENAME_LNRULES); ++ if (loadModConf->contRules == NULL && loadModConf->contRulebase == NULL) ++ loadModConf->contRules = strdup(DFLT_CONTAINER_LNRULES); ++#else ++ if (loadModConf->fnRulebase == NULL) ++ loadModConf->fnRulebase = (uchar *)strdup(DFLT_FILENAME_RULEBASE); ++ if (loadModConf->contRulebase == NULL) ++ loadModConf->contRulebase = (uchar *)strdup(DFLT_CONTAINER_RULEBASE); ++#endif ++ caches = calloc(1, sizeof(struct cache_s *)); ++ ++finalize_it: ++ if(pvals != NULL) ++ cnfparamvalsDestruct(pvals, &modpblk); ++ENDsetModCnf ++ ++ ++BEGINcreateInstance ++CODESTARTcreateInstance ++ENDcreateInstance ++ ++ ++BEGINfreeInstance ++CODESTARTfreeInstance ++ free(pData->kubernetesUrl); ++ msgPropDescrDestruct(pData->srcMetadataDescr); ++ free(pData->srcMetadataDescr); ++ free(pData->dstMetadataPath); ++ free(pData->caCertFile); ++ free(pData->token); ++ free(pData->tokenFile); ++ free(pData->fnRules); ++ free(pData->fnRulebase); ++ ln_exitCtx(pData->fnCtxln); ++ free(pData->contRules); ++ free(pData->contRulebase); ++ ln_exitCtx(pData->contCtxln); ++ free_annotationmatch(&pData->annotation_match); ++ free(pData->de_dot_separator); ++ msgPropDescrDestruct(pData->contNameDescr); ++ free(pData->contNameDescr); ++ msgPropDescrDestruct(pData->contIdFullDescr); ++ free(pData->contIdFullDescr); ++ENDfreeInstance ++ ++static size_t curlCB(char *data, size_t size, size_t nmemb, void *usrptr) ++{ ++ DEFiRet; ++ wrkrInstanceData_t *pWrkrData = (wrkrInstanceData_t *) usrptr; ++ char * buf; ++ size_t newlen; ++ ++ newlen = pWrkrData->curlRplyLen + size * nmemb; ++ CHKmalloc(buf = realloc(pWrkrData->curlRply, newlen)); ++ memcpy(buf + pWrkrData->curlRplyLen, data, size * nmemb); ++ pWrkrData->curlRply = buf; ++ pWrkrData->curlRplyLen = newlen; ++ ++finalize_it: ++ if (iRet != RS_RET_OK) { ++ return 0; ++ } ++ return size * nmemb; ++} ++ ++BEGINcreateWrkrInstance ++CODESTARTcreateWrkrInstance ++ CURL *ctx; ++ struct curl_slist *hdr = NULL; ++ char *tokenHdr = NULL; ++ FILE *fp = NULL; ++ char *token = NULL; ++ ++ hdr = curl_slist_append(hdr, "Content-Type: text/json; charset=utf-8"); ++ if (pWrkrData->pData->token) { ++ if ((-1 == asprintf(&tokenHdr, "Authorization: Bearer %s", pWrkrData->pData->token)) || ++ (!tokenHdr)) { ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ } ++ } else if (pWrkrData->pData->tokenFile) { ++ struct stat statbuf; ++ fp = fopen((const char*)pWrkrData->pData->tokenFile, "r"); ++ if (fp && !fstat(fileno(fp), &statbuf)) { ++ size_t bytesread; ++ CHKmalloc(token = malloc((statbuf.st_size+1)*sizeof(char))); ++ if (0 < (bytesread = fread(token, sizeof(char), statbuf.st_size, fp))) { ++ token[bytesread] = '\0'; ++ if ((-1 == asprintf(&tokenHdr, "Authorization: Bearer %s", token)) || ++ (!tokenHdr)) { ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ } ++ } ++ free(token); ++ token = NULL; ++ } ++ if (fp) { ++ fclose(fp); ++ fp = NULL; ++ } ++ } ++ if (tokenHdr) { ++ hdr = curl_slist_append(hdr, tokenHdr); ++ free(tokenHdr); ++ } ++ pWrkrData->curlHdr = hdr; ++ ctx = curl_easy_init(); ++ curl_easy_setopt(ctx, CURLOPT_HTTPHEADER, hdr); ++ curl_easy_setopt(ctx, CURLOPT_WRITEFUNCTION, curlCB); ++ curl_easy_setopt(ctx, CURLOPT_WRITEDATA, pWrkrData); ++ if(pWrkrData->pData->caCertFile) ++ curl_easy_setopt(ctx, CURLOPT_CAINFO, pWrkrData->pData->caCertFile); ++ if(pWrkrData->pData->allowUnsignedCerts) ++ curl_easy_setopt(ctx, CURLOPT_SSL_VERIFYPEER, 0); ++ ++ pWrkrData->curlCtx = ctx; ++finalize_it: ++ free(token); ++ if (fp) { ++ fclose(fp); ++ } ++ENDcreateWrkrInstance ++ ++ ++BEGINfreeWrkrInstance ++CODESTARTfreeWrkrInstance ++ curl_easy_cleanup(pWrkrData->curlCtx); ++ curl_slist_free_all(pWrkrData->curlHdr); ++ENDfreeWrkrInstance ++ ++ ++static struct cache_s *cacheNew(const uchar *url) ++{ ++ struct cache_s *cache; ++ ++ if (NULL == (cache = calloc(1, sizeof(struct cache_s)))) { ++ FINALIZE; ++ } ++ cache->kbUrl = url; ++ cache->mdHt = create_hashtable(100, hash_from_string, ++ key_equals_string, (void (*)(void *)) json_object_put); ++ cache->nsHt = create_hashtable(100, hash_from_string, ++ key_equals_string, (void (*)(void *)) json_object_put); ++ cache->cacheMtx = malloc(sizeof(pthread_mutex_t)); ++ if (!cache->mdHt || !cache->nsHt || !cache->cacheMtx) { ++ free (cache); ++ cache = NULL; ++ FINALIZE; ++ } ++ pthread_mutex_init(cache->cacheMtx, NULL); ++ ++finalize_it: ++ return cache; ++} ++ ++ ++static void cacheFree(struct cache_s *cache) ++{ ++ hashtable_destroy(cache->mdHt, 1); ++ hashtable_destroy(cache->nsHt, 1); ++ pthread_mutex_destroy(cache->cacheMtx); ++ free(cache->cacheMtx); ++ free(cache); ++} ++ ++ ++BEGINnewActInst ++ struct cnfparamvals *pvals = NULL; ++ int i; ++ FILE *fp; ++ char *rxstr = NULL; ++ char *srcMetadataPath = NULL; ++CODESTARTnewActInst ++ DBGPRINTF("newActInst (mmkubernetes)\n"); ++ ++ pvals = nvlstGetParams(lst, &actpblk, NULL); ++ if(pvals == NULL) { ++ errmsg.LogError(0, RS_RET_MISSING_CNFPARAMS, "mmkubernetes: " ++ "error processing config parameters [action(...)]"); ++ ABORT_FINALIZE(RS_RET_MISSING_CNFPARAMS); ++ } ++ ++ if(Debug) { ++ dbgprintf("action param blk in mmkubernetes:\n"); ++ cnfparamsPrint(&actpblk, pvals); ++ } ++ ++ CODE_STD_STRING_REQUESTnewActInst(1) ++ CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); ++ CHKiRet(createInstance(&pData)); ++ ++ pData->de_dot = loadModConf->de_dot; ++ pData->allowUnsignedCerts = loadModConf->allowUnsignedCerts; ++ for(i = 0 ; i < actpblk.nParams ; ++i) { ++ if(!pvals[i].bUsed) { ++ continue; ++ } else if(!strcmp(actpblk.descr[i].name, "kubernetesurl")) { ++ free(pData->kubernetesUrl); ++ pData->kubernetesUrl = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ } else if(!strcmp(actpblk.descr[i].name, "srcmetadatapath")) { ++ msgPropDescrDestruct(pData->srcMetadataDescr); ++ free(pData->srcMetadataDescr); ++ CHKmalloc(pData->srcMetadataDescr = MALLOC(sizeof(msgPropDescr_t))); ++ srcMetadataPath = es_str2cstr(pvals[i].val.d.estr, NULL); ++ CHKiRet(msgPropDescrFill(pData->srcMetadataDescr, (uchar *)srcMetadataPath, ++ strlen(srcMetadataPath))); ++ /* todo: sanitize the path */ ++ } else if(!strcmp(actpblk.descr[i].name, "dstmetadatapath")) { ++ free(pData->dstMetadataPath); ++ pData->dstMetadataPath = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ /* todo: sanitize the path */ ++ } else if(!strcmp(actpblk.descr[i].name, "tls.cacert")) { ++ free(pData->caCertFile); ++ pData->caCertFile = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)pData->caCertFile, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: certificate file %s couldn't be accessed: %s\n", ++ pData->caCertFile, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++ } else if(!strcmp(actpblk.descr[i].name, "allowunsignedcerts")) { ++ pData->allowUnsignedCerts = pvals[i].val.d.n; ++ } else if(!strcmp(actpblk.descr[i].name, "token")) { ++ free(pData->token); ++ pData->token = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ } else if(!strcmp(actpblk.descr[i].name, "tokenfile")) { ++ free(pData->tokenFile); ++ pData->tokenFile = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)pData->tokenFile, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: token file %s couldn't be accessed: %s\n", ++ pData->tokenFile, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++ } else if(!strcmp(actpblk.descr[i].name, "annotation_match")) { ++ free_annotationmatch(&pData->annotation_match); ++ if (RS_RET_OK != (iRet = init_annotationmatch(&pData->annotation_match, pvals[i].val.d.ar))) ++ ABORT_FINALIZE(iRet); ++ } else if(!strcmp(actpblk.descr[i].name, "de_dot")) { ++ pData->de_dot = pvals[i].val.d.n; ++ } else if(!strcmp(actpblk.descr[i].name, "de_dot_separator")) { ++ free(pData->de_dot_separator); ++ pData->de_dot_separator = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ } else if(!strcmp(modpblk.descr[i].name, "filenamerules")) { ++ free(pData->fnRules); ++ CHKiRet((array_to_rules(pvals[i].val.d.ar, &pData->fnRules))); ++#endif ++ } else if(!strcmp(modpblk.descr[i].name, "filenamerulebase")) { ++ free(pData->fnRulebase); ++ pData->fnRulebase = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)pData->fnRulebase, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: filenamerulebase file %s couldn't be accessed: %s\n", ++ pData->fnRulebase, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ } else if(!strcmp(modpblk.descr[i].name, "containerrules")) { ++ free(pData->contRules); ++ CHKiRet((array_to_rules(pvals[i].val.d.ar, &pData->contRules))); ++#endif ++ } else if(!strcmp(modpblk.descr[i].name, "containerrulebase")) { ++ free(pData->contRulebase); ++ pData->contRulebase = (uchar *) es_str2cstr(pvals[i].val.d.estr, NULL); ++ fp = fopen((const char*)pData->contRulebase, "r"); ++ if(fp == NULL) { ++ char errStr[1024]; ++ rs_strerror_r(errno, errStr, sizeof(errStr)); ++ iRet = RS_RET_NO_FILE_ACCESS; ++ errmsg.LogError(0, iRet, ++ "error: containerrulebase file %s couldn't be accessed: %s\n", ++ pData->contRulebase, errStr); ++ ABORT_FINALIZE(iRet); ++ } else { ++ fclose(fp); ++ } ++ } else { ++ dbgprintf("mmkubernetes: program error, non-handled " ++ "param '%s' in action() block\n", actpblk.descr[i].name); ++ /* todo: error message? */ ++ } ++ } ++ ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ if (pData->fnRules && pData->fnRulebase) { ++ errmsg.LogError(0, RS_RET_CONFIG_ERROR, ++ "mmkubernetes: only 1 of filenamerules or filenamerulebase may be used"); ++ ABORT_FINALIZE(RS_RET_CONFIG_ERROR); ++ } ++ if (pData->contRules && pData->contRulebase) { ++ errmsg.LogError(0, RS_RET_CONFIG_ERROR, ++ "mmkubernetes: only 1 of containerrules or containerrulebase may be used"); ++ ABORT_FINALIZE(RS_RET_CONFIG_ERROR); ++ } ++#endif ++ CHKiRet(set_lnctx(&pData->fnCtxln, pData->fnRules, pData->fnRulebase, ++ loadModConf->fnRules, loadModConf->fnRulebase)); ++ CHKiRet(set_lnctx(&pData->contCtxln, pData->contRules, pData->contRulebase, ++ loadModConf->contRules, loadModConf->contRulebase)); ++ ++ if(pData->kubernetesUrl == NULL) { ++ if(loadModConf->kubernetesUrl == NULL) { ++ CHKmalloc(pData->kubernetesUrl = (uchar *) strdup(DFLT_KUBERNETES_URL)); ++ } else { ++ CHKmalloc(pData->kubernetesUrl = (uchar *) strdup((char *) loadModConf->kubernetesUrl)); ++ } ++ } ++ if(pData->srcMetadataDescr == NULL) { ++ CHKmalloc(pData->srcMetadataDescr = MALLOC(sizeof(msgPropDescr_t))); ++ CHKiRet(msgPropDescrFill(pData->srcMetadataDescr, loadModConf->srcMetadataPath, ++ strlen((char *)loadModConf->srcMetadataPath))); ++ } ++ if(pData->dstMetadataPath == NULL) ++ pData->dstMetadataPath = (uchar *) strdup((char *) loadModConf->dstMetadataPath); ++ if(pData->caCertFile == NULL && loadModConf->caCertFile) ++ pData->caCertFile = (uchar *) strdup((char *) loadModConf->caCertFile); ++ if(pData->token == NULL && loadModConf->token) ++ pData->token = (uchar *) strdup((char *) loadModConf->token); ++ if(pData->tokenFile == NULL && loadModConf->tokenFile) ++ pData->tokenFile = (uchar *) strdup((char *) loadModConf->tokenFile); ++ if(pData->de_dot_separator == NULL && loadModConf->de_dot_separator) ++ pData->de_dot_separator = (uchar *) strdup((char *) loadModConf->de_dot_separator); ++ if((pData->annotation_match.nmemb == 0) && (loadModConf->annotation_match.nmemb > 0)) ++ copy_annotationmatch(&loadModConf->annotation_match, &pData->annotation_match); ++ ++ if(pData->de_dot_separator) ++ pData->de_dot_separator_len = strlen((const char *)pData->de_dot_separator); ++ ++ CHKmalloc(pData->contNameDescr = MALLOC(sizeof(msgPropDescr_t))); ++ CHKiRet(msgPropDescrFill(pData->contNameDescr, (uchar*) DFLT_CONTAINER_NAME, ++ strlen(DFLT_CONTAINER_NAME))); ++ CHKmalloc(pData->contIdFullDescr = MALLOC(sizeof(msgPropDescr_t))); ++ CHKiRet(msgPropDescrFill(pData->contIdFullDescr, (uchar*) DFLT_CONTAINER_ID_FULL, ++ strlen(DFLT_CONTAINER_NAME))); ++ ++ /* get the cache for this url */ ++ for(i = 0; caches[i] != NULL; i++) { ++ if(!strcmp((char *) pData->kubernetesUrl, (char *) caches[i]->kbUrl)) ++ break; ++ } ++ if(caches[i] != NULL) { ++ pData->cache = caches[i]; ++ } else { ++ CHKmalloc(pData->cache = cacheNew(pData->kubernetesUrl)); ++ ++ CHKmalloc(caches = realloc(caches, (i + 2) * sizeof(struct cache_s *))); ++ caches[i] = pData->cache; ++ caches[i + 1] = NULL; ++ } ++CODE_STD_FINALIZERnewActInst ++ if(pvals != NULL) ++ cnfparamvalsDestruct(pvals, &actpblk); ++ free(rxstr); ++ free(srcMetadataPath); ++ENDnewActInst ++ ++ ++/* legacy config format is not supported */ ++BEGINparseSelectorAct ++CODESTARTparseSelectorAct ++CODE_STD_STRING_REQUESTparseSelectorAct(1) ++ if(strncmp((char *) p, ":mmkubernetes:", sizeof(":mmkubernetes:") - 1)) { ++ errmsg.LogError(0, RS_RET_LEGA_ACT_NOT_SUPPORTED, ++ "mmkubernetes supports only v6+ config format, use: " ++ "action(type=\"mmkubernetes\" ...)"); ++ } ++ ABORT_FINALIZE(RS_RET_CONFLINE_UNPROCESSED); ++CODE_STD_FINALIZERparseSelectorAct ++ENDparseSelectorAct ++ ++ ++BEGINendCnfLoad ++CODESTARTendCnfLoad ++ENDendCnfLoad ++ ++ ++BEGINcheckCnf ++CODESTARTcheckCnf ++ENDcheckCnf ++ ++ ++BEGINactivateCnf ++CODESTARTactivateCnf ++ runModConf = pModConf; ++ENDactivateCnf ++ ++ ++BEGINfreeCnf ++CODESTARTfreeCnf ++ int i; ++ ++ free(pModConf->kubernetesUrl); ++ free(pModConf->srcMetadataPath); ++ free(pModConf->dstMetadataPath); ++ free(pModConf->caCertFile); ++ free(pModConf->token); ++ free(pModConf->tokenFile); ++ free(pModConf->de_dot_separator); ++ free(pModConf->fnRules); ++ free(pModConf->fnRulebase); ++ free(pModConf->contRules); ++ free(pModConf->contRulebase); ++ free_annotationmatch(&pModConf->annotation_match); ++ for(i = 0; caches[i] != NULL; i++) ++ cacheFree(caches[i]); ++ free(caches); ++ENDfreeCnf ++ ++ ++BEGINdbgPrintInstInfo ++CODESTARTdbgPrintInstInfo ++ dbgprintf("mmkubernetes\n"); ++ dbgprintf("\tkubernetesUrl='%s'\n", pData->kubernetesUrl); ++ dbgprintf("\tsrcMetadataPath='%s'\n", pData->srcMetadataDescr->name); ++ dbgprintf("\tdstMetadataPath='%s'\n", pData->dstMetadataPath); ++ dbgprintf("\ttls.cacert='%s'\n", pData->caCertFile); ++ dbgprintf("\tallowUnsignedCerts='%d'\n", pData->allowUnsignedCerts); ++ dbgprintf("\ttoken='%s'\n", pData->token); ++ dbgprintf("\ttokenFile='%s'\n", pData->tokenFile); ++ dbgprintf("\tde_dot='%d'\n", pData->de_dot); ++ dbgprintf("\tde_dot_separator='%s'\n", pData->de_dot_separator); ++ dbgprintf("\tfilenamerulebase='%s'\n", pData->fnRulebase); ++ dbgprintf("\tcontainerrulebase='%s'\n", pData->contRulebase); ++#if HAVE_LOADSAMPLESFROMSTRING == 1 ++ dbgprintf("\tfilenamerules='%s'\n", pData->fnRules); ++ dbgprintf("\tcontainerrules='%s'\n", pData->contRules); ++#endif ++ENDdbgPrintInstInfo ++ ++ ++BEGINtryResume ++CODESTARTtryResume ++ENDtryResume ++ ++static rsRetVal ++extractMsgMetadata(smsg_t *pMsg, instanceData *pData, struct json_object **json) ++{ ++ DEFiRet; ++ uchar *filename = NULL, *container_name = NULL, *container_id_full = NULL; ++ rs_size_t fnLen, container_name_len, container_id_full_len; ++ unsigned short freeFn = 0, free_container_name = 0, free_container_id_full = 0; ++ int lnret; ++ struct json_object *cnid = NULL; ++ ++ if (!json) ++ FINALIZE; ++ *json = NULL; ++ /* extract metadata from the CONTAINER_NAME field and see if CONTAINER_ID_FULL is present */ ++ container_name = MsgGetProp(pMsg, NULL, pData->contNameDescr, ++ &container_name_len, &free_container_name, NULL); ++ container_id_full = MsgGetProp( ++ pMsg, NULL, pData->contIdFullDescr, &container_id_full_len, &free_container_id_full, NULL); ++ ++ if (container_name && container_id_full && container_name_len && container_id_full_len) { ++ dbgprintf("mmkubernetes: CONTAINER_NAME: '%s' CONTAINER_ID_FULL: '%s'.\n", ++ container_name, container_id_full); ++ if ((lnret = ln_normalize(pData->contCtxln, (char*)container_name, ++ container_name_len, json))) { ++ if (LN_WRONGPARSER != lnret) { ++ LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: error parsing container_name [%s]: [%d]", ++ container_name, lnret); ++ ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ /* else assume parser didn't find a match and fall through */ ++ } else if (fjson_object_object_get_ex(*json, "pod_name", NULL) && ++ fjson_object_object_get_ex(*json, "namespace_name", NULL) && ++ fjson_object_object_get_ex(*json, "container_name", NULL)) { ++ /* if we have fields for pod name, namespace name, container name, ++ * and container id, we are good to go */ ++ /* add field for container id */ ++ json_object_object_add(*json, "container_id", ++ json_object_new_string_len((const char *)container_id_full, ++ container_id_full_len)); ++ ABORT_FINALIZE(RS_RET_OK); ++ } ++ } ++ ++ /* extract metadata from the file name */ ++ filename = MsgGetProp(pMsg, NULL, pData->srcMetadataDescr, &fnLen, &freeFn, NULL); ++ if((filename == NULL) || (fnLen == 0)) ++ ABORT_FINALIZE(RS_RET_NOT_FOUND); ++ ++ dbgprintf("mmkubernetes: filename: '%s' len %d.\n", filename, fnLen); ++ if ((lnret = ln_normalize(pData->fnCtxln, (char*)filename, fnLen, json))) { ++ if (LN_WRONGPARSER != lnret) { ++ LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: error parsing container_name [%s]: [%d]", ++ filename, lnret); ++ ++ ABORT_FINALIZE(RS_RET_ERR); ++ } else { ++ /* no match */ ++ ABORT_FINALIZE(RS_RET_NOT_FOUND); ++ } ++} ++ /* if we have fields for pod name, namespace name, container name, ++ * and container id, we are good to go */ ++ if (fjson_object_object_get_ex(*json, "pod_name", NULL) && ++ fjson_object_object_get_ex(*json, "namespace_name", NULL) && ++ fjson_object_object_get_ex(*json, "container_name_and_id", &cnid)) { ++ /* parse container_name_and_id into container_name and container_id */ ++ const char *container_name_and_id = json_object_get_string(cnid); ++ const char *last_dash = NULL; ++ if (container_name_and_id && (last_dash = strrchr(container_name_and_id, '-')) && ++ *(last_dash + 1) && (last_dash != container_name_and_id)) { ++ json_object_object_add(*json, "container_name", ++ json_object_new_string_len(container_name_and_id, ++ (int)(last_dash-container_name_and_id))); ++ json_object_object_add(*json, "container_id", ++ json_object_new_string(last_dash + 1)); ++ ABORT_FINALIZE(RS_RET_OK); ++ } ++ } ++ ABORT_FINALIZE(RS_RET_NOT_FOUND); ++finalize_it: ++ if(freeFn) ++ free(filename); ++ if (free_container_name) ++ free(container_name); ++ if (free_container_id_full) ++ free(container_id_full); ++ if (iRet != RS_RET_OK) { ++ json_object_put(*json); ++ *json = NULL; ++ } ++ RETiRet; ++} ++ ++ ++static rsRetVal ++queryKB(wrkrInstanceData_t *pWrkrData, char *url, struct json_object **rply) ++{ ++ DEFiRet; ++ CURLcode ccode; ++ struct json_tokener *jt = NULL; ++ struct json_object *jo; ++ long resp_code = 400; ++ ++ /* query kubernetes for pod info */ ++ ccode = curl_easy_setopt(pWrkrData->curlCtx, CURLOPT_URL, url); ++ if(ccode != CURLE_OK) ++ ABORT_FINALIZE(RS_RET_ERR); ++ if(CURLE_OK != (ccode = curl_easy_perform(pWrkrData->curlCtx))) { ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: failed to connect to [%s] - %d:%s\n", ++ url, ccode, curl_easy_strerror(ccode)); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ if(CURLE_OK != (ccode = curl_easy_getinfo(pWrkrData->curlCtx, ++ CURLINFO_RESPONSE_CODE, &resp_code))) { ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: could not get response code from query to [%s] - %d:%s\n", ++ url, ccode, curl_easy_strerror(ccode)); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ if(resp_code == 401) { ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: Unauthorized: not allowed to view url - " ++ "check token/auth credentials [%s]\n", ++ url); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ if(resp_code == 403) { ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: Forbidden: no access - " ++ "check permissions to view url [%s]\n", ++ url); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ if(resp_code == 404) { ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: Not Found: the resource does not exist at url [%s]\n", ++ url); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ if(resp_code == 429) { ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: Too Many Requests: the server is too heavily loaded " ++ "to provide the data for the requested url [%s]\n", ++ url); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ if(resp_code != 200) { ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_ERR, ++ "mmkubernetes: server returned unexpected code [%ld] for url [%s]\n", ++ resp_code, url); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ /* parse retrieved data */ ++ jt = json_tokener_new(); ++ json_tokener_reset(jt); ++ jo = json_tokener_parse_ex(jt, pWrkrData->curlRply, pWrkrData->curlRplyLen); ++ json_tokener_free(jt); ++ if(!json_object_is_type(jo, json_type_object)) { ++ json_object_put(jo); ++ jo = NULL; ++ errmsg.LogMsg(0, RS_RET_JSON_PARSE_ERR, LOG_INFO, ++ "mmkubernetes: unable to parse string as JSON:[%.*s]\n", ++ (int)pWrkrData->curlRplyLen, pWrkrData->curlRply); ++ ABORT_FINALIZE(RS_RET_JSON_PARSE_ERR); ++ } ++ ++ dbgprintf("mmkubernetes: queryKB reply:\n%s\n", ++ json_object_to_json_string_ext(jo, JSON_C_TO_STRING_PRETTY)); ++ ++ *rply = jo; ++ ++finalize_it: ++ if(pWrkrData->curlRply != NULL) { ++ free(pWrkrData->curlRply); ++ pWrkrData->curlRply = NULL; ++ pWrkrData->curlRplyLen = 0; ++ } ++ RETiRet; ++} ++ ++ ++/* versions < 8.16.0 don't support BEGINdoAction_NoStrings */ ++#if defined(BEGINdoAction_NoStrings) ++BEGINdoAction_NoStrings ++ smsg_t **ppMsg = (smsg_t **) pMsgData; ++ smsg_t *pMsg = ppMsg[0]; ++#else ++BEGINdoAction ++ smsg_t *pMsg = (smsg_t*) ppString[0]; ++#endif ++ const char *podName = NULL, *ns = NULL, *containerName = NULL, ++ *containerID = NULL; ++ char *mdKey = NULL; ++ struct json_object *jMetadata = NULL, *jMetadataCopy = NULL, *jMsgMeta = NULL, ++ *jo = NULL; ++ int add_ns_metadata = 0; ++CODESTARTdoAction ++ CHKiRet_Hdlr(extractMsgMetadata(pMsg, pWrkrData->pData, &jMsgMeta)) { ++ ABORT_FINALIZE((iRet == RS_RET_NOT_FOUND) ? RS_RET_OK : iRet); ++ } ++ ++ if (fjson_object_object_get_ex(jMsgMeta, "pod_name", &jo)) ++ podName = json_object_get_string(jo); ++ if (fjson_object_object_get_ex(jMsgMeta, "namespace_name", &jo)) ++ ns = json_object_get_string(jo); ++ if (fjson_object_object_get_ex(jMsgMeta, "container_name", &jo)) ++ containerName = json_object_get_string(jo); ++ if (fjson_object_object_get_ex(jMsgMeta, "container_id", &jo)) ++ containerID = json_object_get_string(jo); ++ assert(podName != NULL); ++ assert(ns != NULL); ++ assert(containerName != NULL); ++ assert(containerID != NULL); ++ ++ dbgprintf("mmkubernetes:\n podName: '%s'\n namespace: '%s'\n containerName: '%s'\n" ++ " containerID: '%s'\n", podName, ns, containerName, containerID); ++ ++ /* check cache for metadata */ ++ if ((-1 == asprintf(&mdKey, "%s_%s_%s", ns, podName, containerName)) || ++ (!mdKey)) { ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ } ++ pthread_mutex_lock(pWrkrData->pData->cache->cacheMtx); ++ jMetadata = hashtable_search(pWrkrData->pData->cache->mdHt, mdKey); ++ ++ if(jMetadata == NULL) { ++ char *url = NULL; ++ struct json_object *jReply = NULL, *jo2 = NULL, *jNsMeta = NULL, *jPodData = NULL; ++ ++ /* check cache for namespace metadata */ ++ jNsMeta = hashtable_search(pWrkrData->pData->cache->nsHt, (char *)ns); ++ ++ if(jNsMeta == NULL) { ++ /* query kubernetes for namespace info */ ++ /* todo: move url definitions elsewhere */ ++ if ((-1 == asprintf(&url, "%s/api/v1/namespaces/%s", ++ (char *) pWrkrData->pData->kubernetesUrl, ns)) || ++ (!url)) { ++ pthread_mutex_unlock(pWrkrData->pData->cache->cacheMtx); ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ } ++ iRet = queryKB(pWrkrData, url, &jReply); ++ free(url); ++ /* todo: implement support for the .orphaned namespace */ ++ if (iRet != RS_RET_OK) { ++ json_object_put(jReply); ++ jReply = NULL; ++ pthread_mutex_unlock(pWrkrData->pData->cache->cacheMtx); ++ FINALIZE; ++ } ++ ++ if(fjson_object_object_get_ex(jReply, "metadata", &jNsMeta)) { ++ jNsMeta = json_object_get(jNsMeta); ++ parse_labels_annotations(jNsMeta, &pWrkrData->pData->annotation_match, ++ pWrkrData->pData->de_dot, ++ (const char *)pWrkrData->pData->de_dot_separator, ++ pWrkrData->pData->de_dot_separator_len); ++ add_ns_metadata = 1; ++ } else { ++ /* namespace with no metadata??? */ ++ errmsg.LogMsg(0, RS_RET_ERR, LOG_INFO, ++ "mmkubernetes: namespace [%s] has no metadata!\n", ns); ++ jNsMeta = NULL; ++ } ++ ++ json_object_put(jReply); ++ jReply = NULL; ++ } ++ ++ if ((-1 == asprintf(&url, "%s/api/v1/namespaces/%s/pods/%s", ++ (char *) pWrkrData->pData->kubernetesUrl, ns, podName)) || ++ (!url)) { ++ pthread_mutex_unlock(pWrkrData->pData->cache->cacheMtx); ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ } ++ iRet = queryKB(pWrkrData, url, &jReply); ++ free(url); ++ if(iRet != RS_RET_OK) { ++ if(jNsMeta && add_ns_metadata) { ++ hashtable_insert(pWrkrData->pData->cache->nsHt, strdup(ns), jNsMeta); ++ } ++ json_object_put(jReply); ++ jReply = NULL; ++ pthread_mutex_unlock(pWrkrData->pData->cache->cacheMtx); ++ FINALIZE; ++ } ++ ++ jo = json_object_new_object(); ++ if(jNsMeta && fjson_object_object_get_ex(jNsMeta, "uid", &jo2)) ++ json_object_object_add(jo, "namespace_id", json_object_get(jo2)); ++ if(jNsMeta && fjson_object_object_get_ex(jNsMeta, "labels", &jo2)) ++ json_object_object_add(jo, "namespace_labels", json_object_get(jo2)); ++ if(jNsMeta && fjson_object_object_get_ex(jNsMeta, "annotations", &jo2)) ++ json_object_object_add(jo, "namespace_annotations", json_object_get(jo2)); ++ if(jNsMeta && fjson_object_object_get_ex(jNsMeta, "creationTimestamp", &jo2)) ++ json_object_object_add(jo, "creation_timestamp", json_object_get(jo2)); ++ if(fjson_object_object_get_ex(jReply, "metadata", &jPodData)) { ++ if(fjson_object_object_get_ex(jPodData, "uid", &jo2)) ++ json_object_object_add(jo, "pod_id", json_object_get(jo2)); ++ parse_labels_annotations(jPodData, &pWrkrData->pData->annotation_match, ++ pWrkrData->pData->de_dot, ++ (const char *)pWrkrData->pData->de_dot_separator, ++ pWrkrData->pData->de_dot_separator_len); ++ if(fjson_object_object_get_ex(jPodData, "annotations", &jo2)) ++ json_object_object_add(jo, "annotations", json_object_get(jo2)); ++ if(fjson_object_object_get_ex(jPodData, "labels", &jo2)) ++ json_object_object_add(jo, "labels", json_object_get(jo2)); ++ } ++ if(fjson_object_object_get_ex(jReply, "spec", &jPodData)) { ++ if(fjson_object_object_get_ex(jPodData, "nodeName", &jo2)) { ++ json_object_object_add(jo, "host", json_object_get(jo2)); ++ } ++ } ++ json_object_put(jReply); ++ jReply = NULL; ++ ++ if (fjson_object_object_get_ex(jMsgMeta, "pod_name", &jo2)) ++ json_object_object_add(jo, "pod_name", json_object_get(jo2)); ++ if (fjson_object_object_get_ex(jMsgMeta, "namespace_name", &jo2)) ++ json_object_object_add(jo, "namespace_name", json_object_get(jo2)); ++ if (fjson_object_object_get_ex(jMsgMeta, "container_name", &jo2)) ++ json_object_object_add(jo, "container_name", json_object_get(jo2)); ++ json_object_object_add(jo, "master_url", ++ json_object_new_string((const char *)pWrkrData->pData->kubernetesUrl)); ++ jMetadata = json_object_new_object(); ++ json_object_object_add(jMetadata, "kubernetes", jo); ++ jo = json_object_new_object(); ++ if (fjson_object_object_get_ex(jMsgMeta, "container_id", &jo2)) ++ json_object_object_add(jo, "container_id", json_object_get(jo2)); ++ json_object_object_add(jMetadata, "docker", jo); ++ ++ hashtable_insert(pWrkrData->pData->cache->mdHt, mdKey, jMetadata); ++ mdKey = NULL; ++ if(jNsMeta && add_ns_metadata) { ++ hashtable_insert(pWrkrData->pData->cache->nsHt, strdup(ns), jNsMeta); ++ ns = NULL; ++ } ++ } ++ ++ /* make a copy of the metadata for the msg to own */ ++ /* todo: use json_object_deep_copy when implementation available in libfastjson */ ++ /* yes, this is expensive - but there is no other way to make this thread safe - we ++ * can't allow the msg to have a shared pointer to an element inside the cache, ++ * outside of the cache lock ++ */ ++ jMetadataCopy = json_tokener_parse(json_object_get_string(jMetadata)); ++ pthread_mutex_unlock(pWrkrData->pData->cache->cacheMtx); ++ /* the +1 is there to skip the leading '$' */ ++ msgAddJSON(pMsg, (uchar *) pWrkrData->pData->dstMetadataPath + 1, jMetadataCopy, 0, 0); ++ ++finalize_it: ++ json_object_put(jMsgMeta); ++ free(mdKey); ++ENDdoAction ++ ++ ++BEGINisCompatibleWithFeature ++CODESTARTisCompatibleWithFeature ++ENDisCompatibleWithFeature ++ ++ ++/* all the macros bellow have to be in a specific order */ ++BEGINmodExit ++CODESTARTmodExit ++ curl_global_cleanup(); ++ ++ objRelease(regexp, LM_REGEXP_FILENAME); ++ objRelease(errmsg, CORE_COMPONENT); ++ENDmodExit ++ ++ ++BEGINqueryEtryPt ++CODESTARTqueryEtryPt ++CODEqueryEtryPt_STD_OMOD_QUERIES ++CODEqueryEtryPt_STD_OMOD8_QUERIES ++CODEqueryEtryPt_STD_CONF2_QUERIES ++CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES ++CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES ++ENDqueryEtryPt ++ ++ ++BEGINmodInit() ++CODESTARTmodInit ++ *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ ++CODEmodInit_QueryRegCFSLineHdlr ++ DBGPRINTF("mmkubernetes: module compiled with rsyslog version %s.\n", VERSION); ++ CHKiRet(objUse(errmsg, CORE_COMPONENT)); ++ CHKiRet(objUse(regexp, LM_REGEXP_FILENAME)); ++ ++ /* CURL_GLOBAL_ALL initializes more than is needed but the ++ * libcurl documentation discourages use of other values ++ */ ++ curl_global_init(CURL_GLOBAL_ALL); ++ENDmodInit +diff --git a/contrib/mmkubernetes/sample.conf b/contrib/mmkubernetes/sample.conf +new file mode 100644 +index 000000000..4c400ed51 +--- /dev/null ++++ b/contrib/mmkubernetes/sample.conf +@@ -0,0 +1,7 @@ ++module(load="mmkubernetes") # see docs for all module and action parameters ++ ++# $!metadata!filename added by imfile using addmetadata="on" ++# e.g. input(type="imfile" file="/var/log/containers/*.log" tag="kubernetes" addmetadata="on") ++# $!CONTAINER_NAME and $!CONTAINER_ID_FULL added by imjournal ++ ++action(type="mmkubernetes") +-- +2.14.4 + diff --git a/SOURCES/rsyslog-8.24.0-rhbz1545582-imjournal-duplicates.patch b/SOURCES/rsyslog-8.24.0-rhbz1545582-imjournal-duplicates.patch deleted file mode 100644 index 1b392c4..0000000 --- a/SOURCES/rsyslog-8.24.0-rhbz1545582-imjournal-duplicates.patch +++ /dev/null @@ -1,318 +0,0 @@ -From: Jiri Vymazal -Date: Wed, 14 Mar 2018 90:05:01 -0500 - -modification and merge of below patches for RHEL consumers, -also modified journal invalidate/rotation handling to keep possibility -to continue after switch of persistent journal -original: -% -%From a99f9b4b42d261c384aee09306fc421df2cca7a5 Mon Sep 17 00:00:00 2001 -%From: Peter Portante -%Date: Wed, 24 Jan 2018 19:34:41 -0500 -%Subject: [PATCH] Proposed fix for handling journal correctly -% -%The fix is to immediately setup the inotify file descriptor via -%`sd_journal_get_fd()` right after a journal open, and then -%periodically call `sd_journal_process()` to give the client API -%library a chance to detect deleted journal files on disk that need to -%be closed so they can be properly erased by the file system. -% -%We remove the open/close dance and simplify that code as a result. -% -%Fixes issue #2436. -and also: -%From 27f96c84d34ee000fbb5d45b00233f2ec3cf2d8a Mon Sep 17 00:00:00 2001 -%From: Rainer Gerhards -%Date: Tue, 24 Oct 2017 16:14:13 +0200 -%Subject: [PATCH] imjournal bugfix: do not disable itself on error -% -%If some functions calls inside the main loop failed, imjournal exited -%with an error code, actually disabling all logging from the journal. -%This was probably never intended. -% -%This patch makes imjournal recover the situation instead. -% -%closes https://github.com/rsyslog/rsyslog/issues/1895 ---- - plugins/imjournal/imjournal.c | 206 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------------------------------------------------ - 1 file changed, 104 insertions(+), 102 deletions(-) - ---- a/plugins/imjournal/imjournal.c -+++ b/plugins/imjournal/imjournal.c -@@ -114,6 +114,10 @@ /* module-global parameters */ - static const char *pid_field_name; /* read-only after startup */ - static ratelimit_t *ratelimiter = NULL; - static sd_journal *j; -+static int j_inotify_fd; -+static char *last_cursor = NULL; -+ -+#define J_PROCESS_PERIOD 1024 /* Call sd_journal_process() every 1,024 records */ - - static rsRetVal persistJournalState(void); - static rsRetVal loadJournalState(void); -@@ -123,6 +127,14 @@ openJournal(sd_journal** jj) - - if (sd_journal_open(jj, SD_JOURNAL_LOCAL_ONLY) < 0) - iRet = RS_RET_IO_ERROR; -+ int r; -+ -+ if ((r = sd_journal_get_fd(j)) < 0) { -+ errmsg.LogError(-r, RS_RET_IO_ERROR, "imjournal: sd_journal_get_fd() failed"); -+ iRet = RS_RET_IO_ERROR; -+ } else { -+ j_inotify_fd = r; -+ } - RETiRet; - } - -@@ -132,6 +144,7 @@ closeJournal(sd_journal** jj) - persistJournalState(); - } - sd_journal_close(*jj); -+ j_inotify_fd = 0; - } - - -@@ -262,6 +275,7 @@ readjournal(void) - char *message = NULL; - char *sys_iden = NULL; - char *sys_iden_help = NULL; -+ char *c = NULL; - - const void *get; - const void *pidget; -@@ -393,6 +407,12 @@ readjournal(void) - tv.tv_usec = timestamp % 1000000; - } - -+ sd_journal_get_cursor(j, &c); -+ if (c) { -+ free(last_cursor); -+ last_cursor = c; -+ } -+ - /* submit message */ - enqMsg((uchar *)message, (uchar *) sys_iden_help, facility, severity, &tv, json, 0); - -@@ -413,44 +433,41 @@ persistJournalState (void) - DEFiRet; - FILE *sf; /* state file */ - char tmp_sf[MAXFNAME]; -- char *cursor; -- int ret = 0; -+ int r = 0; - -- /* On success, sd_journal_get_cursor() returns 1 in systemd -- 197 or older and 0 in systemd 198 or newer */ -- if ((ret = sd_journal_get_cursor(j, &cursor)) >= 0) { -- /* we create a temporary name by adding a ".tmp" -- * suffix to the end of our state file's name -- */ -- snprintf(tmp_sf, sizeof(tmp_sf), "%s.tmp", cs.stateFile); -- if ((sf = fopen(tmp_sf, "wb")) != NULL) { -- if (fprintf(sf, "%s", cursor) < 0) { -- iRet = RS_RET_IO_ERROR; -- } -- fclose(sf); -- free(cursor); -- /* change the name of the file to the configured one */ -- if (iRet == RS_RET_OK && rename(tmp_sf, cs.stateFile) == -1) { -- char errStr[256]; -- rs_strerror_r(errno, errStr, sizeof(errStr)); -- iRet = RS_RET_IO_ERROR; -- errmsg.LogError(0, iRet, "rename() failed: " -- "'%s', new path: '%s'\n", errStr, cs.stateFile); -- } -+ if (!last_cursor) -+ ABORT_FINALIZE(RS_RET_OK); - -- } else { -- char errStr[256]; -- rs_strerror_r(errno, errStr, sizeof(errStr)); -- errmsg.LogError(0, RS_RET_FOPEN_FAILURE, "fopen() failed: " -- "'%s', path: '%s'\n", errStr, tmp_sf); -- iRet = RS_RET_FOPEN_FAILURE; -- } -- } else { -- char errStr[256]; -- rs_strerror_r(-(ret), errStr, sizeof(errStr)); -- errmsg.LogError(0, RS_RET_ERR, "sd_journal_get_cursor() failed: '%s'\n", errStr); -- iRet = RS_RET_ERR; -- } -+ /* we create a temporary name by adding a ".tmp" -+ * suffix to the end of our state file's name -+ */ -+ snprintf(tmp_sf, sizeof(tmp_sf), "%s.tmp", cs.stateFile); -+ -+ sf = fopen(tmp_sf, "wb"); -+ if (!sf) { -+ errmsg.LogError(errno, RS_RET_FOPEN_FAILURE, "imjournal: fopen() failed for path: '%s'", tmp_sf); -+ ABORT_FINALIZE(RS_RET_FOPEN_FAILURE); -+ } -+ -+ r = fprintf(sf, "%s", last_cursor); -+ if (r < 0) { -+ errmsg.LogError(errno, RS_RET_FOPEN_FAILURE, "imjournal: failed to save cursor to: '%s'", tmp_sf); -+ ABORT_FINALIZE(RS_RET_IO_ERROR); -+ } -+ -+ r = fclose(sf); -+ if (r < 0) { -+ errmsg.LogError(errno, iRet, "imjournal: fclose() failed for path: '%s'", tmp_sf); -+ ABORT_FINALIZE(RS_RET_IO_ERROR); -+ } -+ -+ r = rename(tmp_sf, cs.stateFile); -+ if (r < 0) { -+ errmsg.LogError(errno, iRet, "imjournal: rename() failed for new path: '%s'", cs.stateFile); -+ ABORT_FINALIZE(RS_RET_IO_ERROR); -+ } -+ -+finalize_it: - RETiRet; - } - -@@ -473,64 +473,29 @@ - * except for the special handling of EINTR. - */ - --#define POLL_TIMEOUT 1000 /* timeout for poll is 1s */ -+#define POLL_TIMEOUT 900000 /* timeout for poll is 900ms */ - - static rsRetVal - pollJournal(void) - { - DEFiRet; -- struct pollfd pollfd; -- int pr = 0; -- int jr = 0; -- -- pollfd.fd = sd_journal_get_fd(j); -- pollfd.events = sd_journal_get_events(j); -- pr = poll(&pollfd, 1, POLL_TIMEOUT); -- if (pr == -1) { -- if (errno == EINTR) { -- /* EINTR is also received during termination -- * so return now to check the term state. -- */ -- ABORT_FINALIZE(RS_RET_OK); -- } else { -- char errStr[256]; -- -- rs_strerror_r(errno, errStr, sizeof(errStr)); -- errmsg.LogError(0, RS_RET_ERR, -- "poll() failed: '%s'", errStr); -- ABORT_FINALIZE(RS_RET_ERR); -- } -- } -+ int r; - -+ for (;;) { -+ r = sd_journal_wait(j, POLL_TIMEOUT); -+ break; -+ } - -- jr = sd_journal_process(j); -- -- if (pr == 1 && jr == SD_JOURNAL_INVALIDATE) { -- /* do not persist stateFile sd_journal_get_cursor will fail! */ -- char* tmp = cs.stateFile; -- cs.stateFile = NULL; -+ if (r == SD_JOURNAL_INVALIDATE) { - closeJournal(&j); -- cs.stateFile = tmp; - - iRet = openJournal(&j); -- if (iRet != RS_RET_OK) { -- char errStr[256]; -- rs_strerror_r(errno, errStr, sizeof(errStr)); -- errmsg.LogError(0, RS_RET_IO_ERROR, -- "sd_journal_open() failed: '%s'", errStr); -+ if (iRet != RS_RET_OK) - ABORT_FINALIZE(RS_RET_ERR); -- } - -- if(cs.stateFile != NULL){ -+ if (cs.stateFile) - iRet = loadJournalState(); -- } -- LogMsg(0, RS_RET_OK, LOG_NOTICE, "imjournal: journal reloaded..."); -- } else if (jr < 0) { -- char errStr[256]; -- rs_strerror_r(errno, errStr, sizeof(errStr)); -- errmsg.LogError(0, RS_RET_ERR, -- "sd_journal_process() failed: '%s'", errStr); -- ABORT_FINALIZE(RS_RET_ERR); -+ errmsg.LogMsg(0, RS_RET_OK, LOG_NOTICE, "imjournal: journal reloaded..."); - } - - finalize_it: -@@ -631,8 +612,17 @@ loadJournalState(void) - RETiRet; - } - -+static void -+tryRecover(void) { -+ errmsg.LogMsg(0, RS_RET_OK, LOG_INFO, "imjournal: trying to recover from unexpected " -+ "journal error"); -+ closeJournal(&j); -+ srSleep(10, 0); // do not hammer machine with too-frequent retries -+ openJournal(&j); -+} -+ - BEGINrunInput -- int count = 0; -+ uint64_t count = 0; - CODESTARTrunInput - CHKiRet(ratelimitNew(&ratelimiter, "imjournal", NULL)); - dbgprintf("imjournal: ratelimiting burst %d, interval %d\n", cs.ratelimitBurst, -@@ -665,26 +655,38 @@ CODESTARTrunInput - - r = sd_journal_next(j); - if (r < 0) { -- char errStr[256]; -- -- rs_strerror_r(errno, errStr, sizeof(errStr)); -- errmsg.LogError(0, RS_RET_ERR, -- "sd_journal_next() failed: '%s'", errStr); -- ABORT_FINALIZE(RS_RET_ERR); -+ tryRecover(); -+ continue; - } - - if (r == 0) { - /* No new messages, wait for activity. */ -- CHKiRet(pollJournal()); -+ if (pollJournal() != RS_RET_OK) { -+ tryRecover(); -+ } - continue; - } - -- CHKiRet(readjournal()); -+ if (readjournal() != RS_RET_OK) { -+ tryRecover(); -+ continue; -+ } -+ -+ count++; -+ -+ if ((count % J_PROCESS_PERIOD) == 0) { -+ /* Give the journal a periodic chance to detect rotated journal files to be cleaned up. */ -+ r = sd_journal_process(j); -+ if (r < 0) { -+ errmsg.LogError(-r, RS_RET_ERR, "imjournal: sd_journal_process() failed"); -+ tryRecover(); -+ continue; -+ } -+ } -+ - if (cs.stateFile) { /* can't persist without a state file */ - /* TODO: This could use some finer metric. */ -- count++; -- if (count == cs.iPersistStateInterval) { -- count = 0; -+ if ((count % cs.iPersistStateInterval) == 0) { - persistJournalState(); - } - } diff --git a/SOURCES/rsyslog-8.24.0-rhbz1559408-async-writer.patch b/SOURCES/rsyslog-8.24.0-rhbz1559408-async-writer.patch new file mode 100644 index 0000000..53563de --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1559408-async-writer.patch @@ -0,0 +1,14 @@ +diff -up ./runtime/stream.c.fix ./runtime/stream.c +--- ./runtime/stream.c.fix 2018-06-25 17:39:39.223082288 +0200 ++++ ./runtime/stream.c 2018-06-25 17:40:26.549846798 +0200 +@@ -1427,10 +1427,8 @@ asyncWriterThread(void *pPtr) + } + if(bTimedOut && pThis->iBufPtr > 0) { + /* if we timed out, we need to flush pending data */ +- d_pthread_mutex_unlock(&pThis->mut); + strmFlushInternal(pThis, 1); + bTimedOut = 0; +- d_pthread_mutex_lock(&pThis->mut); + continue; + } + bTimedOut = 0; diff --git a/SOURCES/rsyslog-8.24.0-rhbz1565214-omelasticsearch-replace-cJSON-with-libfastjson.patch b/SOURCES/rsyslog-8.24.0-rhbz1565214-omelasticsearch-replace-cJSON-with-libfastjson.patch new file mode 100644 index 0000000..37c7a95 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1565214-omelasticsearch-replace-cJSON-with-libfastjson.patch @@ -0,0 +1,1105 @@ +From 6267b5a57c432a3be68f362c571beb062d47b3a7 Mon Sep 17 00:00:00 2001 +From: PascalWithopf +Date: Tue, 23 May 2017 15:32:34 +0200 +Subject: [PATCH 10/11] omelasticsearch: replace cJSON with libfastjson + +(cherry picked from commit 7982f50675471220c5ba035371a8f7537a50442b) +(cherry picked from commit 0b09c29db0cec5a215a95d03cfc37a27e486811c) +--- + plugins/omelasticsearch/Makefile.am | 3 +- + plugins/omelasticsearch/cJSON/cjson.c | 525 ------------------------------ + plugins/omelasticsearch/cJSON/cjson.h | 130 -------- + plugins/omelasticsearch/omelasticsearch.c | 171 +++++----- + 12 files changed, 84 insertions(+), 1323 deletions(-) + delete mode 100644 plugins/omelasticsearch/cJSON/cjson.c + delete mode 100644 plugins/omelasticsearch/cJSON/cjson.h + +diff --git a/plugins/omelasticsearch/Makefile.am b/plugins/omelasticsearch/Makefile.am +index ba85a896d..2fadb74dc 100644 +--- a/plugins/omelasticsearch/Makefile.am ++++ b/plugins/omelasticsearch/Makefile.am +@@ -1,7 +1,6 @@ + pkglib_LTLIBRARIES = omelasticsearch.la + +-# TODO: replace cJSON +-omelasticsearch_la_SOURCES = omelasticsearch.c cJSON/cjson.c cJSON/cjson.h ++omelasticsearch_la_SOURCES = omelasticsearch.c + omelasticsearch_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) + omelasticsearch_la_LDFLAGS = -module -avoid-version + omelasticsearch_la_LIBADD = $(CURL_LIBS) $(LIBM) +diff --git a/plugins/omelasticsearch/cJSON/cjson.c b/plugins/omelasticsearch/cJSON/cjson.c +deleted file mode 100644 +index 6f7d43a23..000000000 +--- a/plugins/omelasticsearch/cJSON/cjson.c ++++ /dev/null +@@ -1,525 +0,0 @@ +-/* +- Copyright (c) 2009 Dave Gamble +- +- Permission is hereby granted, free of charge, to any person obtaining a copy +- of this software and associated documentation files (the "Software"), to deal +- in the Software without restriction, including without limitation the rights +- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +- copies of the Software, and to permit persons to whom the Software is +- furnished to do so, subject to the following conditions: +- +- The above copyright notice and this permission notice shall be included in +- all copies or substantial portions of the Software. +- +- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +- THE SOFTWARE. +-*/ +- +-/* this code has several warnings, but we ignore them because +- * this seems to work and we do not want to engage in that code body. If +- * we really run into troubles, it is better to change to libfastjson, which +- * we should do in the medium to long term anyhow... +- */ +-#pragma GCC diagnostic ignored "-Wmissing-prototypes" +-#pragma GCC diagnostic ignored "-Wredundant-decls" +-#pragma GCC diagnostic ignored "-Wstrict-prototypes" +-#pragma GCC diagnostic ignored "-Wswitch-default" +-#pragma GCC diagnostic ignored "-Wold-style-definition" +- +-/* cJSON */ +-/* JSON parser in C. */ +- +-#include +-#include +-#include +-#include +-#include +-#include +-#include +-#include "cjson.h" +- +-static const char *ep; +- +-const char *cJSON_GetErrorPtr() {return ep;} +- +-static int cJSON_strcasecmp(const char *s1,const char *s2) +-{ +- if (!s1) return (s1==s2)?0:1;if (!s2) return 1; +- for(; tolower(*s1) == tolower(*s2); ++s1, ++s2) if(*s1 == 0) return 0; +- return tolower(*(const unsigned char *)s1) - tolower(*(const unsigned char *)s2); +-} +- +-static void *(*cJSON_malloc)(size_t sz) = malloc; +-static void (*cJSON_free)(void *ptr) = free; +- +-static char* cJSON_strdup(const char* str) +-{ +- size_t len; +- char* copy; +- +- len = strlen(str) + 1; +- if (!(copy = (char*)cJSON_malloc(len))) return 0; +- memcpy(copy,str,len); +- return copy; +-} +- +-void cJSON_InitHooks(cJSON_Hooks* hooks) +-{ +- if (!hooks) { /* Reset hooks */ +- cJSON_malloc = malloc; +- cJSON_free = free; +- return; +- } +- +- cJSON_malloc = (hooks->malloc_fn)?hooks->malloc_fn:malloc; +- cJSON_free = (hooks->free_fn)?hooks->free_fn:free; +-} +- +-/* Internal constructor. */ +-static cJSON *cJSON_New_Item() +-{ +- cJSON* node = (cJSON*)cJSON_malloc(sizeof(cJSON)); +- if (node) memset(node,0,sizeof(cJSON)); +- return node; +-} +- +-/* Delete a cJSON structure. */ +-void cJSON_Delete(cJSON *c) +-{ +- cJSON *next; +- while (c) +- { +- next=c->next; +- if (!(c->type&cJSON_IsReference) && c->child) cJSON_Delete(c->child); +- if (!(c->type&cJSON_IsReference) && c->valuestring) cJSON_free(c->valuestring); +- if (c->string) cJSON_free(c->string); +- cJSON_free(c); +- c=next; +- } +-} +- +-/* Parse the input text to generate a number, and populate the result into item. */ +-static const char *parse_number(cJSON *item,const char *num) +-{ +- double n=0,sign=1,scale=0;int subscale=0,signsubscale=1; +- +- /* Could use sscanf for this? */ +- if (*num=='-') sign=-1,num++; /* Has sign? */ +- if (*num=='0') num++; /* is zero */ +- if (*num>='1' && *num<='9') do n=(n*10.0)+(*num++ -'0'); while (*num>='0' && *num<='9'); /* Number? */ +- if (*num=='.' && num[1]>='0' && num[1]<='9') {num++; do n=(n*10.0)+(*num++ -'0'),scale--; while (*num>='0' && *num<='9');} /* Fractional part? */ +- if (*num=='e' || *num=='E') /* Exponent? */ +- { num++;if (*num=='+') num++; else if (*num=='-') signsubscale=-1,num++; /* With sign? */ +- while (*num>='0' && *num<='9') subscale=(subscale*10)+(*num++ - '0'); /* Number? */ +- } +- +- n=sign*n*pow(10.0,(scale+subscale*signsubscale)); /* number = +/- number.fraction * 10^+/- exponent */ +- +- item->valuedouble=n; +- item->valueint=(int)n; +- item->type=cJSON_Number; +- return num; +-} +- +-/* Render the number nicely from the given item into a string. */ +-char *cJSON_print_number(cJSON *item) +-{ +- char *str; +- double d=item->valuedouble; +- if (fabs(((double)item->valueint)-d)<=DBL_EPSILON && d<=INT_MAX && d>=INT_MIN) +- { +- str=(char*)cJSON_malloc(21); /* 2^64+1 can be represented in 21 chars. */ +- if (str) sprintf(str,"%d",item->valueint); +- } +- else +- { +- str=(char*)cJSON_malloc(64); /* This is a nice tradeoff. */ +- if (str) +- { +- if (fabs(floor(d)-d)<=DBL_EPSILON) sprintf(str,"%.0f",d); +- else if (fabs(d)<1.0e-6 || fabs(d)>1.0e9) sprintf(str,"%e",d); +- else sprintf(str,"%f",d); +- } +- } +- return str; +-} +- +-/* Parse the input text into an unescaped cstring, and populate item. */ +-static const unsigned char firstByteMark[7] = { 0x00, 0x00, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC }; +-static const char *parse_string(cJSON *item,const char *str) +-{ +- const char *ptr=str+1;char *ptr2;char *out;int len=0;unsigned uc,uc2; +- if (*str!='\"') {ep=str;return 0;} /* not a string! */ +- +- while (*ptr!='\"' && *ptr && ++len) if (*ptr++ == '\\') ptr++; /* Skip escaped quotes. */ +- +- out=(char*)cJSON_malloc(len+1); /* This is how long we need for the string, roughly. */ +- if (!out) return 0; +- +- ptr=str+1;ptr2=out; +- while (*ptr!='\"' && *ptr) +- { +- if (*ptr!='\\') *ptr2++=*ptr++; +- else +- { +- ptr++; +- switch (*ptr) +- { +- case 'b': *ptr2++='\b'; break; +- case 'f': *ptr2++='\f'; break; +- case 'n': *ptr2++='\n'; break; +- case 'r': *ptr2++='\r'; break; +- case 't': *ptr2++='\t'; break; +- case 'u': /* transcode utf16 to utf8. */ +- sscanf(ptr+1,"%4x",&uc);ptr+=4; /* get the unicode char. */ +- +- if ((uc>=0xDC00 && uc<=0xDFFF) || uc==0) break; // check for invalid. +- +- if (uc>=0xD800 && uc<=0xDBFF) // UTF16 surrogate pairs. +- { +- if (ptr[1]!='\\' || ptr[2]!='u') break; // missing second-half of surrogate. +- sscanf(ptr+3,"%4x",&uc2);ptr+=6; +- if (uc2<0xDC00 || uc2>0xDFFF) break; // invalid second-half of surrogate. +- uc=0x10000 | ((uc&0x3FF)<<10) | (uc2&0x3FF); +- } +- +- len=4;if (uc<0x80) len=1;else if (uc<0x800) len=2;else if (uc<0x10000) len=3; ptr2+=len; +- +- switch (len) { +- case 4: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; +- case 3: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; +- case 2: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; +- case 1: *--ptr2 =(uc | firstByteMark[len]); +- } +- ptr2+=len; +- break; +- default: *ptr2++=*ptr; break; +- } +- ptr++; +- } +- } +- *ptr2=0; +- if (*ptr=='\"') ptr++; +- item->valuestring=out; +- item->type=cJSON_String; +- return ptr; +-} +- +-/* Render the cstring provided to an escaped version that can be printed. */ +-static char *print_string_ptr(const char *str) +-{ +- const char *ptr;char *ptr2,*out;int len=0;unsigned char token; +- +- if (!str) return cJSON_strdup(""); +- ptr=str;while ((token=*ptr) && ++len) {if (strchr("\"\\\b\f\n\r\t",token)) len++; else if (token<32) len+=5;ptr++;} +- +- out=(char*)cJSON_malloc(len+3); +- if (!out) return 0; +- +- ptr2=out;ptr=str; +- *ptr2++='\"'; +- while (*ptr) +- { +- if ((unsigned char)*ptr>31 && *ptr!='\"' && *ptr!='\\') *ptr2++=*ptr++; +- else +- { +- *ptr2++='\\'; +- switch (token=*ptr++) +- { +- case '\\': *ptr2++='\\'; break; +- case '\"': *ptr2++='\"'; break; +- case '\b': *ptr2++='b'; break; +- case '\f': *ptr2++='f'; break; +- case '\n': *ptr2++='n'; break; +- case '\r': *ptr2++='r'; break; +- case '\t': *ptr2++='t'; break; +- default: sprintf(ptr2,"u%04x",token);ptr2+=5; break; /* escape and print */ +- } +- } +- } +- *ptr2++='\"';*ptr2++=0; +- return out; +-} +-/* Invote print_string_ptr (which is useful) on an item. */ +-static char *print_string(cJSON *item) {return print_string_ptr(item->valuestring);} +- +-/* Predeclare these prototypes. */ +-static const char *parse_value(cJSON *item,const char *value); +-static char *print_value(cJSON *item,int depth,int fmt); +-static const char *parse_array(cJSON *item,const char *value); +-static char *print_array(cJSON *item,int depth,int fmt); +-static const char *parse_object(cJSON *item,const char *value); +-static char *print_object(cJSON *item,int depth,int fmt); +- +-/* Utility to jump whitespace and cr/lf */ +-static const char *skip(const char *in) {while (in && *in && (unsigned char)*in<=32) in++; return in;} +- +-/* Parse an object - create a new root, and populate. */ +-cJSON *cJSON_Parse(const char *value) +-{ +- cJSON *c=cJSON_New_Item(); +- ep=0; +- if (!c) return 0; /* memory fail */ +- +- if (!parse_value(c,skip(value))) {cJSON_Delete(c);return 0;} +- return c; +-} +- +-/* Render a cJSON item/entity/structure to text. */ +-char *cJSON_Print(cJSON *item) {return print_value(item,0,1);} +-char *cJSON_PrintUnformatted(cJSON *item) {return print_value(item,0,0);} +- +-/* Parser core - when encountering text, process appropriately. */ +-static const char *parse_value(cJSON *item,const char *value) +-{ +- if (!value) return 0; /* Fail on null. */ +- if (!strncmp(value,"null",4)) { item->type=cJSON_NULL; return value+4; } +- if (!strncmp(value,"false",5)) { item->type=cJSON_False; return value+5; } +- if (!strncmp(value,"true",4)) { item->type=cJSON_True; item->valueint=1; return value+4; } +- if (*value=='\"') { return parse_string(item,value); } +- if (*value=='-' || (*value>='0' && *value<='9')) { return parse_number(item,value); } +- if (*value=='[') { return parse_array(item,value); } +- if (*value=='{') { return parse_object(item,value); } +- +- ep=value;return 0; /* failure. */ +-} +- +-/* Render a value to text. */ +-static char *print_value(cJSON *item,int depth,int fmt) +-{ +- char *out=0; +- if (!item) return 0; +- switch ((item->type)&255) +- { +- case cJSON_NULL: out=cJSON_strdup("null"); break; +- case cJSON_False: out=cJSON_strdup("false");break; +- case cJSON_True: out=cJSON_strdup("true"); break; +- case cJSON_Number: out=cJSON_print_number(item);break; +- case cJSON_String: out=print_string(item);break; +- case cJSON_Array: out=print_array(item,depth,fmt);break; +- case cJSON_Object: out=print_object(item,depth,fmt);break; +- } +- return out; +-} +- +-/* Build an array from input text. */ +-static const char *parse_array(cJSON *item,const char *value) +-{ +- cJSON *child; +- if (*value!='[') {ep=value;return 0;} /* not an array! */ +- +- item->type=cJSON_Array; +- value=skip(value+1); +- if (*value==']') return value+1; /* empty array. */ +- +- item->child=child=cJSON_New_Item(); +- if (!item->child) return 0; /* memory fail */ +- value=skip(parse_value(child,skip(value))); /* skip any spacing, get the value. */ +- if (!value) return 0; +- +- while (*value==',') +- { +- cJSON *new_item; +- if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ +- child->next=new_item;new_item->prev=child;child=new_item; +- value=skip(parse_value(child,skip(value+1))); +- if (!value) return 0; /* memory fail */ +- } +- +- if (*value==']') return value+1; /* end of array */ +- ep=value;return 0; /* malformed. */ +-} +- +-/* Render an array to text */ +-static char *print_array(cJSON *item,int depth,int fmt) +-{ +- char **entries; +- char *out=0,*ptr,*ret;int len=5; +- cJSON *child=item->child; +- int numentries=0,i=0,fail=0; +- +- /* How many entries in the array? */ +- while (child) numentries++,child=child->next; +- /* Allocate an array to hold the values for each */ +- entries=(char**)cJSON_malloc(numentries*sizeof(char*)); +- if (!entries) return 0; +- memset(entries,0,numentries*sizeof(char*)); +- /* Retrieve all the results: */ +- child=item->child; +- while (child && !fail) +- { +- ret=print_value(child,depth+1,fmt); +- entries[i++]=ret; +- if (ret) len+=strlen(ret)+2+(fmt?1:0); else fail=1; +- child=child->next; +- } +- +- /* If we didn't fail, try to malloc the output string */ +- if (!fail) out=(char*)cJSON_malloc(len); +- /* If that fails, we fail. */ +- if (!out) fail=1; +- +- /* Handle failure. */ +- if (fail) +- { +- for (i=0;itype=cJSON_Object; +- value=skip(value+1); +- if (*value=='}') return value+1; /* empty array. */ +- +- item->child=child=cJSON_New_Item(); +- if (!item->child) return 0; +- value=skip(parse_string(child,skip(value))); +- if (!value) return 0; +- child->string=child->valuestring;child->valuestring=0; +- if (*value!=':') {ep=value;return 0;} /* fail! */ +- value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ +- if (!value) return 0; +- +- while (*value==',') +- { +- cJSON *new_item; +- if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ +- child->next=new_item;new_item->prev=child;child=new_item; +- value=skip(parse_string(child,skip(value+1))); +- if (!value) return 0; +- child->string=child->valuestring;child->valuestring=0; +- if (*value!=':') {ep=value;return 0;} /* fail! */ +- value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ +- if (!value) return 0; +- } +- +- if (*value=='}') return value+1; /* end of array */ +- ep=value;return 0; /* malformed. */ +-} +- +-/* Render an object to text. */ +-static char *print_object(cJSON *item,int depth,int fmt) +-{ +- char **entries=0,**names=0; +- char *out=0,*ptr,*ret,*str;int len=7,i=0,j; +- cJSON *child=item->child; +- int numentries=0,fail=0; +- /* Count the number of entries. */ +- while (child) numentries++,child=child->next; +- /* Allocate space for the names and the objects */ +- entries=(char**)cJSON_malloc(numentries*sizeof(char*)); +- if (!entries) return 0; +- names=(char**)cJSON_malloc(numentries*sizeof(char*)); +- if (!names) {cJSON_free(entries);return 0;} +- memset(entries,0,sizeof(char*)*numentries); +- memset(names,0,sizeof(char*)*numentries); +- +- /* Collect all the results into our arrays: */ +- child=item->child;depth++;if (fmt) len+=depth; +- while (child) +- { +- names[i]=str=print_string_ptr(child->string); +- entries[i++]=ret=print_value(child,depth,fmt); +- if (str && ret) len+=strlen(ret)+strlen(str)+2+(fmt?2+depth:0); else fail=1; +- child=child->next; +- } +- +- /* Try to allocate the output string */ +- if (!fail) out=(char*)cJSON_malloc(len); +- if (!out) fail=1; +- +- /* Handle failure */ +- if (fail) +- { +- for (i=0;ichild;int i=0;while(c)i++,c=c->next;return i;} +-cJSON *cJSON_GetArrayItem(cJSON *array,int item) {cJSON *c=array->child; while (c && item>0) item--,c=c->next; return c;} +-cJSON *cJSON_GetObjectItem(cJSON *object,const char *string) {cJSON *c=object->child; while (c && cJSON_strcasecmp(c->string,string)) c=c->next; return c;} +- +-/* Utility for array list handling. */ +-static void suffix_object(cJSON *prev,cJSON *item) {prev->next=item;item->prev=prev;} +-/* Utility for handling references. */ +-static cJSON *create_reference(cJSON *item) {cJSON *ref=cJSON_New_Item();if (!ref) return 0;memcpy(ref,item,sizeof(cJSON));ref->string=0;ref->type|=cJSON_IsReference;ref->next=ref->prev=0;return ref;} +- +-/* Add item to array/object. */ +-void cJSON_AddItemToArray(cJSON *array, cJSON *item) {cJSON *c=array->child;if (!item) return; if (!c) {array->child=item;} else {while (c && c->next) c=c->next; suffix_object(c,item);}} +-void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item) {if (!item) return; if (item->string) cJSON_free(item->string);item->string=cJSON_strdup(string);cJSON_AddItemToArray(object,item);} +-void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item) {cJSON_AddItemToArray(array,create_reference(item));} +-void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item) {cJSON_AddItemToObject(object,string,create_reference(item));} +- +-cJSON *cJSON_DetachItemFromArray(cJSON *array,int which) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return 0; +- if (c->prev) c->prev->next=c->next;if (c->next) c->next->prev=c->prev;if (c==array->child) array->child=c->next;c->prev=c->next=0;return c;} +-void cJSON_DeleteItemFromArray(cJSON *array,int which) {cJSON_Delete(cJSON_DetachItemFromArray(array,which));} +-cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string) {int i=0;cJSON *c=object->child;while (c && cJSON_strcasecmp(c->string,string)) i++,c=c->next;if (c) return cJSON_DetachItemFromArray(object,i);return 0;} +-void cJSON_DeleteItemFromObject(cJSON *object,const char *string) {cJSON_Delete(cJSON_DetachItemFromObject(object,string));} +- +-/* Replace array/object items with new ones. */ +-void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return; +- newitem->next=c->next;newitem->prev=c->prev;if (newitem->next) newitem->next->prev=newitem; +- if (c==array->child) array->child=newitem; else newitem->prev->next=newitem;c->next=c->prev=0;cJSON_Delete(c);} +-void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem){int i=0;cJSON *c=object->child;while(c && cJSON_strcasecmp(c->string,string))i++,c=c->next;if(c){newitem->string=cJSON_strdup(string);cJSON_ReplaceItemInArray(object,i,newitem);}} +- +-/* Create basic types: */ +-cJSON *cJSON_CreateNull() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_NULL;return item;} +-cJSON *cJSON_CreateTrue() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_True;return item;} +-cJSON *cJSON_CreateFalse() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_False;return item;} +-cJSON *cJSON_CreateBool(int b) {cJSON *item=cJSON_New_Item();if(item)item->type=b?cJSON_True:cJSON_False;return item;} +-cJSON *cJSON_CreateNumber(double num) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_Number;item->valuedouble=num;item->valueint=(int)num;}return item;} +-cJSON *cJSON_CreateString(const char *string) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_String;item->valuestring=cJSON_strdup(string);}return item;} +-cJSON *cJSON_CreateArray() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Array;return item;} +-cJSON *cJSON_CreateObject() {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Object;return item;} +- +-/* Create Arrays: */ +-cJSON *cJSON_CreateIntArray(int *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} +-cJSON *cJSON_CreateFloatArray(float *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} +-cJSON *cJSON_CreateDoubleArray(double *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} +-cJSON *cJSON_CreateStringArray(const char **strings,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} +diff --git a/plugins/omelasticsearch/cJSON/cjson.h b/plugins/omelasticsearch/cJSON/cjson.h +deleted file mode 100644 +index a621720ce..000000000 +--- a/plugins/omelasticsearch/cJSON/cjson.h ++++ /dev/null +@@ -1,130 +0,0 @@ +-/* +- Copyright (c) 2009 Dave Gamble +- +- Permission is hereby granted, free of charge, to any person obtaining a copy +- of this software and associated documentation files (the "Software"), to deal +- in the Software without restriction, including without limitation the rights +- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +- copies of the Software, and to permit persons to whom the Software is +- furnished to do so, subject to the following conditions: +- +- The above copyright notice and this permission notice shall be included in +- all copies or substantial portions of the Software. +- +- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +- THE SOFTWARE. +-*/ +- +-#ifndef cJSON__h +-#define cJSON__h +- +-#ifdef __cplusplus +-extern "C" +-{ +-#endif +- +-/* cJSON Types: */ +-#define cJSON_False 0 +-#define cJSON_True 1 +-#define cJSON_NULL 2 +-#define cJSON_Number 3 +-#define cJSON_String 4 +-#define cJSON_Array 5 +-#define cJSON_Object 6 +- +-#define cJSON_IsReference 256 +- +-/* The cJSON structure: */ +-typedef struct cJSON { +- struct cJSON *next,*prev; /* next/prev allow you to walk array/object chains. Alternatively, use GetArraySize/GetArrayItem/GetObjectItem */ +- struct cJSON *child; /* An array or object item will have a child pointer pointing to a chain of the items in the array/object. */ +- +- int type; /* The type of the item, as above. */ +- +- char *valuestring; /* The item's string, if type==cJSON_String */ +- int valueint; /* The item's number, if type==cJSON_Number */ +- double valuedouble; /* The item's number, if type==cJSON_Number */ +- +- char *string; /* The item's name string, if this item is the child of, or is in the list of subitems of an object. */ +-} cJSON; +- +-typedef struct cJSON_Hooks { +- void *(*malloc_fn)(size_t sz); +- void (*free_fn)(void *ptr); +-} cJSON_Hooks; +- +-/* Supply malloc, realloc and free functions to cJSON */ +-extern void cJSON_InitHooks(cJSON_Hooks* hooks); +- +- +-/* Supply a block of JSON, and this returns a cJSON object you can interrogate. Call cJSON_Delete when finished. */ +-extern cJSON *cJSON_Parse(const char *value); +-/* Render a cJSON entity to text for transfer/storage. Free the char* when finished. */ +-extern char *cJSON_Print(cJSON *item); +-/* Render a cJSON entity to text for transfer/storage without any formatting. Free the char* when finished. */ +-extern char *cJSON_PrintUnformatted(cJSON *item); +-/* Delete a cJSON entity and all subentities. */ +-extern void cJSON_Delete(cJSON *c); +- +-/* Returns the number of items in an array (or object). */ +-extern int cJSON_GetArraySize(cJSON *array); +-/* Retrieve item number "item" from array "array". Returns NULL if unsuccessful. */ +-extern cJSON *cJSON_GetArrayItem(cJSON *array,int item); +-/* Get item "string" from object. Case insensitive. */ +-extern cJSON *cJSON_GetObjectItem(cJSON *object,const char *string); +- +-/* For analysing failed parses. This returns a pointer to the parse error. You'll probably need to look a few chars back to make sense of it. Defined when cJSON_Parse() returns 0. 0 when cJSON_Parse() succeeds. */ +-extern const char *cJSON_GetErrorPtr(); +- +-/* These calls create a cJSON item of the appropriate type. */ +-extern cJSON *cJSON_CreateNull(); +-extern cJSON *cJSON_CreateTrue(); +-extern cJSON *cJSON_CreateFalse(); +-extern cJSON *cJSON_CreateBool(int b); +-extern cJSON *cJSON_CreateNumber(double num); +-extern cJSON *cJSON_CreateString(const char *string); +-extern cJSON *cJSON_CreateArray(); +-extern cJSON *cJSON_CreateObject(); +- +-/* These utilities create an Array of count items. */ +-extern cJSON *cJSON_CreateIntArray(int *numbers,int count); +-extern cJSON *cJSON_CreateFloatArray(float *numbers,int count); +-extern cJSON *cJSON_CreateDoubleArray(double *numbers,int count); +-extern cJSON *cJSON_CreateStringArray(const char **strings,int count); +- +-/* Append item to the specified array/object. */ +-extern void cJSON_AddItemToArray(cJSON *array, cJSON *item); +-extern void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item); +-/* Append reference to item to the specified array/object. Use this when you want to add an existing cJSON to a new cJSON, but don't want to corrupt your existing cJSON. */ +-extern void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item); +-extern void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item); +- +-/* Remove/Detatch items from Arrays/Objects. */ +-extern cJSON *cJSON_DetachItemFromArray(cJSON *array,int which); +-extern void cJSON_DeleteItemFromArray(cJSON *array,int which); +-extern cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string); +-extern void cJSON_DeleteItemFromObject(cJSON *object,const char *string); +- +-/* Update array items. */ +-extern void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem); +-extern void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem); +- +-/* rger: added helpers */ +- +-char *cJSON_print_number(cJSON *item); +-#define cJSON_AddNullToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateNull()) +-#define cJSON_AddTrueToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateTrue()) +-#define cJSON_AddFalseToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateFalse()) +-#define cJSON_AddNumberToObject(object,name,n) cJSON_AddItemToObject(object, name, cJSON_CreateNumber(n)) +-#define cJSON_AddStringToObject(object,name,s) cJSON_AddItemToObject(object, name, cJSON_CreateString(s)) +- +-#ifdef __cplusplus +-} +-#endif +- +-#endif +diff --git a/plugins/omelasticsearch/omelasticsearch.c b/plugins/omelasticsearch/omelasticsearch.c +index 88bd5e16c..ed2b47535 100644 +--- a/plugins/omelasticsearch/omelasticsearch.c ++++ b/plugins/omelasticsearch/omelasticsearch.c +@@ -41,7 +41,7 @@ + #if defined(__FreeBSD__) + #include + #endif +-#include "cJSON/cjson.h" ++#include + #include "conf.h" + #include "syslogd-types.h" + #include "srUtils.h" +@@ -626,29 +626,29 @@ finalize_it: + * Dumps entire bulk request and response in error log + */ + static rsRetVal +-getDataErrorDefault(wrkrInstanceData_t *pWrkrData,cJSON **pReplyRoot,uchar *reqmsg,char **rendered) ++getDataErrorDefault(wrkrInstanceData_t *pWrkrData,fjson_object **pReplyRoot,uchar *reqmsg,char **rendered) + { + DEFiRet; +- cJSON *req=0; +- cJSON *errRoot=0; +- cJSON *replyRoot = *pReplyRoot; ++ fjson_object *req=NULL; ++ fjson_object *errRoot=NULL; ++ fjson_object *replyRoot = *pReplyRoot; + +- if((req=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); +- cJSON_AddItemToObject(req, "url", cJSON_CreateString((char*)pWrkrData->restURL)); +- cJSON_AddItemToObject(req, "postdata", cJSON_CreateString((char*)reqmsg)); ++ if((req=fjson_object_new_object()) == NULL) ABORT_FINALIZE(RS_RET_ERR); ++ fjson_object_object_add(req, "url", fjson_object_new_string((char*)pWrkrData->restURL)); ++ fjson_object_object_add(req, "postdata", fjson_object_new_string((char*)reqmsg)); + +- if((errRoot=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); +- cJSON_AddItemToObject(errRoot, "request", req); +- cJSON_AddItemToObject(errRoot, "reply", replyRoot); +- *rendered = cJSON_Print(errRoot); ++ if((errRoot=fjson_object_new_object()) == NULL) ABORT_FINALIZE(RS_RET_ERR); ++ fjson_object_object_add(errRoot, "request", req); ++ fjson_object_object_add(errRoot, "reply", replyRoot); ++ *rendered = strdup((char*)fjson_object_to_json_string(errRoot)); + +- req=0; +- cJSON_Delete(errRoot); ++ req=NULL; ++ fjson_object_put(errRoot); + + *pReplyRoot = NULL; /* tell caller not to delete once again! */ + + finalize_it: +- cJSON_Delete(req); ++ fjson_object_put(req); + RETiRet; + } + +@@ -703,8 +703,8 @@ finalize_it: + /* + * check the status of response from ES + */ +-static int checkReplyStatus(cJSON* ok) { +- return (ok == NULL || ok->type != cJSON_Number || ok->valueint < 0 || ok->valueint > 299); ++static int checkReplyStatus(fjson_object* ok) { ++ return (ok == NULL || !fjson_object_is_type(ok, fjson_type_int) || fjson_object_get_int(ok) < 0 || fjson_object_get_int(ok) > 299); + } + + /* +@@ -712,7 +712,7 @@ static int checkReplyStatus(cJSON* ok) { + */ + typedef struct exeContext{ + int statusCheckOnly; +- cJSON *errRoot; ++ fjson_object *errRoot; + rsRetVal (*prepareErrorFileContent)(struct exeContext *ctx,int itemStatus,char *request,char *response); + + +@@ -722,25 +722,24 @@ typedef struct exeContext{ + * get content to be written in error file using context passed + */ + static rsRetVal +-parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,cJSON **pReplyRoot,uchar *reqmsg,context *ctx) ++parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,fjson_object **pReplyRoot,uchar *reqmsg,context *ctx) + { + DEFiRet; +- cJSON *replyRoot = *pReplyRoot; ++ fjson_object *replyRoot = *pReplyRoot; + int i; + int numitems; +- cJSON *items=0; ++ fjson_object *items=NULL; + + + /*iterate over items*/ +- items = cJSON_GetObjectItem(replyRoot, "items"); +- if(items == NULL || items->type != cJSON_Array) { ++ if(!fjson_object_object_get_ex(replyRoot, "items", &items)) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "bulkmode insert does not return array, reply is: %s\n", + pWrkrData->reply); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + +- numitems = cJSON_GetArraySize(items); ++ numitems = fjson_object_array_length(items); + + DBGPRINTF("omelasticsearch: Entire request %s\n",reqmsg); + const char *lastReqRead= (char*)reqmsg; +@@ -748,32 +747,32 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,cJSON **pReplyRo + DBGPRINTF("omelasticsearch: %d items in reply\n", numitems); + for(i = 0 ; i < numitems ; ++i) { + +- cJSON *item=0; +- cJSON *result=0; +- cJSON *ok=0; ++ fjson_object *item=NULL; ++ fjson_object *result=NULL; ++ fjson_object *ok=NULL; + int itemStatus=0; +- item = cJSON_GetArrayItem(items, i); ++ item = fjson_object_array_get_idx(items, i); + if(item == NULL) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "cannot obtain reply array item %d\n", i); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } +- result = item->child; +- if(result == NULL || result->type != cJSON_Object) { ++ fjson_object_object_get_ex(item, "create", &result); ++ if(result == NULL || !fjson_object_is_type(result, fjson_type_object)) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: " + "cannot obtain 'result' item for #%d\n", i); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + +- ok = cJSON_GetObjectItem(result, "status"); ++ fjson_object_object_get_ex(result, "status", &ok); + itemStatus = checkReplyStatus(ok); +- ++ + char *request =0; + char *response =0; + if(ctx->statusCheckOnly) + { + if(itemStatus) { +- DBGPRINTF("omelasticsearch: error in elasticsearch reply: item %d, status is %d\n", i, ok->valueint); ++ DBGPRINTF("omelasticsearch: error in elasticsearch reply: item %d, status is %d\n", i, fjson_object_get_int(ok)); + DBGPRINTF("omelasticsearch: status check found error.\n"); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } +@@ -786,13 +785,12 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,cJSON **pReplyRo + DBGPRINTF("omelasticsearch: Couldn't get post request\n"); + ABORT_FINALIZE(RS_RET_ERR); + } +- +- response = cJSON_PrintUnformatted(result); ++ response = (char*)fjson_object_to_json_string_ext(result, FJSON_TO_STRING_PLAIN); + + if(response==NULL) + { + free(request);/*as its has been assigned.*/ +- DBGPRINTF("omelasticsearch: Error getting cJSON_PrintUnformatted. Cannot continue\n"); ++ DBGPRINTF("omelasticsearch: Error getting fjson_object_to_string_ext. Cannot continue\n"); + ABORT_FINALIZE(RS_RET_ERR); + } + +@@ -801,7 +799,6 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,cJSON **pReplyRo + + /*free memory in any case*/ + free(request); +- free(response); + + if(ret != RS_RET_OK) + { +@@ -826,23 +823,23 @@ getDataErrorOnly(context *ctx,int itemStatus,char *request,char *response) + DEFiRet; + if(itemStatus) + { +- cJSON *onlyErrorResponses =0; +- cJSON *onlyErrorRequests=0; ++ fjson_object *onlyErrorResponses =NULL; ++ fjson_object *onlyErrorRequests=NULL; + +- if((onlyErrorResponses=cJSON_GetObjectItem(ctx->errRoot, "reply")) == NULL) ++ if(!fjson_object_object_get_ex(ctx->errRoot, "reply", &onlyErrorResponses)) + { + DBGPRINTF("omelasticsearch: Failed to get reply json array. Invalid context. Cannot continue\n"); + ABORT_FINALIZE(RS_RET_ERR); + } +- cJSON_AddItemToArray(onlyErrorResponses, cJSON_CreateString(response)); ++ fjson_object_array_add(onlyErrorResponses, fjson_object_new_string(response)); + +- if((onlyErrorRequests=cJSON_GetObjectItem(ctx->errRoot, "request")) == NULL) ++ if(!fjson_object_object_get_ex(ctx->errRoot, "request", &onlyErrorRequests)) + { + DBGPRINTF("omelasticsearch: Failed to get request json array. Invalid context. Cannot continue\n"); + ABORT_FINALIZE(RS_RET_ERR); + } + +- cJSON_AddItemToArray(onlyErrorRequests, cJSON_CreateString(request)); ++ fjson_object_array_add(onlyErrorRequests, fjson_object_new_string(request)); + + } + +@@ -861,24 +858,24 @@ getDataInterleaved(context *ctx, + char *response) + { + DEFiRet; +- cJSON *interleaved =0; +- if((interleaved=cJSON_GetObjectItem(ctx->errRoot, "response")) == NULL) ++ fjson_object *interleaved =NULL; ++ if(!fjson_object_object_get_ex(ctx->errRoot, "response", &interleaved)) + { + DBGPRINTF("omelasticsearch: Failed to get response json array. Invalid context. Cannot continue\n"); + ABORT_FINALIZE(RS_RET_ERR); + } + +- cJSON *interleavedNode=0; ++ fjson_object *interleavedNode=NULL; + /*create interleaved node that has req and response json data*/ +- if((interleavedNode=cJSON_CreateObject()) == NULL) ++ if((interleavedNode=fjson_object_new_object()) == NULL) + { + DBGPRINTF("omelasticsearch: Failed to create interleaved node. Cann't continue\n"); + ABORT_FINALIZE(RS_RET_ERR); + } +- cJSON_AddItemToObject(interleavedNode,"request", cJSON_CreateString(request)); +- cJSON_AddItemToObject(interleavedNode,"reply", cJSON_CreateString(response)); ++ fjson_object_object_add(interleavedNode,"request", fjson_object_new_string(request)); ++ fjson_object_object_add(interleavedNode,"reply", fjson_object_new_string(response)); + +- cJSON_AddItemToArray(interleaved, interleavedNode); ++ fjson_object_array_add(interleaved, interleavedNode); + + + +@@ -912,24 +909,24 @@ static rsRetVal + initializeErrorOnlyConext(wrkrInstanceData_t *pWrkrData,context *ctx){ + DEFiRet; + ctx->statusCheckOnly=0; +- cJSON *errRoot=0; +- cJSON *onlyErrorResponses =0; +- cJSON *onlyErrorRequests=0; +- if((errRoot=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); ++ fjson_object *errRoot=NULL; ++ fjson_object *onlyErrorResponses =NULL; ++ fjson_object *onlyErrorRequests=NULL; ++ if((errRoot=fjson_object_new_object()) == NULL) ABORT_FINALIZE(RS_RET_ERR); + +- if((onlyErrorResponses=cJSON_CreateArray()) == NULL) { +- cJSON_Delete(errRoot); ++ if((onlyErrorResponses=fjson_object_new_array()) == NULL) { ++ fjson_object_put(errRoot); + ABORT_FINALIZE(RS_RET_ERR); + } +- if((onlyErrorRequests=cJSON_CreateArray()) == NULL) { +- cJSON_Delete(errRoot); +- cJSON_Delete(onlyErrorResponses); ++ if((onlyErrorRequests=fjson_object_new_array()) == NULL) { ++ fjson_object_put(errRoot); ++ fjson_object_put(onlyErrorResponses); + ABORT_FINALIZE(RS_RET_ERR); + } + +- cJSON_AddItemToObject(errRoot, "url", cJSON_CreateString((char*)pWrkrData->restURL)); +- cJSON_AddItemToObject(errRoot,"request",onlyErrorRequests); +- cJSON_AddItemToObject(errRoot, "reply", onlyErrorResponses); ++ fjson_object_object_add(errRoot, "url", fjson_object_new_string((char*)pWrkrData->restURL)); ++ fjson_object_object_add(errRoot,"request",onlyErrorRequests); ++ fjson_object_object_add(errRoot, "reply", onlyErrorResponses); + ctx->errRoot = errRoot; + ctx->prepareErrorFileContent= &getDataErrorOnly; + finalize_it: +@@ -943,17 +940,17 @@ static rsRetVal + initializeInterleavedConext(wrkrInstanceData_t *pWrkrData,context *ctx){ + DEFiRet; + ctx->statusCheckOnly=0; +- cJSON *errRoot=0; +- cJSON *interleaved =0; +- if((errRoot=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); +- if((interleaved=cJSON_CreateArray()) == NULL) { +- cJSON_Delete(errRoot); ++ fjson_object *errRoot=NULL; ++ fjson_object *interleaved =NULL; ++ if((errRoot=fjson_object_new_object()) == NULL) ABORT_FINALIZE(RS_RET_ERR); ++ if((interleaved=fjson_object_new_array()) == NULL) { ++ fjson_object_put(errRoot); + ABORT_FINALIZE(RS_RET_ERR); + } + + +- cJSON_AddItemToObject(errRoot, "url", cJSON_CreateString((char*)pWrkrData->restURL)); +- cJSON_AddItemToObject(errRoot,"response",interleaved); ++ fjson_object_object_add(errRoot, "url", fjson_object_new_string((char*)pWrkrData->restURL)); ++ fjson_object_object_add(errRoot,"response",interleaved); + ctx->errRoot = errRoot; + ctx->prepareErrorFileContent= &getDataInterleaved; + finalize_it: +@@ -965,17 +962,17 @@ static rsRetVal + initializeErrorInterleavedConext(wrkrInstanceData_t *pWrkrData,context *ctx){ + DEFiRet; + ctx->statusCheckOnly=0; +- cJSON *errRoot=0; +- cJSON *interleaved =0; +- if((errRoot=cJSON_CreateObject()) == NULL) ABORT_FINALIZE(RS_RET_ERR); +- if((interleaved=cJSON_CreateArray()) == NULL) { +- cJSON_Delete(errRoot); ++ fjson_object *errRoot=NULL; ++ fjson_object *interleaved =NULL; ++ if((errRoot=fjson_object_new_object()) == NULL) ABORT_FINALIZE(RS_RET_ERR); ++ if((interleaved=fjson_object_new_array()) == NULL) { ++ fjson_object_put(errRoot); + ABORT_FINALIZE(RS_RET_ERR); + } + + +- cJSON_AddItemToObject(errRoot, "url", cJSON_CreateString((char*)pWrkrData->restURL)); +- cJSON_AddItemToObject(errRoot,"response",interleaved); ++ fjson_object_object_add(errRoot, "url", fjson_object_new_string((char*)pWrkrData->restURL)); ++ fjson_object_object_add(errRoot,"response",interleaved); + ctx->errRoot = errRoot; + ctx->prepareErrorFileContent= &getDataErrorOnlyInterleaved; + finalize_it: +@@ -988,7 +985,7 @@ initializeErrorInterleavedConext(wrkrInstanceData_t *pWrkrData,context *ctx){ + * needs to be closed, HUP must be sent. + */ + static rsRetVal +-writeDataError(wrkrInstanceData_t *pWrkrData, instanceData *pData, cJSON **pReplyRoot, uchar *reqmsg) ++writeDataError(wrkrInstanceData_t *pWrkrData, instanceData *pData, fjson_object **pReplyRoot, uchar *reqmsg) + { + char *rendered = NULL; + size_t toWrite; +@@ -1054,7 +1051,7 @@ writeDataError(wrkrInstanceData_t *pWrkrData, instanceData *pData, cJSON **pRepl + DBGPRINTF("omelasticsearch: error creating file content.\n"); + ABORT_FINALIZE(RS_RET_ERR); + } +- rendered = cJSON_Print(ctx.errRoot); ++ rendered = (char*)fjson_object_to_json_string(ctx.errRoot); + } + + +@@ -1084,14 +1081,13 @@ writeDataError(wrkrInstanceData_t *pWrkrData, instanceData *pData, cJSON **pRepl + finalize_it: + if(bMutLocked) + pthread_mutex_unlock(&pData->mutErrFile); +- cJSON_Delete(ctx.errRoot); +- free(rendered); ++ fjson_object_put(ctx.errRoot); + RETiRet; + } + + + static rsRetVal +-checkResultBulkmode(wrkrInstanceData_t *pWrkrData, cJSON *root) ++checkResultBulkmode(wrkrInstanceData_t *pWrkrData, fjson_object *root) + { + DEFiRet; + context ctx; +@@ -1111,11 +1107,11 @@ checkResultBulkmode(wrkrInstanceData_t *pWrkrData, cJSON *root) + static rsRetVal + checkResult(wrkrInstanceData_t *pWrkrData, uchar *reqmsg) + { +- cJSON *root; +- cJSON *status; ++ fjson_object *root; ++ fjson_object *status; + DEFiRet; + +- root = cJSON_Parse(pWrkrData->reply); ++ root = fjson_tokener_parse(pWrkrData->reply); + if(root == NULL) { + DBGPRINTF("omelasticsearch: could not parse JSON result \n"); + ABORT_FINALIZE(RS_RET_ERR); +@@ -1124,10 +1120,7 @@ checkResult(wrkrInstanceData_t *pWrkrData, uchar *reqmsg) + if(pWrkrData->pData->bulkmode) { + iRet = checkResultBulkmode(pWrkrData, root); + } else { +- status = cJSON_GetObjectItem(root, "status"); +- /* as far as we know, no "status" means all went well */ +- if(status != NULL && +- (status->type == cJSON_Number || status->valueint >= 0 || status->valueint <= 299)) { ++ if(fjson_object_object_get_ex(root, "status", &status)) { + iRet = RS_RET_DATAFAIL; + } + } +@@ -1143,7 +1136,7 @@ checkResult(wrkrInstanceData_t *pWrkrData, uchar *reqmsg) + + finalize_it: + if(root != NULL) +- cJSON_Delete(root); ++ fjson_object_put(root); + if(iRet != RS_RET_OK) { + STATSCOUNTER_INC(indexESFail, mutIndexESFail); + } +diff --git a/tests/es-bulk-errfile-empty.sh b/tests/es-bulk-errfile-empty.sh +index 1f27f62fe..95883cb3d 100755 +--- a/tests/es-bulk-errfile-empty.sh ++++ b/tests/es-bulk-errfile-empty.sh +@@ -12,6 +12,7 @@ echo \[es-bulk-errfile-empty\]: basic test for elasticsearch functionality + if [ -f rsyslog.errorfile ] + then + echo "error: error file exists!" ++ cat rsyslog.errorfile + exit 1 + fi + . $srcdir/diag.sh seq-check 0 9999 +-- +2.14.4 + diff --git a/SOURCES/rsyslog-8.24.0-rhbz1565214-omelasticsearch-write-op-types-bulk-rejection-retries.patch b/SOURCES/rsyslog-8.24.0-rhbz1565214-omelasticsearch-write-op-types-bulk-rejection-retries.patch new file mode 100644 index 0000000..14c7bf5 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1565214-omelasticsearch-write-op-types-bulk-rejection-retries.patch @@ -0,0 +1,738 @@ +From 989be897340eb458b00efedfd5e082bb362db79a Mon Sep 17 00:00:00 2001 +From: Rich Megginson +Date: Tue, 15 May 2018 16:03:25 -0600 +Subject: [PATCH 11/11] omelasticsearch: write op types; bulk rejection retries + +Add support for a 'create' write operation type in addition to +the default 'index'. Using create allows specifying a unique id +for each record, and allows duplicate document detection. + +Add support for checking each record returned in a bulk index +request response. Allow specifying a ruleset to send each failed +record to. Add a local variable `omes` which contains the +information in the error response, so that users can control how +to handle responses e.g. retry, or send to an error file. + +Add support for response stats - count successes, duplicates, and +different types of failures. + +Add testing for bulk index rejections. + +(cherry picked from commit 57dd368a2a915d79c94a8dc0de30c93a0bbdc8fe) +(cherry picked from commit 30a15621e1e7e393b2153e9fe5c13f724dea25b5) +--- + plugins/omelasticsearch/omelasticsearch.c | 441 ++++++++++++++++++++++++++++-- + 1 file changed, 415 insertions(+), 26 deletions(-) + +diff --git a/plugins/omelasticsearch/omelasticsearch.c b/plugins/omelasticsearch/omelasticsearch.c +index ed2b47535..ca61ae28f 100644 +--- a/plugins/omelasticsearch/omelasticsearch.c ++++ b/plugins/omelasticsearch/omelasticsearch.c +@@ -51,6 +51,8 @@ + #include "statsobj.h" + #include "cfsysline.h" + #include "unicode-helper.h" ++#include "ratelimit.h" ++#include "ruleset.h" + + #ifndef O_LARGEFILE + # define O_LARGEFILE 0 +@@ -64,6 +66,8 @@ MODULE_CNFNAME("omelasticsearch") + DEF_OMOD_STATIC_DATA + DEFobjCurrIf(errmsg) + DEFobjCurrIf(statsobj) ++DEFobjCurrIf(prop) ++DEFobjCurrIf(ruleset) + + statsobj_t *indexStats; + STATSCOUNTER_DEF(indexSubmit, mutIndexSubmit) +@@ -71,19 +75,35 @@ STATSCOUNTER_DEF(indexHTTPFail, mutIndexHTTPFail) + STATSCOUNTER_DEF(indexHTTPReqFail, mutIndexHTTPReqFail) + STATSCOUNTER_DEF(checkConnFail, mutCheckConnFail) + STATSCOUNTER_DEF(indexESFail, mutIndexESFail) ++STATSCOUNTER_DEF(indexSuccess, mutIndexSuccess) ++STATSCOUNTER_DEF(indexBadResponse, mutIndexBadResponse) ++STATSCOUNTER_DEF(indexDuplicate, mutIndexDuplicate) ++STATSCOUNTER_DEF(indexBadArgument, mutIndexBadArgument) ++STATSCOUNTER_DEF(indexBulkRejection, mutIndexBulkRejection) ++STATSCOUNTER_DEF(indexOtherResponse, mutIndexOtherResponse) + ++static prop_t *pInputName = NULL; + + # define META_STRT "{\"index\":{\"_index\": \"" ++# define META_STRT_CREATE "{\"create\":{\"_index\": \"" + # define META_TYPE "\",\"_type\":\"" + # define META_PARENT "\",\"_parent\":\"" + # define META_ID "\", \"_id\":\"" + # define META_END "\"}}\n" + ++typedef enum { ++ ES_WRITE_INDEX, ++ ES_WRITE_CREATE, ++ ES_WRITE_UPDATE, /* not supported */ ++ ES_WRITE_UPSERT /* not supported */ ++} es_write_ops_t; ++ + /* REST API for elasticsearch hits this URL: + * http://:// + */ ++/* bulk API uses /_bulk */ + typedef struct curl_slist HEADER; +-typedef struct _instanceData { ++typedef struct instanceConf_s { + int defaultPort; + int fdErrFile; /* error file fd or -1 if not open */ + pthread_mutex_t mutErrFile; +@@ -113,8 +133,25 @@ typedef struct _instanceData { + uchar *caCertFile; + uchar *myCertFile; + uchar *myPrivKeyFile; ++ es_write_ops_t writeOperation; ++ sbool retryFailures; ++ int ratelimitInterval; ++ int ratelimitBurst; ++ /* for retries */ ++ ratelimit_t *ratelimiter; ++ uchar *retryRulesetName; ++ ruleset_t *retryRuleset; ++ struct instanceConf_s *next; + } instanceData; + ++typedef instanceConf_t instanceData; ++ ++struct modConfData_s { ++ rsconf_t *pConf; /* our overall config object */ ++ instanceConf_t *root, *tail; ++}; ++static modConfData_t *loadModConf = NULL; /* modConf ptr to use for the current load process */ ++ + typedef struct wrkrInstanceData { + instanceData *pData; + int serverIndex; +@@ -160,7 +197,12 @@ static struct cnfparamdescr actpdescr[] = { + { "allowunsignedcerts", eCmdHdlrBinary, 0 }, + { "tls.cacert", eCmdHdlrString, 0 }, + { "tls.mycert", eCmdHdlrString, 0 }, +- { "tls.myprivkey", eCmdHdlrString, 0 } ++ { "tls.myprivkey", eCmdHdlrString, 0 }, ++ { "writeoperation", eCmdHdlrGetWord, 0 }, ++ { "retryfailures", eCmdHdlrBinary, 0 }, ++ { "ratelimit.interval", eCmdHdlrInt, 0 }, ++ { "ratelimit.burst", eCmdHdlrInt, 0 }, ++ { "retryruleset", eCmdHdlrString, 0 } + }; + static struct cnfparamblk actpblk = + { CNFPARAMBLK_VERSION, +@@ -177,6 +219,9 @@ CODESTARTcreateInstance + pData->caCertFile = NULL; + pData->myCertFile = NULL; + pData->myPrivKeyFile = NULL; ++ pData->ratelimiter = NULL; ++ pData->retryRulesetName = NULL; ++ pData->retryRuleset = NULL; + ENDcreateInstance + + BEGINcreateWrkrInstance +@@ -228,6 +273,9 @@ CODESTARTfreeInstance + free(pData->caCertFile); + free(pData->myCertFile); + free(pData->myPrivKeyFile); ++ free(pData->retryRulesetName); ++ if (pData->ratelimiter != NULL) ++ ratelimitDestruct(pData->ratelimiter); + ENDfreeInstance + + BEGINfreeWrkrInstance +@@ -285,6 +333,10 @@ CODESTARTdbgPrintInstInfo + dbgprintf("\ttls.cacert='%s'\n", pData->caCertFile); + dbgprintf("\ttls.mycert='%s'\n", pData->myCertFile); + dbgprintf("\ttls.myprivkey='%s'\n", pData->myPrivKeyFile); ++ dbgprintf("\twriteoperation='%d'\n", pData->writeOperation); ++ dbgprintf("\tretryfailures='%d'\n", pData->retryFailures); ++ dbgprintf("\tratelimit.interval='%d'\n", pData->ratelimitInterval); ++ dbgprintf("\tratelimit.burst='%d'\n", pData->ratelimitBurst); + ENDdbgPrintInstInfo + + +@@ -557,7 +609,11 @@ finalize_it: + static size_t + computeMessageSize(wrkrInstanceData_t *pWrkrData, uchar *message, uchar **tpls) + { +- size_t r = sizeof(META_STRT)-1 + sizeof(META_TYPE)-1 + sizeof(META_END)-1 + sizeof("\n")-1; ++ size_t r = sizeof(META_TYPE)-1 + sizeof(META_END)-1 + sizeof("\n")-1; ++ if (pWrkrData->pData->writeOperation == ES_WRITE_CREATE) ++ r += sizeof(META_STRT_CREATE)-1; ++ else ++ r += sizeof(META_STRT)-1; + + uchar *searchIndex = 0; + uchar *searchType; +@@ -594,7 +650,10 @@ buildBatch(wrkrInstanceData_t *pWrkrData, uchar *message, uchar **tpls) + DEFiRet; + + getIndexTypeAndParent(pWrkrData->pData, tpls, &searchIndex, &searchType, &parent, &bulkId); +- r = es_addBuf(&pWrkrData->batch.data, META_STRT, sizeof(META_STRT)-1); ++ if (pWrkrData->pData->writeOperation == ES_WRITE_CREATE) ++ r = es_addBuf(&pWrkrData->batch.data, META_STRT_CREATE, sizeof(META_STRT_CREATE)-1); ++ else ++ r = es_addBuf(&pWrkrData->batch.data, META_STRT, sizeof(META_STRT)-1); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, (char*)searchIndex, + ustrlen(searchIndex)); + if(r == 0) r = es_addBuf(&pWrkrData->batch.data, META_TYPE, sizeof(META_TYPE)-1); +@@ -709,13 +768,20 @@ static int checkReplyStatus(fjson_object* ok) { + + /* + * Context object for error file content creation or status check ++ * response_item - the full {"create":{"_index":"idxname",.....}} ++ * response_body - the inner hash of the response_item - {"_index":"idxname",...} ++ * status - the "status" field from the inner hash - "status":500 ++ * should be able to use fjson_object_get_int(status) to get the http result code + */ + typedef struct exeContext{ + int statusCheckOnly; + fjson_object *errRoot; +- rsRetVal (*prepareErrorFileContent)(struct exeContext *ctx,int itemStatus,char *request,char *response); +- +- ++ rsRetVal (*prepareErrorFileContent)(struct exeContext *ctx,int itemStatus,char *request,char *response, ++ fjson_object *response_item, fjson_object *response_body, fjson_object *status); ++ es_write_ops_t writeOperation; ++ ratelimit_t *ratelimiter; ++ ruleset_t *retryRuleset; ++ struct json_tokener *jTokener; + } context; + + /* +@@ -728,8 +794,15 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,fjson_object **p + fjson_object *replyRoot = *pReplyRoot; + int i; + int numitems; +- fjson_object *items=NULL; ++ fjson_object *items=NULL, *jo_errors = NULL; ++ int errors = 0; + ++ if(fjson_object_object_get_ex(replyRoot, "errors", &jo_errors)) { ++ errors = fjson_object_get_boolean(jo_errors); ++ if (!errors && pWrkrData->pData->retryFailures) { ++ return RS_RET_OK; ++ } ++ } + + /*iterate over items*/ + if(!fjson_object_object_get_ex(replyRoot, "items", &items)) { +@@ -741,7 +814,11 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,fjson_object **p + + numitems = fjson_object_array_length(items); + +- DBGPRINTF("omelasticsearch: Entire request %s\n",reqmsg); ++ if (reqmsg) { ++ DBGPRINTF("omelasticsearch: Entire request %s\n", reqmsg); ++ } else { ++ DBGPRINTF("omelasticsearch: Empty request\n"); ++ } + const char *lastReqRead= (char*)reqmsg; + + DBGPRINTF("omelasticsearch: %d items in reply\n", numitems); +@@ -769,8 +846,7 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,fjson_object **p + + char *request =0; + char *response =0; +- if(ctx->statusCheckOnly) +- { ++ if(ctx->statusCheckOnly || (NULL == lastReqRead)) { + if(itemStatus) { + DBGPRINTF("omelasticsearch: error in elasticsearch reply: item %d, status is %d\n", i, fjson_object_get_int(ok)); + DBGPRINTF("omelasticsearch: status check found error.\n"); +@@ -795,7 +871,8 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,fjson_object **p + } + + /*call the context*/ +- rsRetVal ret = ctx->prepareErrorFileContent(ctx, itemStatus, request,response); ++ rsRetVal ret = ctx->prepareErrorFileContent(ctx, itemStatus, request, ++ response, item, result, ok); + + /*free memory in any case*/ + free(request); +@@ -818,11 +895,14 @@ parseRequestAndResponseForContext(wrkrInstanceData_t *pWrkrData,fjson_object **p + * Dumps only failed requests of bulk insert + */ + static rsRetVal +-getDataErrorOnly(context *ctx,int itemStatus,char *request,char *response) ++getDataErrorOnly(context *ctx,int itemStatus,char *request,char *response, ++ fjson_object *response_item, fjson_object *response_body, fjson_object *status) + { + DEFiRet; +- if(itemStatus) +- { ++ (void)response_item; /* unused */ ++ (void)response_body; /* unused */ ++ (void)status; /* unused */ ++ if(itemStatus) { + fjson_object *onlyErrorResponses =NULL; + fjson_object *onlyErrorRequests=NULL; + +@@ -855,9 +935,16 @@ static rsRetVal + getDataInterleaved(context *ctx, + int __attribute__((unused)) itemStatus, + char *request, +- char *response) ++ char *response, ++ fjson_object *response_item, ++ fjson_object *response_body, ++ fjson_object *status ++) + { + DEFiRet; ++ (void)response_item; /* unused */ ++ (void)response_body; /* unused */ ++ (void)status; /* unused */ + fjson_object *interleaved =NULL; + if(!fjson_object_object_get_ex(ctx->errRoot, "response", &interleaved)) + { +@@ -889,11 +976,13 @@ getDataInterleaved(context *ctx, + */ + + static rsRetVal +-getDataErrorOnlyInterleaved(context *ctx,int itemStatus,char *request,char *response) ++getDataErrorOnlyInterleaved(context *ctx,int itemStatus,char *request,char *response, ++ fjson_object *response_item, fjson_object *response_body, fjson_object *status) + { + DEFiRet; + if (itemStatus) { +- if(getDataInterleaved(ctx, itemStatus,request,response)!= RS_RET_OK) { ++ if(getDataInterleaved(ctx, itemStatus,request,response, ++ response_item, response_body, status)!= RS_RET_OK) { + ABORT_FINALIZE(RS_RET_ERR); + } + } +@@ -902,6 +991,141 @@ getDataErrorOnlyInterleaved(context *ctx,int itemStatus,char *request,char *resp + RETiRet; + } + ++/* request string looks like this: ++ * "{\"create\":{\"_index\": \"rsyslog_testbench\",\"_type\":\"test-type\", ++ * \"_id\":\"FAEAFC0D17C847DA8BD6F47BC5B3800A\"}}\n ++ * {\"msgnum\":\"x00000000\",\"viaq_msg_id\":\"FAEAFC0D17C847DA8BD6F47BC5B3800A\"}\n" ++ * we don't want the meta header, only the data part ++ * start = first \n + 1 ++ * end = last \n ++ */ ++static rsRetVal ++createMsgFromRequest(const char *request, context *ctx, smsg_t **msg) ++{ ++ DEFiRet; ++ fjson_object *jo_msg = NULL; ++ const char *datastart, *dataend; ++ size_t datalen; ++ enum json_tokener_error json_error; ++ ++ *msg = NULL; ++ if (!(datastart = strchr(request, '\n')) || (datastart[1] != '{')) { ++ LogError(0, RS_RET_ERR, ++ "omelasticsearch: malformed original request - " ++ "could not find start of original data [%s]", ++ request); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ datastart++; /* advance to { */ ++ if (!(dataend = strchr(datastart, '\n')) || (dataend[1] != '\0')) { ++ LogError(0, RS_RET_ERR, ++ "omelasticsearch: malformed original request - " ++ "could not find end of original data [%s]", ++ request); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ datalen = dataend - datastart; ++ json_tokener_reset(ctx->jTokener); ++ fjson_object *jo_request = json_tokener_parse_ex(ctx->jTokener, datastart, datalen); ++ json_error = fjson_tokener_get_error(ctx->jTokener); ++ if (!jo_request || (json_error != fjson_tokener_success)) { ++ LogError(0, RS_RET_ERR, ++ "omelasticsearch: parse error [%s] - could not convert original " ++ "request JSON back into JSON object [%s]", ++ fjson_tokener_error_desc(json_error), request); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ ++ CHKiRet(msgConstruct(msg)); ++ MsgSetFlowControlType(*msg, eFLOWCTL_FULL_DELAY); ++ MsgSetInputName(*msg, pInputName); ++ if (fjson_object_object_get_ex(jo_request, "message", &jo_msg)) { ++ const char *rawmsg = json_object_get_string(jo_msg); ++ const size_t msgLen = (size_t)json_object_get_string_len(jo_msg); ++ MsgSetRawMsg(*msg, rawmsg, msgLen); ++ } else { ++ MsgSetRawMsg(*msg, request, strlen(request)); ++ } ++ MsgSetMSGoffs(*msg, 0); /* we do not have a header... */ ++ CHKiRet(msgAddJSON(*msg, (uchar*)"!", jo_request, 0, 0)); ++ ++ finalize_it: ++ RETiRet; ++ ++} ++ ++ ++static rsRetVal ++getDataRetryFailures(context *ctx,int itemStatus,char *request,char *response, ++ fjson_object *response_item, fjson_object *response_body, fjson_object *status) ++{ ++ DEFiRet; ++ fjson_object *omes = NULL, *jo = NULL; ++ int istatus = fjson_object_get_int(status); ++ int iscreateop = 0; ++ struct json_object_iterator it = json_object_iter_begin(response_item); ++ struct json_object_iterator itEnd = json_object_iter_end(response_item); ++ const char *optype = NULL; ++ smsg_t *msg = NULL; ++ ++ (void)response; ++ (void)itemStatus; ++ CHKiRet(createMsgFromRequest(request, ctx, &msg)); ++ CHKmalloc(msg); ++ /* add status as local variables */ ++ omes = json_object_new_object(); ++ if (!json_object_iter_equal(&it, &itEnd)) ++ optype = json_object_iter_peek_name(&it); ++ if (optype && !strcmp("create", optype)) ++ iscreateop = 1; ++ if (optype && !strcmp("index", optype) && (ctx->writeOperation == ES_WRITE_INDEX)) ++ iscreateop = 1; ++ if (optype) { ++ jo = json_object_new_string(optype); ++ } else { ++ jo = json_object_new_string("unknown"); ++ } ++ json_object_object_add(omes, "writeoperation", jo); ++ ++ if (!optype) { ++ STATSCOUNTER_INC(indexBadResponse, mutIndexBadResponse); ++ } else if ((istatus == 200) || (istatus == 201)) { ++ STATSCOUNTER_INC(indexSuccess, mutIndexSuccess); ++ } else if ((istatus == 409) && iscreateop) { ++ STATSCOUNTER_INC(indexDuplicate, mutIndexDuplicate); ++ } else if (istatus == 400 || (istatus < 200)) { ++ STATSCOUNTER_INC(indexBadArgument, mutIndexBadArgument); ++ } else { ++ fjson_object *error = NULL, *errtype = NULL; ++ if(fjson_object_object_get_ex(response_body, "error", &error) && ++ fjson_object_object_get_ex(error, "type", &errtype)) { ++ if (istatus == 429) { ++ STATSCOUNTER_INC(indexBulkRejection, mutIndexBulkRejection); ++ } else { ++ STATSCOUNTER_INC(indexOtherResponse, mutIndexOtherResponse); ++ } ++ } else { ++ STATSCOUNTER_INC(indexBadResponse, mutIndexBadResponse); ++ } ++ } ++ /* add response_body fields to local var omes */ ++ it = json_object_iter_begin(response_body); ++ itEnd = json_object_iter_end(response_body); ++ while (!json_object_iter_equal(&it, &itEnd)) { ++ json_object_object_add(omes, json_object_iter_peek_name(&it), ++ json_object_get(json_object_iter_peek_value(&it))); ++ json_object_iter_next(&it); ++ } ++ CHKiRet(msgAddJSON(msg, (uchar*)".omes", omes, 0, 0)); ++ omes = NULL; ++ MsgSetRuleset(msg, ctx->retryRuleset); ++ CHKiRet(ratelimitAddMsg(ctx->ratelimiter, NULL, msg)); ++finalize_it: ++ if (omes) ++ json_object_put(omes); ++ RETiRet; ++} ++ + /* + * get erroronly context + */ +@@ -979,6 +1203,23 @@ initializeErrorInterleavedConext(wrkrInstanceData_t *pWrkrData,context *ctx){ + RETiRet; + } + ++/*get retry failures context*/ ++static rsRetVal ++initializeRetryFailuresContext(wrkrInstanceData_t *pWrkrData,context *ctx){ ++ DEFiRet; ++ ctx->statusCheckOnly=0; ++ fjson_object *errRoot=NULL; ++ if((errRoot=fjson_object_new_object()) == NULL) ABORT_FINALIZE(RS_RET_ERR); ++ ++ ++ fjson_object_object_add(errRoot, "url", fjson_object_new_string((char*)pWrkrData->restURL)); ++ ctx->errRoot = errRoot; ++ ctx->prepareErrorFileContent= &getDataRetryFailures; ++ CHKmalloc(ctx->jTokener = json_tokener_new()); ++ finalize_it: ++ RETiRet; ++} ++ + + /* write data error request/replies to separate error file + * Note: we open the file but never close it before exit. If it +@@ -994,6 +1235,10 @@ writeDataError(wrkrInstanceData_t *pWrkrData, instanceData *pData, fjson_object + char errStr[1024]; + context ctx; + ctx.errRoot=0; ++ ctx.writeOperation = pWrkrData->pData->writeOperation; ++ ctx.ratelimiter = pWrkrData->pData->ratelimiter; ++ ctx.retryRuleset = pWrkrData->pData->retryRuleset; ++ ctx.jTokener = NULL; + DEFiRet; + + if(pData->errorFile == NULL) { +@@ -1039,9 +1284,12 @@ writeDataError(wrkrInstanceData_t *pWrkrData, instanceData *pData, fjson_object + DBGPRINTF("omelasticsearch: error initializing error interleaved context.\n"); + ABORT_FINALIZE(RS_RET_ERR); + } +- } +- else +- { ++ } else if(pData->retryFailures) { ++ if(initializeRetryFailuresContext(pWrkrData, &ctx) != RS_RET_OK) { ++ DBGPRINTF("omelasticsearch: error initializing retry failures context.\n"); ++ ABORT_FINALIZE(RS_RET_ERR); ++ } ++ } else { + DBGPRINTF("omelasticsearch: None of the modes match file write. No data to write.\n"); + ABORT_FINALIZE(RS_RET_ERR); + } +@@ -1082,25 +1330,38 @@ finalize_it: + if(bMutLocked) + pthread_mutex_unlock(&pData->mutErrFile); + fjson_object_put(ctx.errRoot); ++ if (ctx.jTokener) ++ json_tokener_free(ctx.jTokener); ++ free(rendered); + RETiRet; + } + + + static rsRetVal +-checkResultBulkmode(wrkrInstanceData_t *pWrkrData, fjson_object *root) ++checkResultBulkmode(wrkrInstanceData_t *pWrkrData, fjson_object *root, uchar *reqmsg) + { + DEFiRet; + context ctx; +- ctx.statusCheckOnly=1; + ctx.errRoot = 0; +- if(parseRequestAndResponseForContext(pWrkrData,&root,0,&ctx)!= RS_RET_OK) +- { ++ ctx.writeOperation = pWrkrData->pData->writeOperation; ++ ctx.ratelimiter = pWrkrData->pData->ratelimiter; ++ ctx.retryRuleset = pWrkrData->pData->retryRuleset; ++ ctx.statusCheckOnly=1; ++ ctx.jTokener = NULL; ++ if (pWrkrData->pData->retryFailures) { ++ ctx.statusCheckOnly=0; ++ CHKiRet(initializeRetryFailuresContext(pWrkrData, &ctx)); ++ } ++ if(parseRequestAndResponseForContext(pWrkrData,&root,reqmsg,&ctx)!= RS_RET_OK) { + DBGPRINTF("omelasticsearch: error found in elasticsearch reply\n"); + ABORT_FINALIZE(RS_RET_DATAFAIL); + } + +- finalize_it: +- RETiRet; ++finalize_it: ++ fjson_object_put(ctx.errRoot); ++ if (ctx.jTokener) ++ json_tokener_free(ctx.jTokener); ++ RETiRet; + } + + +@@ -1118,7 +1378,7 @@ checkResult(wrkrInstanceData_t *pWrkrData, uchar *reqmsg) + } + + if(pWrkrData->pData->bulkmode) { +- iRet = checkResultBulkmode(pWrkrData, root); ++ iRet = checkResultBulkmode(pWrkrData, root, reqmsg); + } else { + if(fjson_object_object_get_ex(root, "status", &status)) { + iRet = RS_RET_DATAFAIL; +@@ -1397,6 +1657,13 @@ setInstParamDefaults(instanceData *pData) + pData->caCertFile = NULL; + pData->myCertFile = NULL; + pData->myPrivKeyFile = NULL; ++ pData->writeOperation = ES_WRITE_INDEX; ++ pData->retryFailures = 0; ++ pData->ratelimitBurst = 20000; ++ pData->ratelimitInterval = 600; ++ pData->ratelimiter = NULL; ++ pData->retryRulesetName = NULL; ++ pData->retryRuleset = NULL; + } + + BEGINnewActInst +@@ -1495,6 +1762,27 @@ CODESTARTnewActInst + } else { + fclose(fp); + } ++ } else if(!strcmp(actpblk.descr[i].name, "writeoperation")) { ++ char *writeop = es_str2cstr(pvals[i].val.d.estr, NULL); ++ if (writeop && !strcmp(writeop, "create")) { ++ pData->writeOperation = ES_WRITE_CREATE; ++ } else if (writeop && !strcmp(writeop, "index")) { ++ pData->writeOperation = ES_WRITE_INDEX; ++ } else if (writeop) { ++ errmsg.LogError(0, RS_RET_CONFIG_ERROR, ++ "omelasticsearch: invalid value '%s' for writeoperation: " ++ "must be one of 'index' or 'create' - using default value 'index'", writeop); ++ pData->writeOperation = ES_WRITE_INDEX; ++ } ++ free(writeop); ++ } else if(!strcmp(actpblk.descr[i].name, "retryfailures")) { ++ pData->retryFailures = pvals[i].val.d.n; ++ } else if(!strcmp(actpblk.descr[i].name, "ratelimit.burst")) { ++ pData->ratelimitBurst = (int) pvals[i].val.d.n; ++ } else if(!strcmp(actpblk.descr[i].name, "ratelimit.interval")) { ++ pData->ratelimitInterval = (int) pvals[i].val.d.n; ++ } else if(!strcmp(actpblk.descr[i].name, "retryruleset")) { ++ pData->retryRulesetName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else { + dbgprintf("omelasticsearch: program error, non-handled " + "param '%s'\n", actpblk.descr[i].name); +@@ -1661,6 +1949,27 @@ CODESTARTnewActInst + pData->searchIndex = (uchar*) strdup("system"); + if(pData->searchType == NULL) + pData->searchType = (uchar*) strdup("events"); ++ ++ if ((pData->writeOperation != ES_WRITE_INDEX) && (pData->bulkId == NULL)) { ++ errmsg.LogError(0, RS_RET_CONFIG_ERROR, ++ "omelasticsearch: writeoperation '%d' requires bulkid", pData->writeOperation); ++ ABORT_FINALIZE(RS_RET_CONFIG_ERROR); ++ } ++ ++ if (pData->retryFailures) { ++ CHKiRet(ratelimitNew(&pData->ratelimiter, "omelasticsearch", NULL)); ++ ratelimitSetLinuxLike(pData->ratelimiter, pData->ratelimitInterval, pData->ratelimitBurst); ++ ratelimitSetNoTimeCache(pData->ratelimiter); ++ } ++ ++ /* node created, let's add to list of instance configs for the module */ ++ if(loadModConf->tail == NULL) { ++ loadModConf->tail = loadModConf->root = pData; ++ } else { ++ loadModConf->tail->next = pData; ++ loadModConf->tail = pData; ++ } ++ + CODE_STD_FINALIZERnewActInst + cnfparamvalsDestruct(pvals, &actpblk); + if (serverParam) +@@ -1680,6 +1989,51 @@ CODE_STD_STRING_REQUESTparseSelectorAct(1) + CODE_STD_FINALIZERparseSelectorAct + ENDparseSelectorAct + ++ ++BEGINbeginCnfLoad ++CODESTARTbeginCnfLoad ++ loadModConf = pModConf; ++ pModConf->pConf = pConf; ++ pModConf->root = pModConf->tail = NULL; ++ENDbeginCnfLoad ++ ++ ++BEGINendCnfLoad ++CODESTARTendCnfLoad ++ loadModConf = NULL; /* done loading */ ++ENDendCnfLoad ++ ++ ++BEGINcheckCnf ++ instanceConf_t *inst; ++CODESTARTcheckCnf ++ for(inst = pModConf->root ; inst != NULL ; inst = inst->next) { ++ ruleset_t *pRuleset; ++ rsRetVal localRet; ++ ++ if (inst->retryRulesetName) { ++ localRet = ruleset.GetRuleset(pModConf->pConf, &pRuleset, inst->retryRulesetName); ++ if(localRet == RS_RET_NOT_FOUND) { ++ errmsg.LogError(0, localRet, "omelasticsearch: retryruleset '%s' not found - " ++ "no retry ruleset will be used", inst->retryRulesetName); ++ } else { ++ inst->retryRuleset = pRuleset; ++ } ++ } ++ } ++ENDcheckCnf ++ ++ ++BEGINactivateCnf ++CODESTARTactivateCnf ++ENDactivateCnf ++ ++ ++BEGINfreeCnf ++CODESTARTfreeCnf ++ENDfreeCnf ++ ++ + BEGINdoHUP + CODESTARTdoHUP + if(pData->fdErrFile != -1) { +@@ -1691,10 +2045,14 @@ ENDdoHUP + + BEGINmodExit + CODESTARTmodExit ++ if(pInputName != NULL) ++ prop.Destruct(&pInputName); + curl_global_cleanup(); + statsobj.Destruct(&indexStats); + objRelease(errmsg, CORE_COMPONENT); +- objRelease(statsobj, CORE_COMPONENT); ++ objRelease(statsobj, CORE_COMPONENT); ++ objRelease(prop, CORE_COMPONENT); ++ objRelease(ruleset, CORE_COMPONENT); + ENDmodExit + + BEGINqueryEtryPt +@@ -1705,6 +2063,7 @@ CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES + CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES + CODEqueryEtryPt_doHUP + CODEqueryEtryPt_TXIF_OMOD_QUERIES /* we support the transactional interface! */ ++CODEqueryEtryPt_STD_CONF2_QUERIES + ENDqueryEtryPt + + +@@ -1714,6 +2073,8 @@ CODESTARTmodInit + CODEmodInit_QueryRegCFSLineHdlr + CHKiRet(objUse(errmsg, CORE_COMPONENT)); + CHKiRet(objUse(statsobj, CORE_COMPONENT)); ++ CHKiRet(objUse(prop, CORE_COMPONENT)); ++ CHKiRet(objUse(ruleset, CORE_COMPONENT)); + + if (curl_global_init(CURL_GLOBAL_ALL) != 0) { + errmsg.LogError(0, RS_RET_OBJ_CREATION_FAILED, "CURL fail. -elasticsearch indexing disabled"); +@@ -1739,7 +2100,28 @@ CODEmodInit_QueryRegCFSLineHdlr + STATSCOUNTER_INIT(indexESFail, mutIndexESFail); + CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"failed.es", + ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexESFail)); ++ STATSCOUNTER_INIT(indexSuccess, mutIndexSuccess); ++ CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"response.success", ++ ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexSuccess)); ++ STATSCOUNTER_INIT(indexBadResponse, mutIndexBadResponse); ++ CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"response.bad", ++ ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexBadResponse)); ++ STATSCOUNTER_INIT(indexDuplicate, mutIndexDuplicate); ++ CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"response.duplicate", ++ ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexDuplicate)); ++ STATSCOUNTER_INIT(indexBadArgument, mutIndexBadArgument); ++ CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"response.badargument", ++ ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexBadArgument)); ++ STATSCOUNTER_INIT(indexBulkRejection, mutIndexBulkRejection); ++ CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"response.bulkrejection", ++ ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexBulkRejection)); ++ STATSCOUNTER_INIT(indexOtherResponse, mutIndexOtherResponse); ++ CHKiRet(statsobj.AddCounter(indexStats, (uchar *)"response.other", ++ ctrType_IntCtr, CTR_FLAG_RESETTABLE, &indexOtherResponse)); + CHKiRet(statsobj.ConstructFinalize(indexStats)); ++ CHKiRet(prop.Construct(&pInputName)); ++ CHKiRet(prop.SetString(pInputName, UCHAR_CONSTANT("omelasticsearch"), sizeof("omelasticsearch") - 1)); ++ CHKiRet(prop.ConstructFinalize(pInputName)); + ENDmodInit + + /* vi:set ai: diff --git a/SOURCES/rsyslog-8.24.0-rhbz1582517-buffer-overflow-memcpy-in-parser.patch b/SOURCES/rsyslog-8.24.0-rhbz1582517-buffer-overflow-memcpy-in-parser.patch new file mode 100644 index 0000000..96d0695 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1582517-buffer-overflow-memcpy-in-parser.patch @@ -0,0 +1,46 @@ +From cc3098b63174b8aa875d1f2e9c6ea94407b211b8 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Thu, 16 Feb 2017 19:02:36 +0100 +Subject: [PATCH 04/11] Bug 1582517 - rsyslog: Buffer overflow in memcpy() in parser.c + +core: fix potential misadressing in parser message sanitizer + +misadressing could happen when an oversize message made it to the +sanitizer AND contained a control character in the oversize part +of the message. Note that it is an error in itself that such an +oversize message enters the system, but we harden the sanitizer +to handle this gracefully (it will truncate the message). + +Note that truncation may still - as previously - happen if the +number of escape characters makes the string grow above the max +message size. + +(cherry picked from commit 20f8237870eb5e971fa068e4dd4d296f1dbef329) +--- + runtime/parser.c | 8 +++++++- + 1 file changed, 7 insertions(+), 1 deletion(-) + +diff --git a/runtime/parser.c b/runtime/parser.c +index 0574d982a..9645baa40 100644 +--- a/runtime/parser.c ++++ b/runtime/parser.c +@@ -464,9 +464,15 @@ SanitizeMsg(smsg_t *pMsg) + if(maxDest < sizeof(szSanBuf)) + pDst = szSanBuf; + else +- CHKmalloc(pDst = MALLOC(iMaxLine + 1)); ++ CHKmalloc(pDst = MALLOC(maxDest + 1)); + if(iSrc > 0) { + iSrc--; /* go back to where everything is OK */ ++ if(iSrc > maxDest) { ++ DBGPRINTF("parser.Sanitize: have oversize index %zd, " ++ "max %zd - corrected, but should not happen\n", ++ iSrc, maxDest); ++ iSrc = maxDest; ++ } + memcpy(pDst, pszMsg, iSrc); /* fast copy known good */ + } + iDst = iSrc; +-- +2.14.4 + diff --git a/SOURCES/rsyslog-8.24.0-rhbz1591819-msg-loss-shutdown.patch b/SOURCES/rsyslog-8.24.0-rhbz1591819-msg-loss-shutdown.patch new file mode 100644 index 0000000..472823b --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1591819-msg-loss-shutdown.patch @@ -0,0 +1,63 @@ +From 59627f23bee26f3acec19d491d5884bcd1fb672e Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Wed, 6 Jun 2018 17:30:21 +0200 +Subject: [PATCH] core: fix message loss on target unavailibility during + shutdown + +Triggering condition: +- action queue in disk mode (or DA) +- batch is being processed by failed action in retry mode +- rsyslog is shut down without resuming action + +In these cases messages may be lost by not properly writing them +back to the disk queue. + +closes https://github.com/rsyslog/rsyslog/issues/2760 +--- + action.c | 11 +++++++++-- + runtime/queue.c | 3 +++ + 2 files changed, 12 insertions(+), 2 deletions(-) + +diff --git a/action.c b/action.c +index a9f886a43..39fcb1c19 100644 +--- a/action.c ++++ b/action.c +@@ -1554,8 +1554,15 @@ processBatchMain(void *__restrict__ const pVoid, + /* we do not check error state below, because aborting would be + * more harmful than continuing. + */ +- processMsgMain(pAction, pWti, pBatch->pElem[i].pMsg, &ttNow); +- batchSetElemState(pBatch, i, BATCH_STATE_COMM); ++ rsRetVal localRet = processMsgMain(pAction, pWti, pBatch->pElem[i].pMsg, &ttNow); ++ DBGPRINTF("processBatchMain: i %d, processMsgMain iRet %d\n", i, localRet); ++ if( localRet == RS_RET_OK ++ || localRet == RS_RET_DEFER_COMMIT ++ || localRet == RS_RET_ACTION_FAILED ++ || localRet == RS_RET_PREVIOUS_COMMITTED ) { ++ batchSetElemState(pBatch, i, BATCH_STATE_COMM); ++ DBGPRINTF("processBatchMain: i %d, COMM state set\n", i); ++ } + } + } + +diff --git a/runtime/queue.c b/runtime/queue.c +index 74cc217d1..fd163a49f 100644 +--- a/runtime/queue.c ++++ b/runtime/queue.c +@@ -1666,6 +1666,7 @@ DeleteProcessedBatch(qqueue_t *pThis, batch_t *pBatch) + + for(i = 0 ; i < pBatch->nElem ; ++i) { + pMsg = pBatch->pElem[i].pMsg; ++ DBGPRINTF("DeleteProcessedBatch: etry %d state %d\n", i, pBatch->eltState[i]); + if( pBatch->eltState[i] == BATCH_STATE_RDY + || pBatch->eltState[i] == BATCH_STATE_SUB) { + localRet = doEnqSingleObj(pThis, eFLOWCTL_NO_DELAY, MsgAddRef(pMsg)); +@@ -1778,6 +1779,8 @@ DequeueConsumableElements(qqueue_t *pThis, wti_t *pWti, int *piRemainingQueueSiz + /* it is sufficient to persist only when the bulk of work is done */ + qqueueChkPersist(pThis, nDequeued+nDiscarded+nDeleted); + ++ DBGOPRINT((obj_t*) pThis, "dequeued %d consumable elements, szlog %d sz phys %d\n", ++ nDequeued, getLogicalQueueSize(pThis), getPhysicalQueueSize(pThis)); + pWti->batch.nElem = nDequeued; + pWti->batch.nElemDeq = nDequeued + nDiscarded; + pWti->batch.deqID = getNextDeqID(pThis); diff --git a/SOURCES/rsyslog-8.24.0-rhbz1597264-man-page-fix.patch b/SOURCES/rsyslog-8.24.0-rhbz1597264-man-page-fix.patch new file mode 100644 index 0000000..74d3395 --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1597264-man-page-fix.patch @@ -0,0 +1,30 @@ +diff --git a/tools/rsyslogd.8 b/tools/rsyslogd.8 +index 77d0f97..d9a2e32 100644 +--- a/tools/rsyslogd.8 ++++ b/tools/rsyslogd.8 +@@ -127,14 +127,14 @@ reacts to a set of signals. You may easily send a signal to + using the following: + .IP + .nf +-kill -SIGNAL $(cat /var/run/rsyslogd.pid) ++kill -SIGNAL $(cat /var/run/syslogd.pid) + .fi + .PP + Note that -SIGNAL must be replaced with the actual signal + you are trying to send, e.g. with HUP. So it then becomes: + .IP + .nf +-kill -HUP $(cat /var/run/rsyslogd.pid) ++kill -HUP $(cat /var/run/syslogd.pid) + .fi + .PP + .TP +@@ -215,7 +215,7 @@ for exact information. + .I /dev/log + The Unix domain socket to from where local syslog messages are read. + .TP +-.I /var/run/rsyslogd.pid ++.I /var/run/syslogd.pid + The file containing the process id of + .BR rsyslogd . + .TP diff --git a/SOURCES/rsyslog-8.24.0-rhbz1600462-wrktable-realloc-null.patch b/SOURCES/rsyslog-8.24.0-rhbz1600462-wrktable-realloc-null.patch new file mode 100644 index 0000000..81f58ee --- /dev/null +++ b/SOURCES/rsyslog-8.24.0-rhbz1600462-wrktable-realloc-null.patch @@ -0,0 +1,64 @@ +From f2f67932a37539080b7ad3403dd073df3511a410 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Fri, 27 Oct 2017 08:36:19 +0200 +Subject: [PATCH] core/action: fix NULL pointer access under OOM condition + +If a new worker was started while the system ran out of memory +a NULL pointer access could happen. The patch handles this more +gracefully. + +Detected by Coverty Scan, CID 185342. +--- + action.c | 17 +++++++++++++---- + 1 file changed, 13 insertions(+), 4 deletions(-) + +diff --git a/action.c b/action.c +index 986501074..4e467ee8b 100644 +--- a/action.c ++++ b/action.c +@@ -822,8 +822,9 @@ actionDoRetry(action_t * const pThis, wti_t * const pWti) + + + static rsRetVal +-actionCheckAndCreateWrkrInstance(action_t * const pThis, wti_t * const pWti) ++actionCheckAndCreateWrkrInstance(action_t * const pThis, const wti_t *const pWti) + { ++ int locked = 0; + DEFiRet; + if(pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData == NULL) { + DBGPRINTF("wti %p: we need to create a new action worker instance for " +@@ -836,23 +837,31 @@ actionCheckAndCreateWrkrInstance(action_t * const pThis, wti_t * const pWti) + /* maintain worker data table -- only needed if wrkrHUP is requested! */ + + pthread_mutex_lock(&pThis->mutWrkrDataTable); ++ locked = 1; + int freeSpot; + for(freeSpot = 0 ; freeSpot < pThis->wrkrDataTableSize ; ++freeSpot) + if(pThis->wrkrDataTable[freeSpot] == NULL) + break; + if(pThis->nWrkr == pThis->wrkrDataTableSize) { +- // TODO: check realloc, fall back to old table if it fails. Better than nothing... +- pThis->wrkrDataTable = realloc(pThis->wrkrDataTable, ++ void *const newTable = realloc(pThis->wrkrDataTable, + (pThis->wrkrDataTableSize + 1) * sizeof(void*)); ++ if(newTable == NULL) { ++ DBGPRINTF("actionCheckAndCreateWrkrInstance: out of " ++ "memory realloc wrkrDataTable\n") ++ ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); ++ } ++ pThis->wrkrDataTable = newTable; + pThis->wrkrDataTableSize++; + } + pThis->wrkrDataTable[freeSpot] = pWti->actWrkrInfo[pThis->iActionNbr].actWrkrData; + pThis->nWrkr++; +- pthread_mutex_unlock(&pThis->mutWrkrDataTable); + DBGPRINTF("wti %p: created action worker instance %d for " + "action %d\n", pWti, pThis->nWrkr, pThis->iActionNbr); + } + finalize_it: ++ if(locked) { ++ pthread_mutex_unlock(&pThis->mutWrkrDataTable); ++ } + RETiRet; + } + diff --git a/SPECS/rsyslog.spec b/SPECS/rsyslog.spec index 9eeefb8..7f5f89e 100644 --- a/SPECS/rsyslog.spec +++ b/SPECS/rsyslog.spec @@ -14,10 +14,9 @@ Summary: Enhanced system logging and kernel message trapping daemon Name: rsyslog Version: 8.24.0 -Release: 16%{?dist}.4 +Release: 34%{?dist} License: (GPLv3+ and ASL 2.0) Group: System Environment/Daemons -ExcludeArch: i686 ppc s390 URL: http://www.rsyslog.com/ Source0: http://www.rsyslog.com/files/download/rsyslog/%{name}-%{version}.tar.gz Source1: http://www.rsyslog.com/files/download/rsyslog/%{name}-doc-%{version}.tar.gz @@ -63,7 +62,7 @@ Patch7: rsyslog-8.24.0-rhbz1401870-watermark.patch Patch8: rsyslog-8.24.0-rhbz1403831-missing-cmd-line-switches.patch Patch9: rsyslog-8.24.0-rhbz1245194-imjournal-ste-file.patch -Patch10: rsyslog-8.24.0-rhbz1507028-recover_qi-doc.patch +Patch10: rsyslog-8.24.0-doc-rhbz1507028-recover_qi.patch Patch11: rsyslog-8.24.0-rhbz1088021-systemd-time-backwards.patch Patch12: rsyslog-8.24.0-rhbz1403907-imudp-deprecated-parameter.patch Patch13: rsyslog-8.24.0-rhbz1196230-ratelimit-add-source.patch @@ -81,13 +80,29 @@ Patch21: rsyslog-8.24.0-rhbz1431616-pmrfc3164sd-backport.patch Patch22: rsyslog-8.24.0-rhbz1056548-getaddrinfo.patch Patch23: rsyslog-8.24.0-rhbz1401456-sd-service-network.patch -Patch24: rsyslog-8.24.0-rhbz1459896-queues-defaults-doc.patch +Patch24: rsyslog-8.24.0-doc-rhbz1459896-queues-defaults.patch Patch25: rsyslog-8.24.0-rhbz1497985-journal-reloaded-message.patch Patch26: rsyslog-8.24.0-rhbz1462160-set.statement-crash.patch Patch27: rsyslog-8.24.0-rhbz1488186-fixed-nullptr-check.patch Patch28: rsyslog-8.24.0-rhbz1505103-omrelp-rebindinterval.patch -Patch29: rsyslog-8.24.0-rhbz1545582-imjournal-duplicates.patch +Patch29: rsyslog-8.24.0-rhbz1538372-imjournal-duplicates.patch +Patch30: rsyslog-8.24.0-rhbz1511485-deserialize-property-name.patch + +Patch31: rsyslog-8.24.0-rhbz1512551-caching-sockaddr.patch +Patch32: rsyslog-8.24.0-rhbz1531295-imfile-rewrite-with-symlink.patch +Patch33: rsyslog-8.24.0-rhbz1582517-buffer-overflow-memcpy-in-parser.patch +Patch34: rsyslog-8.24.0-rhbz1591819-msg-loss-shutdown.patch +Patch35: rsyslog-8.24.0-rhbz1539193-mmkubernetes-new-plugin.patch +Patch36: rsyslog-8.24.0-rhbz1507145-omelastic-client-cert.patch +Patch37: rsyslog-8.24.0-doc-rhbz1507145-omelastic-client-cert-and-config.patch +Patch38: rsyslog-8.24.0-rhbz1565214-omelasticsearch-replace-cJSON-with-libfastjson.patch +Patch39: rsyslog-8.24.0-rhbz1565214-omelasticsearch-write-op-types-bulk-rejection-retries.patch +Patch40: rsyslog-8.24.0-doc-rhbz1539193-mmkubernetes-new-plugin.patch +Patch41: rsyslog-8.24.0-doc-rhbz1538372-imjournal-duplicates.patch +Patch42: rsyslog-8.24.0-rhbz1597264-man-page-fix.patch +Patch43: rsyslog-8.24.0-rhbz1559408-async-writer.patch +Patch44: rsyslog-8.24.0-rhbz1600462-wrktable-realloc-null.patch %package crypto Summary: Encryption support @@ -124,7 +139,8 @@ Requires: %name = %version-%release Summary: Log normalization support for rsyslog Group: System Environment/Daemons Requires: %name = %version-%release -BuildRequires: libee-devel liblognorm-devel +BuildRequires: libee-devel +BuildRequires: liblognorm-devel %package mmaudit Summary: Message modification module supporting Linux audit format @@ -202,6 +218,18 @@ Group: System Environment/Daemons Requires: %name = %version-%release BuildRequires: libnet-devel +%package kafka +Summary: Provides kafka support for rsyslog +Group: System Environment/Daemons +Requires: %name = %version-%release +BuildRequires: librdkafka-devel + +%package mmkubernetes +Summary: Provides the mmkubernetes module +Group: System Environment/Daemons +Requires: %name = %version-%release +BuildRequires: libcurl-devel + %description Rsyslog is an enhanced, multi-threaded syslog daemon. It supports MySQL, syslog/TCP, RFC 3195, permitted sender lists, filtering on any message part, @@ -289,12 +317,22 @@ This module is similar to the regular UDP forwarder, but permits to spoof the sender address. Also, it enables to circle through a number of source ports. +%description kafka +The rsyslog-kafka package provides module for Apache Kafka output. + +%description mmkubernetes +The rsyslog-mmkubernetes package provides module for adding kubernetes +container metadata. + %prep # set up rsyslog-doc sources %setup -q -a 1 -T -c %patch4 -p1 %patch10 -p1 %patch24 -p1 +%patch37 -p1 +%patch40 -p1 +%patch41 -p1 #regenerate the docs mv build/searchindex.js searchindex_backup.js sphinx-build -b html source build @@ -334,13 +372,29 @@ mv build doc %patch22 -p1 -b .getaddrinfo %patch23 -p1 -b .sd-service-network -#%%patch24 is applied right after doc setup +#%patch24 is applied right after doc setup %patch25 -p1 -b .journal-reloaded %patch26 -p1 -b .set-statement-crash %patch27 -p1 -b .nullptr-check %patch28 -p1 -b .rebindinterval %patch29 -p1 -b .imjournal-duplicates +%patch30 -p1 -b .property-deserialize + +%patch31 -p1 -b .caching-sockaddr +%patch32 -p1 -b .imfile-symlink +%patch33 -p1 -b .buffer-overflow +%patch34 -p1 -b .msg-loss-shutdown +%patch35 -p1 -b .kubernetes-metadata +%patch36 -p1 -b .omelasticsearch-cert +#%patch37 is applied right after doc setup +%patch38 -p1 -b .omelasticsearch-libfastjson +%patch39 -p1 -b .omelasticsearch-bulk-rejection +#%patch40 is applied right after doc setup +#%patch41 is applied right after doc setup +%patch42 -p1 -b .manpage +%patch43 -p1 -b .async-writer +%patch44 -p1 -b .null-realloc-chk autoreconf @@ -359,6 +413,7 @@ export LDFLAGS="-pie -Wl,-z,relro -Wl,-z,now" export HIREDIS_CFLAGS=-I/usr/include/hiredis export HIREDIS_LIBS=-L%{_libdir} %endif +sed -i 's/%{version}/%{version}-%{release}/g' configure.ac %configure \ --prefix=/usr \ --disable-static \ @@ -382,6 +437,7 @@ export HIREDIS_LIBS=-L%{_libdir} --enable-mmnormalize \ --enable-mmsnmptrapd \ --enable-mmutf8fix \ + --enable-mmkubernetes \ --enable-mysql \ %if %{want_hiredis} --enable-omhiredis \ @@ -398,6 +454,7 @@ export HIREDIS_LIBS=-L%{_libdir} --enable-omstdout \ --enable-omudpspoof \ --enable-omuxsock \ + --enable-omkafka \ --enable-pgsql \ --enable-pmaixforwardedfrom \ --enable-pmcisconames \ @@ -590,11 +647,156 @@ done %defattr(-,root,root) %{_libdir}/rsyslog/omudpspoof.so +%files kafka +%{_libdir}/rsyslog/omkafka.so + +%files mmkubernetes +%{_libdir}/rsyslog/mmkubernetes.so + %changelog -* Mon Apr 16 2018 Jiri Vymazal - 8.24.0-16.5 -RHEL 7.5.z ERRATUM -- fixed imjournal duplicating msgs under some conditions - resolves: rhbz#1545582 +* Tue Aug 07 2018 Jiri Vymazal - 8.24.0-34 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch with parent name bugfix + resolves: rhbz#1531295 + +* Tue Aug 07 2018 Jiri Vymazal - 8.24.0-33 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch with extended symlink watching + resolves: rhbz#1531295 +- updated mmkubernetes patch to accept dots in pod name + resolves: rhbz#1539193 + +* Fri Aug 03 2018 Jiri Vymazal - 8.24.0-32 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch with no log on EACCES + resolves: rhbz#1531295 +- removed now needless build-deps + +* Mon Jul 30 2018 Jiri Vymazal - 8.24.0-31 +RHEL 7.6 ERRATUM +- added new patch fixing ompipe dropping messages when pipe full + resolves: rhbz#1591819 +- updated mmkubernetes patch to accept non-kubernetes containers + resolves: rhbz#1539193 + resolves: rhbz#1609023 +- removed json-parsing patches as the bug is now fixed in liblognorm + +* Wed Jul 25 2018 Jiri Vymazal - 8.24.0-30 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch with next bugfix + resolves: rhbz#1531295 +- updated imjournal duplicates patch making slower code optional + and added corresponding doc patch + resolves: rhbz#1538372 + +* Mon Jul 23 2018 Jiri Vymazal - 8.24.0-29 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch with another bugfix + resolves: rhbz#1531295 + +* Fri Jul 20 2018 Jiri Vymazal - 8.24.0-28 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch fixing next round of regressions + resolves: rhbz#1531295 + resolves: rhbz#1602156 +- updated mmkubernetes patch with NULL ret-check + resolves: rhbz#1539193 + +* Tue Jul 17 2018 Jiri Vymazal - 8.24.0-27 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch fixing last update regressions + resolves: rhbz#1531295 +- added patch fixing deadlock in async logging + resolves: rhbz#1559408 +- added patch fixing NULL access in worktable create + resolves: rhbz#1600462 +- now putting release number into configure to have it present + in error messages + +* Mon Jul 09 2018 Jiri Vymazal - 8.24.0-26 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch according to early testing + resolves: rhbz#1531295 +- added patch fixing pid file name in manpage + resolves: rhbz#1597264 +- updated json-parsing patch with one more bugfix + resolves: rhbz#1565219 + +* Fri Jun 29 2018 Jiri Vymazal - 8.24.0-24 +RHEL 7.6 ERRATUM +- updated imfile rewrite patch with fixes from covscan + resolves: rhbz#1531295 +- updated mmkubernetes patch with fixes from covscan + resolves: rhbz#1539193 +- updated imjournal duplicates patch with fixes from covscan + resolves: rhbz#1538372 +- updated omelastic enhancement patch with fixes from covscan + resolves: rhbz#1565214 + +* Wed Jun 27 2018 Jiri Vymazal - 8.24.0-23 +RHEL 7.6 ERRATUM +- added backport of leading $ support to json-parsing patch + resolves: rhbz#1565219 +- The required info is already contained in rsyslog-doc package + so there is no patch for this one + resolves: rhbz#1553700 + +* Tue Jun 26 2018 Jiri Vymazal - 8.24.0-22 +RHEL 7.6 ERRATUM +- edited patch for top-level json parsing with bugfix + resolves: rhbz#1565219 +- renamed doc patches and added/updated new ones for mmkubernetes + omelasticsearch and json parsing +- renamed patch fixing buffer overflow in parser - memcpy() + resolves: rhbz#1582517 + +* Mon Jun 25 2018 Jiri Vymazal - 8.24.0-21 +RHEL 7.6 ERRATUM +- fixed imfile rewrite backport patch, added few more bugfixes + resolves: rhbz#1531295 +- added also doc patch for omelastic client certs + resolves: rhbz#1507145 +- cleaned and shortened patch for omelastic error handling + resolves: rhbz#1565214 +- enabled patch for json top-level parsing + resolves: rhbz#1565219 +- merged mmkubernetes patches into one and enabled the module + resolves: rhbz#1539193 + resolves: rhbz#1589924 + resolves: rhbz#1590582 + +* Sun Jun 24 2018 Noriko Hosoi - 8.24.0-21 +RHEL 7.6 ERRATUM +resolves: rhbz#1582517 - Buffer overflow in memcpy() in parser.c +resolves: rhbz#1539193 - RFE: Support for mm kubernetes plugin +resolves: rhbz#1589924 - RFE: Several fixes for mmkubernetes +resolves: rhbz#1590582 - mmkubernetes - use version=2 in rulebase files to avoid memory leak +resolves: rhbz#1507145 - RFE: omelasticsearch support client cert authentication +resolves: rhbz#1565214 - omelasticsearch needs better handling for bulk index rejections and other errors +Disables Patch32: rsyslog-8.24.0-rhbz1531295-imfile-rewrite-with-symlink.patch +Disables Patch34: rsyslog-8.24.0-rhbz1565219-parse-json-into-top-level-fields-in-mess.patch; It BuildRequires/Requires: libfastjson >= 0.99.4-3 + +* Fri Jun 01 2018 Jiri Vymazal - 8.24.0-20 +RHEL 7.6 ERRATUM +- added a patch backporting imfile module rewrite and + adding symlink support + resolves: rhbz#1531295 + +* Tue May 29 2018 Jiri Vymazal - 8.24.0-19 +RHEL 7.6 ERRATUM +- added new kafka sub-package with enabling of omkafka module + resolves: rhbz#1482819 + +* Thu May 17 2018 Radovan Sroka - 8.24.0-18 +- caching the whole sockaddr structure instead of sin_addr causing memory leak + resolves: rhbz#1512551 + +* Fri Apr 27 2018 Jiri Vymazal - 8.24.0-17 +RHEL 7.6 ERRATUM +- fixed imjournal duplicating messages on log rotation + resolves: rhbz#1538372 +- re-enabled 32-bit arches to not break dependent packages + resolves: rhbz#1571850 * Thu Nov 09 2017 Jiri Vymazal - 8.24.0-16 RHEL 7.5 ERRATUM