*********
Templates
*********

Description
===========

Templates are a key feature of rsyslog. They allow to specify any format
a user might want. They are also used for dynamic file name generation.
Every output in rsyslog uses templates - this holds true for files, user
messages and so on. The database writer expects its template to be a
proper SQL statement - so this is highly customizable too. You might ask
how does all of this work when no templates at all are specified. Good
question ;). The answer is simple, though. Templates are compatible with the
stock syslogd formats which are hardcoded into rsyslogd. So if no template is
specified, we use one of those hardcoded templates. Search for
"template\_" in rsconf.c and you will find the hardcoded ones.

Templates are specified by template() statements. They can also be
specified via $template legacy statements.

.. note::

   Note: key elements of templates are rsyslog properties. See the
   :doc:`rsyslog properties reference <properties>` for a list of which
   are available.


TODO: :doc:`rsyslog properties reference <properties>` for a list of which


Template processing
===================

Due to lack of standardization regarding logs formats, when a template is
specified it's supposed to include HEADER, as defined in `RFC5424 <https://tools.ietf.org/html/rfc5424>`_

It's very important to have this in mind, and also how to understand how
`rsyslog parsing <http://www.rsyslog.com/doc/syslog_parsing.html>`_ works

For example, if MSG field is set to "this:is a message" and no HOSTNAME,
neither TAG are specified, outgoing parser will split the message as:

.. code-block:: none

   TAG:this:
   MSG:is a message


The template() statement
========================

The template() statement is used to define templates. Note that it is a
**static** statement, that means all templates are defined when rsyslog
reads the config file. As such, templates are not affected by
if-statements or config nesting.

The basic structure of the template statement is as follows:

.. code-block:: none

   template(parameters)

In addition to this simpler syntax, list templates (to be described
below) support an extended syntax:

.. code-block:: none

   template(parameters) { list-descriptions }

Each template has a parameter **name**, which specifies the template
name, and a parameter **type**, which specifies the template type. The
name parameter must be unique, and behaviour is unpredictable if it is
not. The **type** parameter specifies different template types.
Different types simply enable different ways to specify the template
content. The template type **does not** affect what an (output) plugin
can do with it. So use the type that best fits your needs (from a config
writing point of view!). The following types are available:

-  list
-  subtree
-  string
-  plugin

The various types are described below.

List
----

In this case, the template is generated by a list of constant and
variable statements. These follow the template spec in curly braces.
This type is also primarily meant for use with structure-aware outputs,
like ommongodb. However, it also works perfectly with text-based
outputs. We recommend to use this mode if more complex property
substitutions need to be done. In that case, the list-based template
syntax is much clearer than the simple string-based one.

The list template contains the template header (with **type="list"**)
and is followed by **constant** and **property** statements, given in
curly braces to signify the template statement they belong to. As the
name says, **constant** statements describe constant text and
**property** describes property access. There are many options to
**property**, described further below. Most of these options are used to
extract only partial property contents or to modify the text obtained
(like to change its case to upper or lower case, only).

To grasp the idea, an actual sample is:

.. code-block:: none

   template(name="tpl1" type="list") {
        constant(value="Syslog MSG is: '")
        property(name="msg")
        constant(value="', ")
        property(name="timereported" dateFormat="rfc3339" caseConversion="lower")
        constant(value="\n")
        }

This sample is probably primarily targeted at the usual file-based
output.

Constant statement
^^^^^^^^^^^^^^^^^^

This provides a way to specify constant text. The text is used
literally. It is primarily intended for text-based output, so that some
constant text can be included. For example, if a complex template is
built for file output, one usually needs to finish it by a newline,
which can be introduced by a constant statement. Here is an actual
sample of that use case from the rsylsog testbench:

::

    template(name="outfmt" type="list") {
        property(name="$!usr!msgnum")
        constant(value="\n")
    }

The following escape sequences are recognized inside the constant text:

-  \\\\ - single backslash
-  \\n - LF
-  \\ooo - (three octal digits) - represents character with this
   numerical value (e.g. \\101 equals "A"). Note that three octal digits
   must be given (in contrast to some languages, where between one and
   three are valid). While we support octal notation, we recommend to
   use hex notation as this is better known.
-  \\xhh - (where h is a hex digit) - represents character with this
   numerical value (e.g. \\x41 equals "A"). Note that two hexadecimal
   digits must be given (in contrast to some languages where one or two
   are valid).
-  ... some others ... list needs to be extended

Note: if an unsupported character follows a backslash, this is treated
as an error. Behaviour is unpredictable in this case.

To aid usage of the same template both for text-based outputs and
structured ones, constant text without an "outname" parameter will be
ignored when creating the name/value tree for structured outputs. So if
you want to supply some constant text e.g. to mongodb, you must include
an outname, as can be seen here:

.. code-block:: none

    template(name="outfmt" type="list") {
        property(name="$!usr!msgnum")
        constant(value="\n" outname="IWantThisInMyDB")
    }

To generate a constant json field, the `format` parameter can be used, as
in this example

.. code-block:: none

   template(name="outfmt" type="list" option.jsonf="on") {
             property(outname="message" name="msg" format="jsonf")
             constant(outname="@version" value="1" format="jsonf")
   }

The constant in this case will generate `"@version":"1"`. Note that to do
this, both the `value` and `format` parameter must be given.

The "constant" statement supports the following parameters:

- value - the constant value to use
- outname - output field name (for structured outputs)
- format - can be either empty or `jsonf`


Property statement
^^^^^^^^^^^^^^^^^^

This statement is used to include property text. It can access all
properties. Also, options permit to specify picking only part of a
property or modifying it. It supports the following parameters:

-  **name** - the name of the property to access

-  **outname** - output field name (for structured outputs)

-  **dateformat** - Date format to use (only for date-related properties).
   `Here <property_replacer.html#property-options>`_ you can find a list of all
   property options.

   To create a customised format you can use multiple property options
   together. The following example would result in **YYYY-MM-DD**.

.. code-block:: none

    property(name="timereported" dateformat="date-year")
    constant(value="-")
    property(name="timereported" dateformat="date-month")
    constant(value="-")
    property(name="timereported" dateformat="date-day")

-  **date.inUTC** - date shall be shown in UTC (please note that this
   requires a bit more performance due to the necessary conversions)
   Available since 8.18.0.

-  **caseconversion** - permits to convert case of the text. Supported
   values are "lower" and "upper"

-  **controlcharacters** - specifies how to handle control characters.
   Supported values are "escape", which escapes them, "space", which
   replaces them by a single space, and "drop", which simply removes
   them from the string.

-  **securepath** - used for creating pathnames suitable for use in dynafile
   templates

-  **format** - specify format on a field basis. Supported values are:

   -  "`csv <property_replacer.html#csv>`_\ " for use when csv-data is
      generated
   -  "`json <property_replacer.html#json>`_\ " which formats proper
      json content (but without a field header)
   -  "`jsonf <property_replacer.html#jsonf>`_\ " which formats as a
      complete json field
   -  "`jsonr <property_replacer.html#jsonr>`_\ " which avoids double
      escaping the value but makes it safe for a json field
   -  "`jsonfr <property_replacer.html#jsonfr>`_\ " which is the
      combination of "jsonf" and "jsonr".

-  **position.from** - obtain substring starting from this position (1 is
   the first position)

-  **position.to** - obtain substring up to this position

-  **position.relativeToEnd** - the from and to position is relative to the
   end of the string instead of the usual start of string. (available
   since rsyslog v7.3.10)

-  **fixedwidth** - changes behaviour of position.to so that it pads the
   source string with spaces up to the value of position.to if the source
   string is shorter. "on" or "off" (default) (available since rsyslog
   v8.13.0)

-  **compressspace** - compresses multiple spaces (US-ASCII SP character)
   inside the string to a single one. This compression happens at a very
   late stage in processing. Most importantly, it happens after substring
   extraction, so the **position.from** and **position.to** positions
   are **NOT** affected by this option. (available since v8.18.0).

-  **field.number** - obtain this field match

-  **field.delimiter** - decimal value of delimiter character for field
   extraction

-  **regex.expression** - expression to use

-  **regex.type** - either ERE or BRE

-  **regex.nomatchmode** - what to do if we have no match

-  **regex.match** - match to use

-  **regex.submatch** - submatch to use

-  **droplastlf** - drop a trailing LF, if it is present

-  **mandatory** - signifies a field as mandatory. If set to "on", this
   field will always be present in data passed to structured outputs,
   even if it is empty. If "off" (the default) empty fields will not be
   passed to structured outputs. This is especially useful for outputs
   that support dynamic schemas (like ommongodb).

-  **spifno1stsp** - expert options for RFC3164 template processing


Subtree
-------

Available since rsyslog 7.1.4

In this case, the template is generated based on a complete (CEE)
subtree. This type of template is most useful for outputs that know how
to process hierarchical structure, like ommongodb. With that type, the
parameter **subtree** must be specified, which tells which subtree to
use. For example template(name="tpl1" type="subtree" subtree="$!")
includes all CEE data, while template(name="tpl2" type="subtree"
subtree="$!usr!tpl2") includes only the subtree starting at $!usr!tpl2.
The core idea when using this type of template is that the actual data
is prefabricated via set and unset script statements, and the resulting
structure is then used inside the template. This method MUST be used if
a complete subtree needs to be placed *directly* into the object's root.
With all other template types, only subcontainers can be generated. Note
that subtree type can also be used with text-based outputs, like omfile.
HOWEVER, you do not have any capability to specify constant text, and as
such cannot include line breaks. As a consequence, using this template
type for text outputs is usually only useful for debugging or very
special cases (e.g. where the text is interpreted by a JSON parser later
on).


Use case
^^^^^^^^

A typical use case is to first create a custom subtree and then include
it into the template, like in this small example:

.. code-block:: none

   set $!usr!tpl2!msg = $msg;
   set $!usr!tpl2!dataflow = field($msg, 58, 2);
   template(name="tpl2" type="subtree" subtree="$!usr!tpl2")


Here, we assume that $msg contains various fields, and the data from a
field is to be extracted and stored - together with the message - as
field content.


String
------

This closely resembles the legacy template statement. It has a mandatory
parameter **string**, which holds the template string to be applied. A
template string is a mix of constant text and replacement variables (see
property replacer). These variables are taken from message or other
dynamic content when the final string to be passed to a plugin is
generated. String-based templates are a great way to specify textual
content, especially if no complex manipulation to properties is
necessary.

This is a sample for a string-based template:

.. code-block:: none

   template(name="tpl3" type="string"
            string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
           )


The text between percent signs ('%') is interpreted by the rsyslog
:doc:`property replacer <property_replacer>`. In a nutshell,
it contains the property to use as well as options for formatting
and further processing. This is very similar to what the ``property``
object in list templates does (it actually is just a different language to
express most of the same things).

Everything outside of the percent signs is constant text. In the
above case, we have mostly spaces between the property values. At the
end of the string, an escape sequence is used.

Escape sequences permit to specify nonprintable characters. They work
very similar to escape sequences in C and many other languages. They
are initiated by the backslash characters and followed by one or more
characters that specify the actual character. For example \\7 is the
US-ASCII BEL character and \\n is a newline. The set is similar to
what C and perl support, but a bit more limited.


Plugin
------

In this case, the template is generated by a plugin (which is then
called a "strgen" or "string generator"). The format is fixed as it is
coded. While this is inflexible, it provides superior performance, and
is often used for that reason (not that "regular" templates are slow -
but in very demanding environments that "last bit" can make a
difference). Refer to the plugin's documentation for further details.
For this type, the parameter **plugin** must be specified and must
contain the name of the plugin as it identifies itself. Note that the
plugin must be loaded prior to being used inside a template.
Config example:

   ``template(name="tpl4" type="plugin" plugin="mystrgen")``


Options
^^^^^^^

The <options> part is optional. It carries options influencing the
template as a whole and is a part of the template parameters. See details
below. Be sure NOT to mistake template options with property options -
the latter ones are processed by the property replacer and apply to a
SINGLE property, only (and not the whole template).
Template options are case-insensitive. Currently defined are:

**option.sql** - format the string suitable for a SQL statement in MySQL
format. This will replace single quotes ("'") and the backslash
character by their backslash-escaped counterpart ("\\'" and "\\\\")
inside each field. Please note that in MySQL configuration, the
``NO_BACKSLASH_ESCAPES`` mode must be turned off for this format to work
(this is the default).

**option.stdsql** - format the string suitable for a SQL statement that
is to be sent to a standards-compliant sql server. This will replace
single quotes ("'") by two single quotes ("''") inside each field. You
must use stdsql together with MySQL if in MySQL configuration the
``NO_BACKSLASH_ESCAPES`` is turned on.

**option.json** - format the string suitable for a json statement. This
will replace single quotes ("'") by two single quotes ("''") inside each
field.

**option.jsonf** - format the string as JSON object. This means a leading
and trailing curly brace "{" will be added as well as a comma between all
non-terminal properties and constants.

**option.casesensitive** - treat property name references as case
sensitive. The default is "off", where all property name references are
first converted to lowercase during template definition. With this
option turned "on", property names are looked up as defined in the
template. Use this option if you have JSON (``$!*``), local (``!.*``),
or global (``$!\\*``) properties which contain uppercase letters. The
normal Rsyslog properties are case-insensitive, so this option is not
needed for properly referencing those properties.

Use of the options **option.sql**, **option.stdsql**, and
**option.json** are mutually exclusive. Using more than one at the same
time can cause unpredictable behaviour.

Either the **sql** or **stdsql** option **must** be specified when a
template is used for writing to a database, otherwise injection might
occur. Please note that due to the unfortunate fact that several vendors
have violated the sql standard and introduced their own escape methods,
it is impossible to have a single option doing all the work.  So you
yourself must make sure you are using the right format. **If you choose
the wrong one, you are still vulnerable to sql injection.**
Please note that the database writer *checks* that the sql option is
present in the template. If it is not present, the write database action
is disabled. This is to guard you against accidentally forgetting it and
then becoming vulnerable to SQL injection. The sql option can also be
useful with files - especially if you want to import them into a
database on another machine for performance reasons. However, do NOT use
it if you do not have a real need for it - among others, it takes some
toll on the processing time. Not much, but on a really busy system you
might notice it.

The default template for the write to database action has the sql option
set. As we currently support only MySQL and the sql option matches the
default MySQL configuration, this is a good choice. However, if you have
turned on ``NO_BACKSLASH_ESCAPES`` in your MySQL config, you need to
supply a template with the stdsql option. Otherwise you will become
vulnerable to SQL injection.

.. code-block:: none

   template (name="TraditionalFormat" type="string"
   string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n"


Examples
========

Standard Template for Writing to Files
--------------------------------------

.. code-block:: none

    template(name="FileFormat" type="list") {
        property(name="timestamp" dateFormat="rfc3339")
        constant(value=" ")
        property(name="hostname")
        constant(value=" ")
        property(name="syslogtag")
        property(name="msg" spifno1stsp="on" )
        property(name="msg" droplastlf="on" )
        constant(value="\n")
        }

The equivalent string template looks like this:

.. code-block:: none

    template(name="FileFormat" type="string"
             string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
            )


.. note::

   The template string itself must be on a single line.


Standard Template for Forwarding to a Remote Host (RFC3164 mode)
----------------------------------------------------------------

.. code-block:: none

    template(name="ForwardFormat" type="list") {
        constant(value="<")
        property(name="pri")
        constant(value=">")
        property(name="timestamp" dateFormat="rfc3339")
        constant(value=" ")
        property(name="hostname")
        constant(value=" ")
        property(name="syslogtag" position.from="1" position.to="32")
        property(name="msg" spifno1stsp="on" )
	property(name="msg")
        }

The equivalent string template looks like this:

.. code-block:: none

    template(name="forwardFormat" type="string"
             string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
            )

.. note::

   The template string itself must be on a single line.


Standard Template for writing to the MySQL database
---------------------------------------------------

.. code-block:: none

    template(name="StdSQLformat" type="list" option.sql="on") {
            constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)")
            constant(value=" values ('")
            property(name="msg")
            constant(value="', ")
            property(name="syslogfacility")
            constant(value=", '")
            property(name="hostname")
            constant(value="', ")
            property(name="syslogpriority")
            constant(value=", '")
            property(name="timereported" dateFormat="mysql")
            constant(value="', '")
            property(name="timegenerated" dateFormat="mysql")
            constant(value="', ")
            property(name="iut")
            constant(value=", '")
            property(name="syslogtag")
            constant(value="')")
            }

The equivalent string template looks like this:

.. code-block:: none

    template(name="stdSQLformat" type="string" option.sql="on"
             string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')"
            )

.. note::

   The template string itself must be on a single line.


Generating JSON
---------------

This is especially useful for RESTful APIs, like for example ElasticSearch provides.

This template

.. code-block:: none

    template(name="outfmt" type="list" option.jsonf="on") {
             property(outname="@timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
             property(outname="host" name="hostname" format="jsonf")
             property(outname="severity" name="syslogseverity-text" caseConversion="upper" format="jsonf")
             property(outname="facility" name="syslogfacility-text" format="jsonf")
             property(outname="syslog-tag" name="syslogtag" format="jsonf")
             property(outname="source" name="app-name" format="jsonf")
             property(outname="message" name="msg" format="jsonf")

     }

Generates output similar to this

.. code-block:: none

    {"@timestamp":"2018-03-01T01:00:00+00:00", "host":"172.20.245.8", "severity":"DEBUG", "facility":"local4", "syslog-tag":"tag", "source":"tag", "message":" msgnum:00000000:"}

Pretty-printed this looks like

.. code-block:: none

    {
      "@timestamp": "2018-03-01T01:00:00+00:00",
      "host": "172.20.245.8",
      "severity": "DEBUG",
      "facility": "local4",
      "syslog-tag": "tag",
      "source": "tag",
      "message": " msgnum:00000000:"
    }


.. note::

   The output is **not** pretty-printed as this is just waste of resources when
   used in RESTful APIs.


Creating Dynamic File Names for omfile
--------------------------------------

Templates can be used to generate actions with dynamic file names.
For example, if you would like to split syslog messages from different hosts
to different files (one per host), you can define the following template:

.. code-block:: none

   template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")


Reserved Template Names
-----------------------

Template names beginning with "RSYSLOG\_" are reserved for rsyslog use.
Do NOT use them if, otherwise you may receive a conflict in the future
(and quite unpredictable behaviour). There is a small set of pre-defined
templates that you can use without the need to define it:

-  **RSYSLOG_TraditionalFileFormat** - the "old style" default log file
   format with low-precision timestamps
-  **RSYSLOG_FileFormat** - a modern-style logfile format similar to
   TraditionalFileFormat, both with high-precision timestamps and
   timezone information
-  **RSYSLOG_TraditionalForwardFormat** - the traditional forwarding format
   with low-precision timestamps. Most useful if you send messages to
   other syslogd's or rsyslogd below version 3.12.5.
-  **RSYSLOG_SysklogdFileFormat** - sysklogd compatible log file format. If
   used with options: ``$SpaceLFOnReceive on``,
   ``$EscapeControlCharactersOnReceive off``,
   ``$DropTrailingLFOnReception off``, the log format will conform to sysklogd log format.
-  **RSYSLOG_ForwardFormat** - a new high-precision forwarding format very
   similar to the traditional one, but with high-precision timestamps
   and timezone information. Recommended to be used when sending
   messages to rsyslog 3.12.5 or above.
-  **RSYSLOG_SyslogProtocol23Format** - the format specified in IETF's
   internet-draft ietf-syslog-protocol-23, which is very close to the actual
   syslog standard `RFC5424 <https://tools.ietf.org/html/rfc5424>`_ (we couldn't
   update this template as things were in production for quite some time when
   RFC5424 was finally approved). This format includes several improvements.
   You may use this format with all relatively recent versions of rsyslog or syslogd.
-  **RSYSLOG_DebugFormat** - a special format used for troubleshooting
   property problems. This format is meant to be written to a log file.
   Do **not** use for production or remote forwarding.


See Also
--------

-  `How to bind a
   template <http://www.rsyslog.com/how-to-bind-a-template/>`_
-  `Adding the BOM to a
   message <http://www.rsyslog.com/adding-the-bom-to-a-message/>`_
-  `How to separate log files by host name of the sending
   device <http://www.rsyslog.com/article60/>`_

