Templates
=========

Templates are a key feature of rsyslog. They allow to specify any format
a user might want. They are also used for dynamic file name generation.
Every output in rsyslog uses templates - this holds true for files, user
messages and so on. The database writer expects its template to be a
proper SQL statement - so this is highly customizable too. You might ask
how does all of this work when no templates at all are specified. Good
question ;). The answer is simple, though. Templates are compatible with the
stock syslogd formats which are hardcoded into rsyslogd. So if no template is
specified, we use one of those hardcoded templates. Search for
"template\_" in rsconf.c and you will find the hardcoded ones.

Templates are specified by template() statements. They can also be
specified via $template legacy statements.

**Note: key elements of templates are rsyslog properties.** See the
:doc:`rsyslog properties reference <properties>` for a list of which
are available.

Template processing
-------------------

Due to lack of standarization regarding logs formats, when a template is 
specified it's supposed to include HEADER, as defined in `RFC5424 <https://tools.ietf.org/html/rfc5424>`_

It's very important to have this in mind, and also how to understand how 
`rsyslog parsing <http://www.rsyslog.com/doc/syslog_parsing.html>`_ works

For example, if MSG field is set to "this:is a message" and no HOSTNAME, 
neither TAG are specified, outgoing parser will split the message as:

::

  TAG:this:
  MSG:is a message


The template() statement
------------------------

The template() statement is used to define templates. Note that it is a
**static** statement, that means all templates are defined when rsyslog
reads the config file. As such, templates are not affected by
if-statements or config nesting.

The basic structure of the template statement is as follows:

::

   template(parameters)

In addition to this simpler syntax, list templates (to be described
below) support an extended syntax:

::

   template(parameters) { list-descriptions }

Each template has a parameter **name**, which specifies the template
name, and a parameter **type**, which specifies the template type. The
name parameter must be unique, and behaviour is unpredictable if it is
not. The **type** parameter specifies different template types.
Different types simply enable different ways to specify the template
content. The template type **does not** affect what an (output) plugin
can do with it. So use the type that best fits your needs (from a config
writing point of view!). The following types are available:

-  list
-  subtree
-  string
-  plugin

The various types are described below.

list
~~~~

In this case, the template is generated by a list of constant and
variable statements. These follow the template spec in curly braces.
This type is also primarily meant for use with structure-aware outputs,
like ommongodb. However, it also works perfectly with text-based
outputs. We recommend to use this mode if more complex property
substitutions need to be done. In that case, the list-based template
syntax is much clearer than the simple string-based one.

The list template contains the template header (with **type="list"**)
and is followed by **constant** and **property** statements, given in
curly braces to signify the template statement they belong to. As the
name says, **constant** statements describe constant text and
**property** describes property access. There are many options to
**property**, described further below. Most of these options are used to
extract only partial property contents or to modify the text obtained
(like to change its case to upper or lower case, only).

To grasp the idea, an actual sample is:

::

    template(name="tpl1" type="list") {
        constant(value="Syslog MSG is: '")
        property(name="msg")
        constant(value="', ")
        property(name="timereported" dateFormat="rfc3339" caseConversion="lower")
        constant(value="\n")
        }

This sample is probably primarily targeted at the usual file-based
output.

constant statement
^^^^^^^^^^^^^^^^^^

This provides a way to specify constant text. The text is used
literally. It is primarily intended for text-based output, so that some
constant text can be included. For example, if a complex template is
built for file output, one usually needs to finish it by a newline,
which can be introduced by a constant statement. Here is an actual
sample of that use case from the rsylsog testbench:

::

    template(name="outfmt" type="list") {
        property(name="$!usr!msgnum")
        constant(value="\n")
    }

The following escape sequences are recognized inside the constant text:

-  \\\\ - single backslash
-  \\n - LF
-  \\ooo - (three octal digits) - represents character with this
   numerical value (e.g. \\101 equals "A"). Note that three octal digits
   must be given (in contrast to some languages, where between one and
   three are valid). While we support octal notation, we recommend to
   use hex notation as this is better known.
-  \\xhh - (where h is a hex digit) - represents character with this
   numerical value (e.g. \\x41 equals "A"). Note that two hexadecimal
   digits must be given (in contrast to some languages where one or two
   are valid).
-  ... some others ... list needs to be extended

Note: if an unsupported character follows a backslash, this is treated
as an error. Behaviour is unpredictable in this case.

To aid usage of the same template both for text-based outputs and
structured ones, constant text without an "outname" parameter will be
ignored when creating the name/value tree for structured outputs. So if
you want to supply some constant text e.g. to mongodb, you must include
an outname, as can be seen here:

::

    template(name="outfmt" type="list") {
        property(name="$!usr!msgnum")
        constant(value="\n" outname="IWantThisInMyDB")
    }

The "constant" statement supports the following parameters:

-  value - the constant value to use
-  outname - output field name (for structured outputs)

property statement
^^^^^^^^^^^^^^^^^^

This statement is used to include property text. It can access all
properties. Also, options permit to specify picking only part of a
property or modifying it. It supports the following parameters:

-  name - the name of the property to access
-  outname - output field name (for structured outputs)
-  dateformat - date format to use (only for date-related properties)
-  date.inUTC - date shall be shown in UTC (please note that this
   requires a bit more performance due to the necessary conversions)
   Available since 8.18.0.
-  caseconversion - permits to convert case of the text. Supported
   values are "lower" and "upper"
-  controlcharacters - specifies how to handle control characters.
   Supported values are "escape", which escapes them, "space", which
   replaces them by a single space, and "drop", which simply removes
   them from the string.
-  securepath - used for creating pathnames suitable for use in dynafile
   templates
-  format - specify format on a field basis. Supported values are:

   -  "`csv <property_replacer.html#csv>`_\ " for use when csv-data is
      generated
   -  "`json <property_replacer.html#json>`_\ " which formats proper
      json content (but without a field header)
   -  "`jsonf <property_replacer.html#jsonf>`_\ " which formats as a
      complete json field
   -  "`jsonr <property_replacer.html#jsonr>`_\ " which avoids double
      escaping the value but makes it safe for a json field
   -  "`jsonfr <property_replacer.html#jsonfr>`_\ " which is the
      combination of "jsonf" and "jsonr".

-  position.from - obtain substring starting from this position (1 is
   the first position)
-  position.to - obtain substring up to this position
-  position.relativeToEnd - the from and to position is relative to the
   end of the string instead of the usual start of string. (available
   since rsyslog v7.3.10)
-  fixedwidth - changes behaviour of position.to so that it pads the
   source string with spaces up to the value of position.to if the source
   string is shorter. "on" or "off" (default) (available since rsyslog
   v8.13.0)
- compressspace - compresses multiple spaces (US-ASCII SP character) inside the
   string to a single one. This compression happens at a very late
   stage in processing. Most importantly, it happens after substring
   extraction, so the **position.from** and **position.to** positions
   are **NOT** affected by this option. (available since v8.18.0).
-  field.number - obtain this field match
-  field.delimiter - decimal value of delimiter character for field
   extraction
-  regex.expression - expression to use
-  regex.type - either ERE or BRE
-  regex.nomatchmode - what to do if we have no match
-  regex.match - match to use
-  regex.submatch - submatch to use
-  droplastlf - drop a trailing LF, if it is present
-  mandatory - signifies a field as mandatory. If set to "on", this
   field will always be present in data passed to structured outputs,
   even if it is empty. If "off" (the default) empty fields will not be
   passed to structured outputs. This is especially useful for outputs
   that support dynamic schemas (like ommongodb).
-  spifno1stsp - expert options for RFC3164 template processing

subtree
~~~~~~~

Available since rsyslog 7.1.4

In this case, the template is generated based on a complete (CEE)
subtree. This type of template is most useful for outputs that know how
to process hierarchical structure, like ommongodb. With that type, the
parameter **subtree** must be specified, which tells which subtree to
use. For example template(name="tpl1" type="subtree" subtree="$!")
includes all CEE data, while template(name="tpl2" type="subtree"
subtree="$!usr!tpl2") includes only the subtree starting at $!usr!tpl2.
The core idea when using this type of template is that the actual data
is prefabricated via set and unset script statements, and the resulting
structure is then used inside the template. This method MUST be used if
a complete subtree needs to be placed *directly* into the object's root.
With all other template types, only subcontainers can be generated. Note
that subtree type can also be used with text-based outputs, like omfile.
HOWEVER, you do not have any capability to specify constant text, and as
such cannot include line breaks. As a consequence, using this template
type for text outputs is usually only useful for debugging or very
special cases (e.g. where the text is interpreted by a JSON parser later
on).

Use case
^^^^^^^^

A typical use case is to first create a custom subtree and then include
it into the template, like in this small example:

::

   set $!usr!tpl2!msg = $msg;
   set $!usr!tpl2!dataflow = field($msg, 58, 2);
   template(name="tpl2" type="subtree" subtree="$!usr!tpl2")

Here, we assume that $msg contains various fields, and the data from a
field is to be extracted and stored - together with the message - as
field content.

string
~~~~~~

This closely resembles the legacy template statement. It has a mandatory
parameter **string**, which holds the template string to be applied. A
template string is a mix of constant text and replacement variables (see
property replacer). These variables are taken from message or other
dynamic content when the final string to be passed to a plugin is
generated. String-based templates are a great way to specify textual
content, especially if no complex manipulation to properties is
necessary.

This is a sample for a string-based template:

::

   template(name="tpl3" type="string"
            string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
           )

The text between percent signs ('%') is interpreted by the rsyslog
:doc:`property replacer <property_replacer>`. In a nutshell,
it contains the property to use as well as options for formatting
and further processing. This is very similar to what the ``property``
object in list templates does (it actually is just a different language to
express most of the same things).

Everything outside of the percent signs is constant text. In the
above case, we have mostly spaces between the property values. At the
end of the string, an escape sequence is used.

Escape sequences permit to specify nonprintable characters. They work
very similar to escape sequences in C and many other languages. They
are initiated by the backslash characters and followed by one or more
characters that specify the actual character. For example \\7 is the
US-ASCII BEL character and \\n is a newline. The set is similar to
what C and perl support, but a bit more limited.

plugin
~~~~~~

In this case, the template is generated by a plugin (which is then
called a "strgen" or "string generator"). The format is fixed as it is
coded. While this is inflexible, it provides superior performance, and
is often used for that reason (not that "regular" templates are slow -
but in very demanding environments that "last bit" can make a
difference). Refer to the plugin's documentation for further details.
For this type, the parameter **plugin** must be specified and must
contain the name of the plugin as it identifies itself. Note that the
plugin must be loaded prior to being used inside a template.
Config example:

    ``template(name="tpl4" type="plugin" plugin="mystrgen")``

options
~~~~~~~

The <options> part is optional. It carries options influencing the
template as a whole and is a part of the template parameters. See details
below. Be sure NOT to mistake template options with property options -
the latter ones are processed by the property replacer and apply to a
SINGLE property, only (and not the whole template).
Template options are case-insensitive. Currently defined are:

**option.sql** - format the string suitable for a SQL statement in MySQL
format. This will replace single quotes ("'") and the backslash
character by their backslash-escaped counterpart ("\\'" and "\\\\")
inside each field. Please note that in MySQL configuration, the
``NO_BACKSLASH_ESCAPES`` mode must be turned off for this format to work
(this is the default).

**option.stdsql** - format the string suitable for a SQL statement that
is to be sent to a standards-compliant sql server. This will replace
single quotes ("'") by two single quotes ("''") inside each field. You
must use stdsql together with MySQL if in MySQL configuration the
``NO_BACKSLASH_ESCAPES`` is turned on.

**option.json** - format the string suitable for a json statement. This
will replace single quotes ("'") by two single quotes ("''") inside each
field.

**option.casesensitive** - treat property name references as case
sensitive. The default is "off", where all property name references are
first converted to lowercase during template definition. With this
option turned "on", property names are looked up as defined in the
template. Use this option if you have JSON (``$!*``), local (``!.*``),
or global (``$!\\*``) properties which container uppercase letters. The
normal Rsyslog properties are case-insensitive, so this option is not
needed for properly referencing those properties.

The use the options **option.sql**, **option.stdsql**, and
**option.json** are mutually exclusive. Using more than one at the same
time can cause unpredictable behaviour.

Either the **sql** or **stdsql** option **must** be specified when a
template is used for writing to a database, otherwise injection might
occur. Please note that due to the unfortunate fact that several vendors
have violated the sql standard and introduced their own escape methods,
it is impossible to have a single option doing all the work.  So you
yourself must make sure you are using the right format. **If you choose
the wrong one, you are still vulnerable to sql injection.**
Please note that the database writer *checks* that the sql option is
present in the template. If it is not present, the write database action
is disabled. This is to guard you against accidentally forgetting it and
then becoming vulnerable to SQL injection. The sql option can also be
useful with files - especially if you want to import them into a
database on another machine for performance reasons. However, do NOT use
it if you do not have a real need for it - among others, it takes some
toll on the processing time. Not much, but on a really busy system you
might notice it.

The default template for the write to database action has the sql option
set. As we currently support only MySQL and the sql option matches the
default MySQL configuration, this is a good choice. However, if you have
turned on ``NO_BACKSLASH_ESCAPES`` in your MySQL config, you need to
supply a template with the stdsql option. Otherwise you will become
vulnerable to SQL injection.

::

   template (name="TraditionalFormat" type="string"
   string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n"

Examples
~~~~~~~~

Standard Template for Writing to Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

    template(name="FileFormat" type="list") {
        property(name="timestamp" dateFormat="rfc3339")
        constant(value=" ")
        property(name="hostname")
        constant(value=" ")
        property(name="syslogtag")
        property(name="msg" spifno1stsp="on" )
        property(name="msg" droplastlf="on" )
        constant(value="\n")
        }

The equivalent string template looks like this:

::

    template(name="FileFormat" type="string"
             string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
            )

Note that the template string itself must be on a single line.

Standard Template for Forwarding to a Remote Host (RFC3164 mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

    template(name="ForwardFormat" type="list") {
        constant(value="<")
        property(name="pri")
        constant(value=">")
        property(name="timestamp" dateFormat="rfc3339")
        constant(value=" ")
        property(name="hostname")
        constant(value=" ")
        property(name="syslogtag" position.from="1" position.to="32")
        property(name="msg" spifno1stsp="on" )
	property(name="msg")
        }

The equivalent string template looks like this:

::

    template(name="forwardFormat" type="string"
             string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
            )

Note that the template string itself must be on a single line.

Standard Template for write to the MySQL database
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

    template(name="StdSQLformat" type="list" option.sql="on") {
            constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)")
            constant(value=" values ('")
            property(name="msg")
            constant(value="', ")
            property(name="syslogfacility")
            constant(value=", '")
            property(name="hostname")
            constant(value="', ")
            property(name="syslogpriority")
            constant(value=", '")
            property(name="timereported" dateFormat="mysql")
            constant(value="', '")
            property(name="timegenerated" dateFormat="mysql")
            constant(value="', ")
            property(name="iut")
            constant(value=", '")
            property(name="syslogtag")
            constant(value="')")
            }

The equivalent string template looks like this:

::

    template(name="stdSQLformat" type="string" option.sql="on"
             string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')"
            )

Note that the template string itself must be on a single line.

Creating Dynamic File Names for omfile
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Templates can be used to generate actions with dynamic file names.
For example, if you would like to split syslog messages from different hosts
to different files (one per host), you can define the following template:

::

   template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log")

Legacy example:

::

   $template DynFile,"/var/log/system-%HOSTNAME%.log"

This template can then be used when defining an action. It
will result in something like "/var/log/system-localhost.log"

legacy format
-------------

In pre v6-versions of rsyslog, you need to use the ``$template``
statement to configure templates. They provide the equivalent to string-
and plugin-based templates. The legacy syntax continuous to work in v7,
however we recommend to avoid legacy format for newly written config
files. Legacy and current config statements can coexist within the same
config file.

The general format is

::

  $template name,param[,options]

where "name" is the template name and
"param" is a single parameter that specifies template content. The
optional "options" part is used to set template options.

string
~~~~~~

The parameter is the same string that with the current-style format you
specify in the **string** parameter, for example:

::

  $template strtpl,"PRI: %pri%, MSG: %msg%\n"

Note that list templates are not available in legacy format, so you need
to use complex property replacer constructs to do complex things.

plugin
~~~~~~

This is equivalent to the "plugin"-type template directive. Here, the
parameter is the plugin name, with an equal sign prepended. An example
is:

::

   $template plugintpl,=myplugin

Reserved Template Names
-----------------------

Template names beginning with "RSYSLOG\_" are reserved for rsyslog use.
Do NOT use them if, otherwise you may receive a conflict in the future
(and quite unpredictable behaviour). There is a small set of pre-defined
templates that you can use without the need to define it:

-  **RSYSLOG\_TraditionalFileFormat** - the "old style" default log file
   format with low-precision timestamps
-  **RSYSLOG\_FileFormat** - a modern-style logfile format similar to
   TraditionalFileFormat, both with high-precision timestamps and
   timezone information
-  **RSYSLOG\_TraditionalForwardFormat** - the traditional forwarding format
   with low-precision timestamps. Most useful if you send messages to
   other syslogd's or rsyslogd below version 3.12.5.
-  **RSYSLOG\_SysklogdFileFormat** - sysklogd compatible log file format. If
   used with options: ``$SpaceLFOnReceive on``,
   ``$EscapeControlCharactersOnReceive off``,
   ``$DropTrailingLFOnReception off``, the log format will conform to sysklogd log format.
-  **RSYSLOG\_ForwardFormat** - a new high-precision forwarding format very
   similar to the traditional one, but with high-precision timestamps
   and timezone information. Recommended to be used when sending
   messages to rsyslog 3.12.5 or above.
-  **RSYSLOG\_SyslogProtocol23Format** - the format specified in IETF's
   internet-draft ietf-syslog-protocol-23, which is assumed to become
   the new syslog standard RFC. This format includes several
   improvements. The rsyslog message parser understands this format, so
   you can use it together with all relatively recent versions of
   rsyslog. Other syslogd's may get hopelessly confused if receiving
   that format, so check before you use it. Note that the format is
   unlikely to change when the final RFC comes out, but this may happen.
-  **RSYSLOG\_DebugFormat** - a special format used for troubleshooting
   property problems. This format is meant to be written to a log file.
   Do **not** use for production or remote forwarding.




Legacy String-based Template Samples
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This section provides some default templates in legacy format, as used
in rsyslog previous to version 6. Note that this format is still
supported, so there is no hard need to upgrade existing configurations.
However, it is strongly recommended that the legacy constructs are not
used when crafting new templates. Note that each $template statement is
on a **single** line, but probably broken across several lines for
display purposes by your browsers. Lines are separated by empty lines.
Keep in mind, that line breaks are important in legacy format.

::

  $template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
  $template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
  $template ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
  $template TraditionalForwardFormat,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"
  $template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL``

See Also
--------

-  `How to bind a
   template <http://www.rsyslog.com/how-to-bind-a-template/>`_
-  `Adding the BOM to a
   message <http://www.rsyslog.com/adding-the-bom-to-a-message/>`_
-  `How to separate log files by host name of the sending
   device <http://www.rsyslog.com/article60/>`_


This documentation is part of the `rsyslog <http://www.rsyslog.com/>`_ project.
Copyright © 2008-2014 by `Rainer Gerhards <http://www.gerhards.net/rainer>`_
and `Adiscon <http://www.adiscon.com/>`_. Released under the GNU GPL
version 2 or higher.
