anitazha / rpms / systemd

Forked from rpms/systemd 3 years ago
Clone

Blame SOURCES/0569-journald-make-maximum-size-of-stream-log-lines-confi.patch

dd65c9
From 32244f8d21a3e06f6519c47234289da696f6b151 Mon Sep 17 00:00:00 2001
dd65c9
From: Lennart Poettering <lennart@poettering.net>
dd65c9
Date: Sun, 8 Oct 2017 09:05:59 +0200
dd65c9
Subject: [PATCH] journald: make maximum size of stream log lines configurable
dd65c9
 and bump it to 48K (#6838)
dd65c9
dd65c9
This adds a new setting LineMax= to journald.conf, and sets it by
dd65c9
default to 48K. When we convert stream-based stdout/stderr logging into
dd65c9
record-based log entries, read up to the specified amount of bytes
dd65c9
before forcing a line-break.
dd65c9
dd65c9
This also makes three related changes:
dd65c9
dd65c9
- When a NUL byte is read we'll not recognize this as alternative line
dd65c9
  break, instead of silently dropping everything after it. (see #4863)
dd65c9
dd65c9
- The reason for a line-break is now encoded in the log record, if it
dd65c9
  wasn't a plain newline. Specifically, we distuingish "nul",
dd65c9
  "line-max" and "eof", for line breaks due to NUL byte, due to the
dd65c9
  maximum line length as configured with LineMax= or due to end of
dd65c9
  stream. This data is stored in the new implicit _LINE_BREAK= field.
dd65c9
  It's not synthesized for plain \n line breaks.
dd65c9
dd65c9
- A randomized 128bit ID is assigned to each log stream.
dd65c9
dd65c9
With these three changes in place it's (mostly) possible to reconstruct
dd65c9
the original byte streams from log data, as (most) of the context of
dd65c9
the conversion from the byte stream to log records is saved now. (So,
dd65c9
the only bits we still drop are empty lines. Which might be something to
dd65c9
look into in a future change, and which is outside of the scope of this
dd65c9
work)
dd65c9
dd65c9
Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=86465
dd65c9
See: #4863
dd65c9
Replaces: #4875
dd65c9
dd65c9
(cherry picked from commit ec20fe5ffb8a00469bab209fff6c069bb93c6db2)
dd65c9
dd65c9
Resolves: #1442262
dd65c9
dd65c9
[msekleta: I had to manually rewrite upstream commit, because git
dd65c9
did very poor job merging old and new code and identified a lot of merge
dd65c9
conflicts in a code that was totally unrelated.]
dd65c9
---
23b3cf
 man/journald.conf.xml            |  18 +++++
23b3cf
 man/systemd.journal-fields.xml   |  22 ++++++
dd65c9
 src/journal/journald-gperf.gperf |   1 +
23b3cf
 src/journal/journald-server.c    |  68 ++++++++++++++++++
dd65c9
 src/journal/journald-server.h    |   3 +
23b3cf
 src/journal/journald-stream.c    | 116 +++++++++++++++++++++++++------
dd65c9
 src/journal/journald.conf        |   1 +
dd65c9
 7 files changed, 209 insertions(+), 20 deletions(-)
dd65c9
dd65c9
diff --git a/man/journald.conf.xml b/man/journald.conf.xml
dd65c9
index 46a498b67..e2d6a1225 100644
dd65c9
--- a/man/journald.conf.xml
dd65c9
+++ b/man/journald.conf.xml
dd65c9
@@ -354,6 +354,24 @@
dd65c9
         <filename>/dev/console</filename>.</para></listitem>
dd65c9
       </varlistentry>
dd65c9
 
dd65c9
+      <varlistentry>
dd65c9
+        <term><varname>LineMax=</varname></term>
dd65c9
+
dd65c9
+        <listitem><para>The maximum line length to permit when converting stream logs into record logs. When a systemd
dd65c9
+        unit's standard output/error are connected to the journal via a stream socket, the data read is split into
dd65c9
+        individual log records at newline (<literal>\n</literal>, ASCII 10) and NUL characters. If no such delimiter is
dd65c9
+        read for the specified number of bytes a hard log record boundary is artifically inserted, breaking up overly
dd65c9
+        long lines into multiple log records. Selecting overly large values increases the possible memory usage of the
dd65c9
+        Journal daemon for each stream client, as in the worst case the journal daemon needs to buffer the specified
dd65c9
+        number of bytes in memory before it can flush a new log record to disk. Also note that permitting overly large
dd65c9
+        line maximum line lengths affects compatibility with traditional log protocols as log records might not fit
dd65c9
+        anymore into a single <constant>AF_UNIX</constant> or <constant>AF_INET</constant> datagram. Takes a size in
dd65c9
+        bytes. If the value is suffixed with K, M, G or T, the specified size is parsed as Kilobytes, Megabytes,
dd65c9
+        Gigabytes, or Terabytes (with the base 1024), respectively. Defaults to 48K, which is relatively large but
dd65c9
+        still small enough so that log records likely fit into network datagrams along with extra room for
dd65c9
+        metadata. Note that values below 79 are not accepted and will be bumped to 79.</para></listitem>
dd65c9
+      </varlistentry>
dd65c9
+
dd65c9
     </variablelist>
dd65c9
 
dd65c9
   </refsect1>
dd65c9
diff --git a/man/systemd.journal-fields.xml b/man/systemd.journal-fields.xml
dd65c9
index 7d6c5c715..a53f8def2 100644
dd65c9
--- a/man/systemd.journal-fields.xml
dd65c9
+++ b/man/systemd.journal-fields.xml
dd65c9
@@ -311,6 +311,28 @@
dd65c9
           </variablelist>
dd65c9
         </listitem>
dd65c9
       </varlistentry>
dd65c9
+      <varlistentry>
dd65c9
+        <term><varname>_STREAM_ID=</varname></term>
dd65c9
+        <listitem>
dd65c9
+          <para>Only applies to <literal>_TRANSPORT=stream</literal> records: specifies a randomized 128bit ID assigned
dd65c9
+          to the stream connection when it was first created. This ID is useful to reconstruct individual log streams
dd65c9
+          from the log records: all log records carrying the same stream ID originate from the same stream.</para>
dd65c9
+        </listitem>
dd65c9
+      </varlistentry>
dd65c9
+      <varlistentry>
dd65c9
+        <term><varname>_LINE_BREAK=</varname></term>
dd65c9
+        <listitem>
dd65c9
+          <para>Only applies to <literal>_TRANSPORT=stream</literal> records: indicates that the log message in the
dd65c9
+          standard output/error stream was not terminated with a normal newline character (<literal>\n</literal>,
dd65c9
+          i.e. ASCII 10). Specifically, when set this field is one of <option>nul</option> (in case the line was
dd65c9
+          terminated by a NUL byte), <option>line-max</option> (in case the maximum log line length was reached, as
dd65c9
+          configured with <varname>LineMax=</varname> in
dd65c9
+          <citerefentry><refentrytitle>journald.conf</refentrytitle><manvolnum>5</manvolnum></citerefentry>) or
dd65c9
+          <option>eof</option> (if this was the last log record of a stream and the stream ended without a final
dd65c9
+          newline character). Note that this record is not generated when a normal newline character was used for
dd65c9
+          marking the log line end.</para>
dd65c9
+        </listitem>
dd65c9
+      </varlistentry>
dd65c9
     </variablelist>
dd65c9
   </refsect1>
dd65c9
 
dd65c9
diff --git a/src/journal/journald-gperf.gperf b/src/journal/journald-gperf.gperf
dd65c9
index 74554c1c3..73327c10e 100644
dd65c9
--- a/src/journal/journald-gperf.gperf
dd65c9
+++ b/src/journal/journald-gperf.gperf
dd65c9
@@ -40,3 +40,4 @@ Journal.MaxLevelKMsg,       config_parse_log_level,  0, offsetof(Server, max_lev
dd65c9
 Journal.MaxLevelConsole,    config_parse_log_level,  0, offsetof(Server, max_level_console)
dd65c9
 Journal.MaxLevelWall,       config_parse_log_level,  0, offsetof(Server, max_level_wall)
dd65c9
 Journal.SplitMode,          config_parse_split_mode, 0, offsetof(Server, split_mode)
dd65c9
+Journal.LineMax,            config_parse_line_max,   0, offsetof(Server, line_max)
dd65c9
\ No newline at end of file
dd65c9
diff --git a/src/journal/journald-server.c b/src/journal/journald-server.c
dd65c9
index f6f8c30eb..daeecd519 100644
dd65c9
--- a/src/journal/journald-server.c
dd65c9
+++ b/src/journal/journald-server.c
dd65c9
@@ -67,6 +67,10 @@
dd65c9
 
dd65c9
 #define RECHECK_AVAILABLE_SPACE_USEC (30*USEC_PER_SEC)
dd65c9
 
dd65c9
+/* Pick a good default that is likely to fit into AF_UNIX and AF_INET SOCK_DGRAM datagrams, and even leaves some room
dd65c9
++ * for a bit of additional metadata. */
dd65c9
+#define DEFAULT_LINE_MAX (48*1024)
dd65c9
+
dd65c9
 static const char* const storage_table[_STORAGE_MAX] = {
dd65c9
         [STORAGE_AUTO] = "auto",
dd65c9
         [STORAGE_VOLATILE] = "volatile",
dd65c9
@@ -83,9 +87,71 @@ static const char* const split_mode_table[_SPLIT_MAX] = {
dd65c9
         [SPLIT_NONE] = "none",
dd65c9
 };
dd65c9
 
dd65c9
+
dd65c9
 DEFINE_STRING_TABLE_LOOKUP(split_mode, SplitMode);
dd65c9
 DEFINE_CONFIG_PARSE_ENUM(config_parse_split_mode, split_mode, SplitMode, "Failed to parse split mode setting");
dd65c9
 
dd65c9
+int config_parse_line_max(
dd65c9
+                const char* unit,
dd65c9
+                const char *filename,
dd65c9
+                unsigned line,
dd65c9
+                const char *section,
dd65c9
+                unsigned section_line,
dd65c9
+                const char *lvalue,
dd65c9
+                int ltype,
dd65c9
+                const char *rvalue,
dd65c9
+                void *data,
dd65c9
+                void *userdata) {
dd65c9
+
dd65c9
+        size_t *sz = data;
dd65c9
+        int r;
dd65c9
+
dd65c9
+        assert(filename);
dd65c9
+        assert(lvalue);
dd65c9
+        assert(rvalue);
dd65c9
+        assert(data);
dd65c9
+
dd65c9
+        if (isempty(rvalue))
dd65c9
+                /* Empty assignment means default */
dd65c9
+                *sz = DEFAULT_LINE_MAX;
dd65c9
+        else {
dd65c9
+                uint64_t v;
dd65c9
+                off_t u;
dd65c9
+
dd65c9
+                r = parse_size(rvalue, 1024, &u);
dd65c9
+                if (r < 0) {
dd65c9
+                        log_syntax(unit, LOG_ERR, filename, line, r, "Failed to parse LineMax= value, ignoring: %s", rvalue);
dd65c9
+                        return 0;
dd65c9
+                }
dd65c9
+
dd65c9
+                /* Backport note */
dd65c9
+                /* Upstream ditched use of off_t however our parse_size implementation still takes off_t*
dd65c9
+                 * as an argument. Since we compile with -Werror, we have two choices, either disable sign-compare
dd65c9
+                 * warning or do this casting so we don't have to change rest of the code. I think it is
dd65c9
+                 * better to do cast here instead of rewriting the code so it deals with off_t instead of
dd65c9
+                 * uint64_t. Doing conversion off_t -> uint64_t is something that we should think about. */
dd65c9
+                v = (uint64_t) u;
dd65c9
+
dd65c9
+                if (v < 79) {
dd65c9
+                        /* Why specify 79 here as minimum line length? Simply, because the most common traditional
dd65c9
+                         * terminal size is 80ch, and it might make sense to break one character before the natural
dd65c9
+                         * line break would occur on that. */
dd65c9
+                        log_syntax(unit, LOG_WARNING, filename, line, 0, "LineMax= too small, clamping to 79: %s", rvalue);
dd65c9
+                        *sz = 79;
dd65c9
+                } else if (v > (uint64_t) (SSIZE_MAX-1)) {
dd65c9
+                        /* So, why specify SSIZE_MAX-1 here? Because that's one below the largest size value read()
dd65c9
+                         * can return, and we need one extra byte for the trailing NUL byte. Of course IRL such large
dd65c9
+                         * memory allocations will fail anyway, hence this limit is mostly theoretical anyway, as we'll
dd65c9
+                         * fail much earlier anyway. */
dd65c9
+                        log_syntax(unit, LOG_WARNING, filename, line, 0, "LineMax= too large, clamping to %" PRIu64 ": %s", (uint64_t) (SSIZE_MAX-1), rvalue);
dd65c9
+                        *sz = SSIZE_MAX-1;
dd65c9
+                } else
dd65c9
+                        *sz = (size_t) v;
dd65c9
+        }
dd65c9
+
dd65c9
+        return 0;
dd65c9
+}
dd65c9
+
dd65c9
 static uint64_t available_space(Server *s, bool verbose) {
dd65c9
         char ids[33];
dd65c9
         _cleanup_free_ char *p = NULL;
dd65c9
@@ -1518,6 +1584,8 @@ int server_init(Server *s) {
dd65c9
         s->max_level_console = LOG_INFO;
dd65c9
         s->max_level_wall = LOG_EMERG;
dd65c9
 
dd65c9
+        s->line_max = DEFAULT_LINE_MAX;
dd65c9
+
dd65c9
         memset(&s->system_metrics, 0xFF, sizeof(s->system_metrics));
dd65c9
         memset(&s->runtime_metrics, 0xFF, sizeof(s->runtime_metrics));
dd65c9
 
dd65c9
diff --git a/src/journal/journald-server.h b/src/journal/journald-server.h
dd65c9
index 7a456c2d5..b29410778 100644
dd65c9
--- a/src/journal/journald-server.h
dd65c9
+++ b/src/journal/journald-server.h
dd65c9
@@ -143,6 +143,8 @@ typedef struct Server {
dd65c9
 
dd65c9
         /* Cached cgroup root, so that we don't have to query that all the time */
dd65c9
         char *cgroup_root;
dd65c9
+
dd65c9
+        size_t line_max;
dd65c9
 } Server;
dd65c9
 
dd65c9
 #define N_IOVEC_META_FIELDS 20
dd65c9
@@ -157,6 +159,7 @@ void server_driver_message(Server *s, sd_id128_t message_id, const char *format,
dd65c9
 const struct ConfigPerfItem* journald_gperf_lookup(const char *key, unsigned length);
dd65c9
 
dd65c9
 int config_parse_storage(const char *unit, const char *filename, unsigned line, const char *section, unsigned section_line, const char *lvalue, int ltype, const char *rvalue, void *data, void *userdata);
dd65c9
+int config_parse_line_max(const char *unit, const char *filename, unsigned line, const char *section, unsigned section_line, const char *lvalue, int ltype, const char *rvalue, void *data, void *userdata);
dd65c9
 
dd65c9
 const char *storage_to_string(Storage s) _const_;
dd65c9
 Storage storage_from_string(const char *s) _pure_;
dd65c9
diff --git a/src/journal/journald-stream.c b/src/journal/journald-stream.c
dd65c9
index 15c9110c0..4d6b7ad18 100644
dd65c9
--- a/src/journal/journald-stream.c
dd65c9
+++ b/src/journal/journald-stream.c
dd65c9
@@ -53,6 +53,16 @@ typedef enum StdoutStreamState {
dd65c9
         STDOUT_STREAM_RUNNING
dd65c9
 } StdoutStreamState;
dd65c9
 
dd65c9
+/* The different types of log record terminators: a real \n was read, a NUL character was read, the maximum line length
dd65c9
+ * was reached, or the end of the stream was reached */
dd65c9
+
dd65c9
+typedef enum LineBreak {
dd65c9
+        LINE_BREAK_NEWLINE,
dd65c9
+        LINE_BREAK_NUL,
dd65c9
+        LINE_BREAK_LINE_MAX,
dd65c9
+        LINE_BREAK_EOF,
dd65c9
+} LineBreak;
dd65c9
+
dd65c9
 struct StdoutStream {
dd65c9
         Server *server;
dd65c9
         StdoutStreamState state;
dd65c9
@@ -71,14 +81,17 @@ struct StdoutStream {
dd65c9
 
dd65c9
         bool fdstore:1;
dd65c9
 
dd65c9
-        char buffer[LINE_MAX+1];
dd65c9
+        char *buffer;
dd65c9
         size_t length;
dd65c9
+        size_t allocated;
dd65c9
 
dd65c9
         sd_event_source *event_source;
dd65c9
 
dd65c9
         char *state_file;
dd65c9
 
dd65c9
         LIST_FIELDS(StdoutStream, stdout_stream);
dd65c9
+
dd65c9
+        char id_field[sizeof("_STREAM_ID=")-1 + SD_ID128_STRING_MAX];
dd65c9
 };
dd65c9
 
dd65c9
 void stdout_stream_free(StdoutStream *s) {
dd65c9
@@ -101,6 +114,7 @@ void stdout_stream_free(StdoutStream *s) {
dd65c9
         free(s->identifier);
dd65c9
         free(s->unit_id);
dd65c9
         free(s->state_file);
dd65c9
+        free(s->buffer);
dd65c9
 
dd65c9
         free(s);
dd65c9
 }
dd65c9
@@ -151,12 +165,14 @@ static int stdout_stream_save(StdoutStream *s) {
dd65c9
                 "LEVEL_PREFIX=%i\n"
dd65c9
                 "FORWARD_TO_SYSLOG=%i\n"
dd65c9
                 "FORWARD_TO_KMSG=%i\n"
dd65c9
-                "FORWARD_TO_CONSOLE=%i\n",
dd65c9
+                "FORWARD_TO_CONSOLE=%i\n"
dd65c9
+                "STREAM_ID=%s\n",
dd65c9
                 s->priority,
dd65c9
                 s->level_prefix,
dd65c9
                 s->forward_to_syslog,
dd65c9
                 s->forward_to_kmsg,
dd65c9
-                s->forward_to_console);
dd65c9
+                s->forward_to_console,
dd65c9
+                s->id_field + strlen("_STREAM_ID="));
dd65c9
 
dd65c9
         if (!isempty(s->identifier)) {
dd65c9
                 _cleanup_free_ char *escaped;
dd65c9
@@ -211,8 +227,8 @@ finish:
dd65c9
         return r;
dd65c9
 }
dd65c9
 
dd65c9
-static int stdout_stream_log(StdoutStream *s, const char *p) {
dd65c9
-        struct iovec iovec[N_IOVEC_META_FIELDS + 5];
dd65c9
+static int stdout_stream_log(StdoutStream *s, const char *p, LineBreak line_break) {
dd65c9
+        struct iovec iovec[N_IOVEC_META_FIELDS + 7];
dd65c9
         int priority;
dd65c9
         char syslog_priority[] = "PRIORITY=\0";
dd65c9
         char syslog_facility[sizeof("SYSLOG_FACILITY=")-1 + DECIMAL_STR_MAX(int) + 1];
dd65c9
@@ -245,6 +261,8 @@ static int stdout_stream_log(StdoutStream *s, const char *p) {
dd65c9
 
dd65c9
         IOVEC_SET_STRING(iovec[n++], "_TRANSPORT=stdout");
dd65c9
 
dd65c9
+        IOVEC_SET_STRING(iovec[n++], s->id_field);
dd65c9
+
dd65c9
         syslog_priority[strlen("PRIORITY=")] = '0' + LOG_PRI(priority);
dd65c9
         IOVEC_SET_STRING(iovec[n++], syslog_priority);
dd65c9
 
dd65c9
@@ -259,6 +277,18 @@ static int stdout_stream_log(StdoutStream *s, const char *p) {
dd65c9
                         IOVEC_SET_STRING(iovec[n++], syslog_identifier);
dd65c9
         }
dd65c9
 
dd65c9
+        if (line_break != LINE_BREAK_NEWLINE) {
dd65c9
+                const char *c;
dd65c9
+
dd65c9
+                /* If this log message was generated due to an uncommon line break then mention this in the log
dd65c9
+                 * entry */
dd65c9
+
dd65c9
+                c =     line_break == LINE_BREAK_NUL ?      "_LINE_BREAK=nul" :
dd65c9
+                        line_break == LINE_BREAK_LINE_MAX ? "_LINE_BREAK=line-max" :
dd65c9
+                                                            "_LINE_BREAK=eof";
dd65c9
+                IOVEC_SET_STRING(iovec[n++], c);
dd65c9
+        }
dd65c9
+
dd65c9
         message = strappend("MESSAGE=", p);
dd65c9
         if (message)
dd65c9
                 IOVEC_SET_STRING(iovec[n++], message);
dd65c9
@@ -268,12 +298,18 @@ static int stdout_stream_log(StdoutStream *s, const char *p) {
dd65c9
         return 0;
dd65c9
 }
dd65c9
 
dd65c9
-static int stdout_stream_line(StdoutStream *s, char *p) {
dd65c9
+static int stdout_stream_line(StdoutStream *s, char *p, LineBreak line_break) {
dd65c9
         int r;
dd65c9
 
dd65c9
         assert(s);
dd65c9
         assert(p);
dd65c9
 
dd65c9
+        /* line breaks by NUL, line max length or EOF are not permissible during the negotiation part of the protocol */
dd65c9
+        if (line_break != LINE_BREAK_NEWLINE && s->state != STDOUT_STREAM_RUNNING) {
dd65c9
+                log_warning("Control protocol line not properly terminated.");
dd65c9
+                return -EINVAL;
dd65c9
+        }
dd65c9
+
dd65c9
         p = strstrip(p);
dd65c9
 
dd65c9
         switch (s->state) {
dd65c9
@@ -362,7 +398,7 @@ static int stdout_stream_line(StdoutStream *s, char *p) {
dd65c9
                 return 0;
dd65c9
 
dd65c9
         case STDOUT_STREAM_RUNNING:
dd65c9
-                return stdout_stream_log(s, p);
dd65c9
+                return stdout_stream_log(s, p, line_break);
dd65c9
         }
dd65c9
 
dd65c9
         assert_not_reached("Unknown stream state");
dd65c9
@@ -378,21 +414,32 @@ static int stdout_stream_scan(StdoutStream *s, bool force_flush) {
dd65c9
         p = s->buffer;
dd65c9
         remaining = s->length;
dd65c9
         for (;;) {
dd65c9
-                char *end;
dd65c9
+                LineBreak line_break;
dd65c9
                 size_t skip;
dd65c9
 
dd65c9
-                end = memchr(p, '\n', remaining);
dd65c9
-                if (end)
dd65c9
-                        skip = end - p + 1;
dd65c9
-                else if (remaining >= sizeof(s->buffer) - 1) {
dd65c9
-                        end = p + sizeof(s->buffer) - 1;
dd65c9
+                char *end1, *end2;
dd65c9
+
dd65c9
+                end1 = memchr(p, '\n', remaining);
dd65c9
+                end2 = memchr(p, 0, end1 ? (size_t) (end1 - p) : remaining);
dd65c9
+
dd65c9
+                if (end2) {
dd65c9
+                        /* We found a NUL terminator */
dd65c9
+                        skip = end2 - p + 1;
dd65c9
+                        line_break = LINE_BREAK_NUL;
dd65c9
+                } else if (end1) {
dd65c9
+                        /* We found a \n terminator */
dd65c9
+                        *end1 = 0;
dd65c9
+                        skip = end1 - p + 1;
dd65c9
+                        line_break = LINE_BREAK_NEWLINE;
dd65c9
+                } else if (remaining >= s->server->line_max) {
dd65c9
+                        /* Force a line break after the maximum line length */
dd65c9
+                        *(p + s->server->line_max) = 0;
dd65c9
                         skip = remaining;
dd65c9
+                        line_break = LINE_BREAK_LINE_MAX;
dd65c9
                 } else
dd65c9
                         break;
dd65c9
 
dd65c9
-                *end = 0;
dd65c9
-
dd65c9
-                r = stdout_stream_line(s, p);
dd65c9
+                r = stdout_stream_line(s, p, line_break);
dd65c9
                 if (r < 0)
dd65c9
                         return r;
dd65c9
 
dd65c9
@@ -402,7 +449,7 @@ static int stdout_stream_scan(StdoutStream *s, bool force_flush) {
dd65c9
 
dd65c9
         if (force_flush && remaining > 0) {
dd65c9
                 p[remaining] = 0;
dd65c9
-                r = stdout_stream_line(s, p);
dd65c9
+                r = stdout_stream_line(s, p, LINE_BREAK_EOF);
dd65c9
                 if (r < 0)
dd65c9
                         return r;
dd65c9
 
dd65c9
@@ -420,6 +467,7 @@ static int stdout_stream_scan(StdoutStream *s, bool force_flush) {
dd65c9
 
dd65c9
 static int stdout_stream_process(sd_event_source *es, int fd, uint32_t revents, void *userdata) {
dd65c9
         StdoutStream *s = userdata;
dd65c9
+        size_t limit;
dd65c9
         ssize_t l;
dd65c9
         int r;
dd65c9
 
dd65c9
@@ -430,9 +478,20 @@ static int stdout_stream_process(sd_event_source *es, int fd, uint32_t revents,
dd65c9
                 goto terminate;
dd65c9
         }
dd65c9
 
dd65c9
-        l = read(s->fd, s->buffer+s->length, sizeof(s->buffer)-1-s->length);
dd65c9
-        if (l < 0) {
dd65c9
+        /* If the buffer is full already (discounting the extra NUL we need), add room for another 1K */
dd65c9
+        if (s->length + 1 >= s->allocated) {
dd65c9
+                if (!GREEDY_REALLOC(s->buffer, s->allocated, s->length + 1 + 1024)) {
dd65c9
+                        log_oom();
dd65c9
+                        goto terminate;
dd65c9
+                }
dd65c9
+        }
dd65c9
+
dd65c9
+        /* Try to make use of the allocated buffer in full, but never read more than the configured line size. Also,
dd65c9
+         * always leave room for a terminating NUL we might need to add. */
dd65c9
+        limit = MIN(s->allocated - 1, s->server->line_max);
dd65c9
 
dd65c9
+        l = read(s->fd, s->buffer + s->length, limit - s->length);
dd65c9
+        if (l < 0) {
dd65c9
                 if (errno == EAGAIN)
dd65c9
                         return 0;
dd65c9
 
dd65c9
@@ -459,11 +518,16 @@ terminate:
dd65c9
 
dd65c9
 static int stdout_stream_install(Server *s, int fd, StdoutStream **ret) {
dd65c9
         _cleanup_(stdout_stream_freep) StdoutStream *stream = NULL;
dd65c9
+        sd_id128_t id;
dd65c9
         int r;
dd65c9
 
dd65c9
         assert(s);
dd65c9
         assert(fd >= 0);
dd65c9
 
dd65c9
+        r = sd_id128_randomize(&id;;
dd65c9
+        if (r < 0)
dd65c9
+                return log_error_errno(r, "Failed to generate stream ID: %m");
dd65c9
+
dd65c9
         stream = new0(StdoutStream, 1);
dd65c9
         if (!stream)
dd65c9
                 return log_oom();
dd65c9
@@ -471,6 +535,8 @@ static int stdout_stream_install(Server *s, int fd, StdoutStream **ret) {
dd65c9
         stream->fd = -1;
dd65c9
         stream->priority = LOG_INFO;
dd65c9
 
dd65c9
+        xsprintf(stream->id_field, "_STREAM_ID=" SD_ID128_FORMAT_STR, SD_ID128_FORMAT_VAL(id));
dd65c9
+
dd65c9
         r = getpeercred(fd, &stream->ucred);
dd65c9
         if (r < 0)
dd65c9
                 return log_error_errno(r, "Failed to determine peer credentials: %m");
dd65c9
@@ -545,7 +611,8 @@ static int stdout_stream_load(StdoutStream *stream, const char *fname) {
dd65c9
                 *level_prefix = NULL,
dd65c9
                 *forward_to_syslog = NULL,
dd65c9
                 *forward_to_kmsg = NULL,
dd65c9
-                *forward_to_console = NULL;
dd65c9
+                *forward_to_console = NULL,
dd65c9
+                *stream_id = NULL;
dd65c9
         int r;
dd65c9
 
dd65c9
         assert(stream);
dd65c9
@@ -565,6 +632,7 @@ static int stdout_stream_load(StdoutStream *stream, const char *fname) {
dd65c9
                            "FORWARD_TO_CONSOLE", &forward_to_console,
dd65c9
                            "IDENTIFIER", &stream->identifier,
dd65c9
                            "UNIT", &stream->unit_id,
dd65c9
+                           "STREAM_ID", &stream_id,
dd65c9
                            NULL);
dd65c9
         if (r < 0)
dd65c9
                 return log_error_errno(r, "Failed to read: %s", stream->state_file);
dd65c9
@@ -601,6 +669,14 @@ static int stdout_stream_load(StdoutStream *stream, const char *fname) {
dd65c9
                         stream->forward_to_console = r;
dd65c9
         }
dd65c9
 
dd65c9
+        if (stream_id) {
dd65c9
+                sd_id128_t id;
dd65c9
+
dd65c9
+                r = sd_id128_from_string(stream_id, &id;;
dd65c9
+                if (r >= 0)
dd65c9
+                        xsprintf(stream->id_field, "_STREAM_ID=" SD_ID128_FORMAT_STR, SD_ID128_FORMAT_VAL(id));
dd65c9
+        }
dd65c9
+
dd65c9
         return 0;
dd65c9
 }
dd65c9
 
dd65c9
diff --git a/src/journal/journald.conf b/src/journal/journald.conf
dd65c9
index 3907dfb7f..5355ec2b2 100644
dd65c9
--- a/src/journal/journald.conf
dd65c9
+++ b/src/journal/journald.conf
dd65c9
@@ -37,3 +37,4 @@
dd65c9
 #MaxLevelKMsg=notice
dd65c9
 #MaxLevelConsole=info
dd65c9
 #MaxLevelWall=emerg
dd65c9
+#LineMax=48K