alexk / rpms / rpm

Forked from rpms/rpm 2 years ago
Clone
2f13d7
From 7bd31ce85b2ed377f495c31fcea2422a07739e24 Mon Sep 17 00:00:00 2001
2f13d7
From: Matthew Almond <malmond@fb.com>
2f13d7
Date: Fri, 8 Nov 2019 09:29:43 -0800
2f13d7
Subject: [PATCH 01/30] RPM with Copy on Write
2f13d7
2f13d7
This is part of https://fedoraproject.org/wiki/Changes/RPMCoW
2f13d7
2f13d7
The majority of changes are in two new programs:
2f13d7
2f13d7
= rpm2extents
2f13d7
2f13d7
Modeled as a 'stream processor'. It reads a regular .rpm file on stdin,
2f13d7
and produces a modified .rpm file on stdout. The lead, signature and
2f13d7
headers are preserved 1:1 to allow all the normal metadata inspection,
2f13d7
signature verification to work as expected. Only the 'payload' is
2f13d7
modified.
2f13d7
2f13d7
The primary motivation for this tool is to re-organize the payload as a
2f13d7
sequence of raw file extents (hence the name). The files are organized
2f13d7
by their digest identity instead of path/filename. If any digest is
2f13d7
repeated, then the file is skipped/de-duped. Only regular files are
2f13d7
represented. All other entries like directories, symlinks, devices are
2f13d7
fully described in the headers and are omitted.
2f13d7
2f13d7
The files are padded so they start on `sysconf(_SC_PAGESIZE)` boundries
2f13d7
to permit 'reflink' syscalls to work in the `reflink` plugin.
2f13d7
2f13d7
At the end of the file is a footer with 3 sections:
2f13d7
2f13d7
1. List of calculated digests of the input stream. This is used in
2f13d7
   `librepo` because the file *written* is a derivative, and not the
2f13d7
   same as the repo metadata describes. `rpm2extents` takes one or more
2f13d7
   positional arguments that described which digest algorithms are
2f13d7
   desired. This is often just `SHA256`. This program is only measuring
2f13d7
   and recording the digest - it does not express an opinion on whether
2f13d7
   the file is correct. Due to the API on most compression libraries
2f13d7
   directly reading the source file, the whole file digest is measured
2f13d7
   using a subprocess and pipes. I don't love it, but it works.
2f13d7
2. Sorted List of file content digests + offset pairs. This is used in
2f13d7
   the plugin with a trivial binary search to locate the start of file
2f13d7
   content. The size is not needed because it's part of normal headers.
2f13d7
3. (offset of 1., offset of 2., 8 byte MAGIC value) triple
2f13d7
2f13d7
= reflink plugin
2f13d7
2f13d7
Looks for the 8 byte magic value at the end of the rpm file. If present
2f13d7
it alters the `RPMTAG_PAYLOADFORMAT` in memory to `clon`, and reads in
2f13d7
the digest-> offset table.
2f13d7
2f13d7
`rpmPackageFilesInstall()` in `fsm.c` is
2f13d7
modified to alter the enumeration strategy from
2f13d7
`rpmfiNewArchiveReader()` to `rpmfilesIter()` if not `cpio`. This is
2f13d7
needed because there is no cpio to enumerate. In the same function, if
2f13d7
`rpmpluginsCallFsmFilePre()` returns `RPMRC_PLUGIN_CONTENTS` then
2f13d7
`fsmMkfile()` is skipped as it is assumed the plugin did the work.
2f13d7
2f13d7
The majority of the work is in `reflink_fsm_file_pre()` - the per file
2f13d7
hook for RPM plugins. If the file enumerated in
2f13d7
`rpmPackageFilesInstall()` is a regular file, this function will look up
2f13d7
the offset in the digest->offset table and will try to reflink it, then
2f13d7
fall back to a regular copy. If reflinking does work: we will have
2f13d7
reflinked a whole number of pages, so we truncate the file to the
2f13d7
expected size. Therefore installing most files does involve two writes:
2f13d7
the reflink of the full size, then a fork/copy on write for the last
2f13d7
page worth.
2f13d7
2f13d7
If the file passed to `reflink_fsm_file_pre()` is anything other than a
2f13d7
regular file, it return `RPMRC_OK` so the normal mechanics of
2f13d7
`rpmPackageFilesInstall()` are used. That handles directories, symlinks
2f13d7
and other non file types.
2f13d7
2f13d7
= New API for internal use
2f13d7
2f13d7
1. `rpmReadPackageRaw()` is used within `rpm2extents` to read all the
2f13d7
   headers without trying to validate signatures. This eliminates the
2f13d7
   runtime dependency on rpmdb.
2f13d7
2. `rpmteFd()` exposes the Fd behind the rpmte, so plugins can interact
2f13d7
   with the rpm itself.
2f13d7
3. `RPMRC_PLUGIN_CONTENTS` in `rpmRC_e` for use in
2f13d7
   `rpmpluginsCallFsmFilePre()` specifically.
2f13d7
4. `pgpStringVal()` is used to help parse the command line in
2f13d7
   `rpm2extents` - the positional arguments are strings, and this
2f13d7
   converts the values back to the values in the table.
2f13d7
2f13d7
Nothing has been removed, and none of the changes are intended to be
2f13d7
used externally, so I don't think a soname bump is warranted here.
2f13d7
---
2f13d7
 Makefile.am         |   6 +-
2f13d7
 lib/depends.c       |   2 +
2f13d7
 lib/fsm.c           |  45 +++-
2f13d7
 lib/package.c       |  40 ++++
2f13d7
 lib/rpmlib.h        |   9 +
2f13d7
 lib/rpmplugins.c    |  21 +-
2f13d7
 lib/rpmte.c         |   5 +
2f13d7
 lib/rpmte.h         |   2 +
2f13d7
 lib/rpmtypes.h      |   3 +-
2f13d7
 macros.in           |   1 +
2f13d7
 plugins/Makefile.am |   4 +
2f13d7
 plugins/reflink.c   | 340 +++++++++++++++++++++++++++++
2f13d7
 rpm2extents.c       | 519 ++++++++++++++++++++++++++++++++++++++++++++
2f13d7
 rpmio/rpmpgp.c      |  10 +
2f13d7
 rpmio/rpmpgp.h      |   9 +
2f13d7
 15 files changed, 1004 insertions(+), 12 deletions(-)
2f13d7
 create mode 100644 plugins/reflink.c
2f13d7
 create mode 100644 rpm2extents.c
2f13d7
2f13d7
diff --git a/Makefile.am b/Makefile.am
2f13d7
index e5c75d7b4..288668819 100644
2f13d7
--- a/Makefile.am
2f13d7
+++ b/Makefile.am
2f13d7
@@ -99,7 +99,7 @@ pkginclude_HEADERS += build/rpmfc.h
2f13d7
 pkginclude_HEADERS += build/rpmspec.h
2f13d7
 
2f13d7
 
2f13d7
-bin_PROGRAMS =		rpm rpm2cpio rpmbuild rpmdb rpmkeys rpmsign rpmspec
2f13d7
+bin_PROGRAMS =		rpm rpm2cpio rpmbuild rpmdb rpmkeys rpmsign rpmspec rpm2extents
2f13d7
 if WITH_ARCHIVE
2f13d7
 bin_PROGRAMS += 	rpm2archive 
2f13d7
 endif
2f13d7
@@ -154,6 +154,10 @@ rpm2cpio_SOURCES =	rpm2cpio.c debug.h system.h
2f13d7
 rpm2cpio_LDADD =	lib/librpm.la rpmio/librpmio.la
2f13d7
 rpm2cpio_LDADD +=	@WITH_POPT_LIB@
2f13d7
 
2f13d7
+rpm2extents_SOURCES =	rpm2extents.c debug.h system.h
2f13d7
+rpm2extents_LDADD =	lib/librpm.la rpmio/librpmio.la
2f13d7
+rpm2extents_LDADD +=	@WITH_POPT_LIB@
2f13d7
+
2f13d7
 rpm2archive_SOURCES =	rpm2archive.c debug.h system.h
2f13d7
 rpm2archive_LDADD =	lib/librpm.la rpmio/librpmio.la
2f13d7
 rpm2archive_LDADD +=	@WITH_POPT_LIB@ @WITH_ARCHIVE_LIB@
2f13d7
diff --git a/lib/depends.c b/lib/depends.c
2f13d7
index 30234df3d..8998afcd3 100644
2f13d7
--- a/lib/depends.c
2f13d7
+++ b/lib/depends.c
2f13d7
@@ -81,6 +81,8 @@ static rpmRC headerCheckPayloadFormat(Header h) {
2f13d7
      */
2f13d7
     if (!payloadfmt) return rc;
2f13d7
 
2f13d7
+    if (rstreq(payloadfmt, "clon")) return rc;
2f13d7
+
2f13d7
     if (!rstreq(payloadfmt, "cpio")) {
2f13d7
         char *nevra = headerGetAsString(h, RPMTAG_NEVRA);
2f13d7
         if (payloadfmt && rstreq(payloadfmt, "drpm")) {
2f13d7
diff --git a/lib/fsm.c b/lib/fsm.c
2f13d7
index 935a0a5c6..90193c749 100644
2f13d7
--- a/lib/fsm.c
2f13d7
+++ b/lib/fsm.c
2f13d7
@@ -8,6 +8,7 @@
2f13d7
 
2f13d7
 #include <utime.h>
2f13d7
 #include <errno.h>
2f13d7
+#include <stdbool.h>
2f13d7
 #if WITH_CAP
2f13d7
 #include <sys/capability.h>
2f13d7
 #endif
2f13d7
@@ -19,6 +20,7 @@
2f13d7
 
2f13d7
 #include "rpmio/rpmio_internal.h"	/* fdInit/FiniDigest */
2f13d7
 #include "lib/fsm.h"
2f13d7
+#include "lib/rpmlib.h"
2f13d7
 #include "lib/rpmte_internal.h"	/* XXX rpmfs */
2f13d7
 #include "lib/rpmplugins.h"	/* rpm plugins hooks */
2f13d7
 #include "lib/rpmug.h"
2f13d7
@@ -52,6 +54,7 @@ struct filedata_s {
2f13d7
     int stage;
2f13d7
     int setmeta;
2f13d7
     int skip;
2f13d7
+    int plugin_contents;
2f13d7
     rpmFileAction action;
2f13d7
     const char *suffix;
2f13d7
     char *fpath;
2f13d7
@@ -891,6 +894,14 @@ int rpmPackageFilesInstall(rpmts ts, rpmte te, rpmfiles files,
2f13d7
     struct filedata_s *fdata = xcalloc(fc, sizeof(*fdata));
2f13d7
     struct filedata_s *firstlink = NULL;
2f13d7
 
2f13d7
+    Header h = rpmteHeader(te);
2f13d7
+    const char *payloadfmt = headerGetString(h, RPMTAG_PAYLOADFORMAT);
2f13d7
+    bool cpio = true;
2f13d7
+
2f13d7
+    if (payloadfmt && rstreq(payloadfmt, "clon")) {
2f13d7
+	cpio = false;
2f13d7
+    }
2f13d7
+
2f13d7
     /* transaction id used for temporary path suffix while installing */
2f13d7
     rasprintf(&tid, ";%08x", (unsigned)rpmtsGetTid(ts));
2f13d7
 
2f13d7
@@ -911,12 +922,23 @@ int rpmPackageFilesInstall(rpmts ts, rpmte te, rpmfiles files,
2f13d7
 	/* Remap file perms, owner, and group. */
2f13d7
 	rc = rpmfiStat(fi, 1, &fp->sb);
2f13d7
 
2f13d7
-	setFileState(fs, fx);
2f13d7
 	fsmDebug(fp->fpath, fp->action, &fp->sb);
2f13d7
 
2f13d7
 	/* Run fsm file pre hook for all plugins */
2f13d7
 	rc = rpmpluginsCallFsmFilePre(plugins, fi, fp->fpath,
2f13d7
 				      fp->sb.st_mode, fp->action);
2f13d7
+	fp->plugin_contents = false;
2f13d7
+	switch (rc) {
2f13d7
+	case RPMRC_OK:
2f13d7
+	    setFileState(fs, fx);
2f13d7
+	    break;
2f13d7
+	case RPMRC_PLUGIN_CONTENTS:
2f13d7
+	    fp->plugin_contents = true;
2f13d7
+	    // reduce reads on cpio to this value. Could be zero if
2f13d7
+	    // this is from a hard link.
2f13d7
+	    rc = RPMRC_OK;
2f13d7
+	    break;
2f13d7
+	}
2f13d7
 	fp->stage = FILE_PRE;
2f13d7
     }
2f13d7
     fi = rpmfiFree(fi);
2f13d7
@@ -924,10 +946,14 @@ int rpmPackageFilesInstall(rpmts ts, rpmte te, rpmfiles files,
2f13d7
     if (rc)
2f13d7
 	goto exit;
2f13d7
 
2f13d7
-    fi = rpmfiNewArchiveReader(payload, files, RPMFI_ITER_READ_ARCHIVE);
2f13d7
-    if (fi == NULL) {
2f13d7
-        rc = RPMERR_BAD_MAGIC;
2f13d7
-        goto exit;
2f13d7
+    if (cpio) {
2f13d7
+	fi = rpmfiNewArchiveReader(payload, files, RPMFI_ITER_READ_ARCHIVE);
2f13d7
+	if (fi == NULL) {
2f13d7
+	    rc = RPMERR_BAD_MAGIC;
2f13d7
+	    goto exit;
2f13d7
+	}
2f13d7
+    } else {
2f13d7
+	fi = rpmfilesIter(files, RPMFI_ITER_FWD);
2f13d7
     }
2f13d7
 
2f13d7
     /* Detect and create directories not explicitly in package. */
2f13d7
@@ -969,8 +995,12 @@ int rpmPackageFilesInstall(rpmts ts, rpmte te, rpmfiles files,
2f13d7
 
2f13d7
             if (S_ISREG(fp->sb.st_mode)) {
2f13d7
 		if (rc == RPMERR_ENOENT) {
2f13d7
-		    rc = fsmMkfile(fi, fp, files, psm, nodigest,
2f13d7
-				   &firstlink, &firstlinkfile);
2f13d7
+		    if(fp->plugin_contents) {
2f13d7
+			rc = RPMRC_OK;
2f13d7
+		    }else {
2f13d7
+			rc = fsmMkfile(fi, fp, files, psm, nodigest,
2f13d7
+			    &firstlink, &firstlinkfile);
2f13d7
+		    }
2f13d7
 		}
2f13d7
             } else if (S_ISDIR(fp->sb.st_mode)) {
2f13d7
                 if (rc == RPMERR_ENOENT) {
2f13d7
@@ -1078,6 +1108,7 @@ int rpmPackageFilesInstall(rpmts ts, rpmte te, rpmfiles files,
2f13d7
     rpmswAdd(rpmtsOp(ts, RPMTS_OP_DIGEST), fdOp(payload, FDSTAT_DIGEST));
2f13d7
 
2f13d7
 exit:
2f13d7
+    h = headerFree(h);
2f13d7
     fi = rpmfiFree(fi);
2f13d7
     Fclose(payload);
2f13d7
     free(tid);
2f13d7
diff --git a/lib/package.c b/lib/package.c
2f13d7
index 281275029..90bd0d8a7 100644
2f13d7
--- a/lib/package.c
2f13d7
+++ b/lib/package.c
2f13d7
@@ -404,5 +404,45 @@ exit:
2f13d7
     return rc;
2f13d7
 }
2f13d7
 
2f13d7
+rpmRC rpmReadPackageRaw(FD_t fd, Header * sigp, Header * hdrp)
2f13d7
+{
2f13d7
+    char *msg = NULL;
2f13d7
+    hdrblob sigblob = hdrblobCreate();
2f13d7
+    hdrblob blob = hdrblobCreate();
2f13d7
+    Header h = NULL;
2f13d7
+    Header sigh = NULL;
2f13d7
+
2f13d7
+    rpmRC rc = rpmLeadRead(fd, &msg;;
2f13d7
+    if (rc != RPMRC_OK)
2f13d7
+	goto exit;
2f13d7
+
2f13d7
+    rc = hdrblobRead(fd, 1, 0, RPMTAG_HEADERSIGNATURES, sigblob, &msg;;
2f13d7
+    if (rc != RPMRC_OK)
2f13d7
+	goto exit;
2f13d7
+
2f13d7
+    rc = hdrblobRead(fd, 1, 1, RPMTAG_HEADERIMMUTABLE, blob, &msg;;
2f13d7
+    if (rc != RPMRC_OK)
2f13d7
+	goto exit;
2f13d7
+
2f13d7
+    rc = hdrblobImport(sigblob, 0, &sigh, &msg;;
2f13d7
+    if (rc)
2f13d7
+	goto exit;
2f13d7
 
2f13d7
+    rc = hdrblobImport(blob, 0, &h, &msg;;
2f13d7
+    if (rc)
2f13d7
+	goto exit;
2f13d7
 
2f13d7
+    *sigp = headerLink(sigh);
2f13d7
+    *hdrp = headerLink(h);
2f13d7
+
2f13d7
+exit:
2f13d7
+    if (rc != RPMRC_OK && msg)
2f13d7
+	rpmlog(RPMLOG_ERR, "%s: %s\n", Fdescr(fd), msg);
2f13d7
+    hdrblobFree(sigblob);
2f13d7
+    hdrblobFree(blob);
2f13d7
+    headerFree(sigh);
2f13d7
+    headerFree(h);
2f13d7
+    free(msg);
2f13d7
+
2f13d7
+    return rc;
2f13d7
+}
2f13d7
diff --git a/lib/rpmlib.h b/lib/rpmlib.h
2f13d7
index 0879d04e5..a09ba0daf 100644
2f13d7
--- a/lib/rpmlib.h
2f13d7
+++ b/lib/rpmlib.h
2f13d7
@@ -155,6 +155,15 @@ rpmRC rpmReadHeader(rpmts ts, FD_t fd, Header *hdrp, char ** msg);
2f13d7
 rpmRC rpmReadPackageFile(rpmts ts, FD_t fd,
2f13d7
 		const char * fn, Header * hdrp);
2f13d7
 
2f13d7
+/** \ingroup header
2f13d7
+ * Return package signature, header from file handle, no verification.
2f13d7
+ * @param fd		file handle
2f13d7
+ * @param[out] sigp		address of header (or NULL)
2f13d7
+ * @param[out] hdrp		address of header (or NULL)
2f13d7
+ * @return		RPMRC_OK on success
2f13d7
+ */
2f13d7
+rpmRC rpmReadPackageRaw(FD_t fd, Header * sigp, Header * hdrp);
2f13d7
+
2f13d7
 /** \ingroup rpmtrans
2f13d7
  * Install source package.
2f13d7
  * @param ts		transaction set
2f13d7
diff --git a/lib/rpmplugins.c b/lib/rpmplugins.c
2f13d7
index 62d75c4cf..c5084d398 100644
2f13d7
--- a/lib/rpmplugins.c
2f13d7
+++ b/lib/rpmplugins.c
2f13d7
@@ -356,13 +356,28 @@ rpmRC rpmpluginsCallFsmFilePre(rpmPlugins plugins, rpmfi fi, const char *path,
2f13d7
     plugin_fsm_file_pre_func hookFunc;
2f13d7
     int i;
2f13d7
     rpmRC rc = RPMRC_OK;
2f13d7
+    rpmRC hook_rc;
2f13d7
 
2f13d7
     for (i = 0; i < plugins->count; i++) {
2f13d7
 	rpmPlugin plugin = plugins->plugins[i];
2f13d7
 	RPMPLUGINS_SET_HOOK_FUNC(fsm_file_pre);
2f13d7
-	if (hookFunc && hookFunc(plugin, fi, path, file_mode, op) == RPMRC_FAIL) {
2f13d7
-	    rpmlog(RPMLOG_ERR, "Plugin %s: hook fsm_file_pre failed\n", plugin->name);
2f13d7
-	    rc = RPMRC_FAIL;
2f13d7
+	if (hookFunc) {
2f13d7
+	    hook_rc = hookFunc(plugin, fi, path, file_mode, op);
2f13d7
+	    if (hook_rc == RPMRC_FAIL) {
2f13d7
+		rpmlog(RPMLOG_ERR, "Plugin %s: hook fsm_file_pre failed\n", plugin->name);
2f13d7
+		rc = RPMRC_FAIL;
2f13d7
+	    } else if (hook_rc == RPMRC_PLUGIN_CONTENTS && rc != RPMRC_FAIL) {
2f13d7
+		if (rc == RPMRC_PLUGIN_CONTENTS) {
2f13d7
+		    /*
2f13d7
+		    Another plugin already said it'd handle contents. It's undefined how
2f13d7
+		    these would combine, so treat this as a failure condition.
2f13d7
+		    */
2f13d7
+		    rc = RPMRC_FAIL;
2f13d7
+		} else {
2f13d7
+		    /* Plugin will handle content */
2f13d7
+		    rc = RPMRC_PLUGIN_CONTENTS;
2f13d7
+		}
2f13d7
+	    }
2f13d7
 	}
2f13d7
     }
2f13d7
 
2f13d7
diff --git a/lib/rpmte.c b/lib/rpmte.c
2f13d7
index 3663604e7..d43dc41ad 100644
2f13d7
--- a/lib/rpmte.c
2f13d7
+++ b/lib/rpmte.c
2f13d7
@@ -423,6 +423,11 @@ FD_t rpmteSetFd(rpmte te, FD_t fd)
2f13d7
     return NULL;
2f13d7
 }
2f13d7
 
2f13d7
+FD_t rpmteFd(rpmte te)
2f13d7
+{
2f13d7
+    return (te != NULL ? te->fd : NULL);
2f13d7
+}
2f13d7
+
2f13d7
 fnpyKey rpmteKey(rpmte te)
2f13d7
 {
2f13d7
     return (te != NULL ? te->key : NULL);
2f13d7
diff --git a/lib/rpmte.h b/lib/rpmte.h
2f13d7
index 81acf7a19..6fc0a9f91 100644
2f13d7
--- a/lib/rpmte.h
2f13d7
+++ b/lib/rpmte.h
2f13d7
@@ -209,6 +209,8 @@ const char * rpmteNEVR(rpmte te);
2f13d7
  */
2f13d7
 const char * rpmteNEVRA(rpmte te);
2f13d7
 
2f13d7
+FD_t rpmteFd(rpmte te);
2f13d7
+
2f13d7
 /** \ingroup rpmte
2f13d7
  * Retrieve key from transaction element.
2f13d7
  * @param te		transaction element
2f13d7
diff --git a/lib/rpmtypes.h b/lib/rpmtypes.h
2f13d7
index e8e69b506..af2611e9e 100644
2f13d7
--- a/lib/rpmtypes.h
2f13d7
+++ b/lib/rpmtypes.h
2f13d7
@@ -106,7 +106,8 @@ typedef	enum rpmRC_e {
2f13d7
     RPMRC_NOTFOUND	= 1,	/*!< Generic not found code. */
2f13d7
     RPMRC_FAIL		= 2,	/*!< Generic failure code. */
2f13d7
     RPMRC_NOTTRUSTED	= 3,	/*!< Signature is OK, but key is not trusted. */
2f13d7
-    RPMRC_NOKEY		= 4	/*!< Public key is unavailable. */
2f13d7
+    RPMRC_NOKEY		= 4,	/*!< Public key is unavailable. */
2f13d7
+    RPMRC_PLUGIN_CONTENTS = 5     /*!< fsm_file_pre plugin is handling content */
2f13d7
 } rpmRC;
2f13d7
 
2f13d7
 #ifdef __cplusplus
2f13d7
diff --git a/macros.in b/macros.in
2f13d7
index e90cefa9a..363252b0f 100644
2f13d7
--- a/macros.in
2f13d7
+++ b/macros.in
2f13d7
@@ -1143,6 +1143,7 @@ package or when debugging this package.\
2f13d7
 
2f13d7
 # Transaction plugin macros
2f13d7
 %__plugindir		%{_libdir}/rpm-plugins
2f13d7
+%__transaction_reflink		%{__plugindir}/reflink.so
2f13d7
 %__transaction_systemd_inhibit	%{__plugindir}/systemd_inhibit.so
2f13d7
 %__transaction_selinux		%{__plugindir}/selinux.so
2f13d7
 %__transaction_syslog		%{__plugindir}/syslog.so
2f13d7
diff --git a/plugins/Makefile.am b/plugins/Makefile.am
2f13d7
index 3a929d0ce..ad0d3bce7 100644
2f13d7
--- a/plugins/Makefile.am
2f13d7
+++ b/plugins/Makefile.am
2f13d7
@@ -42,6 +42,10 @@ prioreset_la_SOURCES = prioreset.c
2f13d7
 prioreset_la_LIBADD = $(top_builddir)/lib/librpm.la $(top_builddir)/rpmio/librpmio.la
2f13d7
 plugins_LTLIBRARIES += prioreset.la
2f13d7
 
2f13d7
+reflink_la_SOURCES = reflink.c
2f13d7
+reflink_la_LIBADD = $(top_builddir)/lib/librpm.la $(top_builddir)/rpmio/librpmio.la
2f13d7
+plugins_LTLIBRARIES += reflink.la
2f13d7
+
2f13d7
 syslog_la_SOURCES = syslog.c
2f13d7
 syslog_la_LIBADD = $(top_builddir)/lib/librpm.la $(top_builddir)/rpmio/librpmio.la
2f13d7
 plugins_LTLIBRARIES += syslog.la
2f13d7
diff --git a/plugins/reflink.c b/plugins/reflink.c
2f13d7
new file mode 100644
2f13d7
index 000000000..d7f19acd9
2f13d7
--- /dev/null
2f13d7
+++ b/plugins/reflink.c
2f13d7
@@ -0,0 +1,340 @@
2f13d7
+#include "system.h"
2f13d7
+
2f13d7
+#include <errno.h>
2f13d7
+#include <sys/resource.h>
2f13d7
+#include <unistd.h>
2f13d7
+#include <sys/types.h>
2f13d7
+#include <sys/stat.h>
2f13d7
+#include <fcntl.h>
2f13d7
+#if defined(__linux__)
2f13d7
+#include <linux/fs.h>        /* For FICLONE */
2f13d7
+#endif
2f13d7
+
2f13d7
+#include <rpm/rpmlog.h>
2f13d7
+#include "lib/rpmlib.h"
2f13d7
+#include "lib/rpmplugin.h"
2f13d7
+#include "lib/rpmte_internal.h"
2f13d7
+#include <rpm/rpmfileutil.h>
2f13d7
+#include "rpmio/rpmio_internal.h"
2f13d7
+
2f13d7
+
2f13d7
+#include "debug.h"
2f13d7
+
2f13d7
+#include <sys/ioctl.h>
2f13d7
+
2f13d7
+/* use hash table to remember inode -> ix (for rpmfilesFN(ix)) lookups */
2f13d7
+#undef HASHTYPE
2f13d7
+#undef HTKEYTYPE
2f13d7
+#undef HTDATATYPE
2f13d7
+#define HASHTYPE inodeIndexHash
2f13d7
+#define HTKEYTYPE rpm_ino_t
2f13d7
+#define HTDATATYPE int
2f13d7
+#include "lib/rpmhash.H"
2f13d7
+#include "lib/rpmhash.C"
2f13d7
+
2f13d7
+/*
2f13d7
+We use this in find to indicate a key wasn't found. This is an unrecoverable
2f13d7
+error, but we can at least show a decent error. 0 is never a valid offset
2f13d7
+because it's the offset of the start of the file.
2f13d7
+*/
2f13d7
+#define NOT_FOUND 0
2f13d7
+
2f13d7
+#define BUFFER_SIZE (1024 * 128)
2f13d7
+
2f13d7
+/* magic value at end of file (64 bits) that indicates this is a transcoded rpm */
2f13d7
+#define MAGIC 3472329499408095051
2f13d7
+
2f13d7
+struct reflink_state_s {
2f13d7
+  /* Stuff that's used across rpms */
2f13d7
+  long fundamental_block_size;
2f13d7
+  char *buffer;
2f13d7
+
2f13d7
+  /* stuff that's used/updated per psm */
2f13d7
+  uint32_t keys, keysize;
2f13d7
+
2f13d7
+  // table for current rpm, keys * (keysize + sizeof(rpm_loff_t))
2f13d7
+  unsigned char *table;
2f13d7
+  FD_t fd;
2f13d7
+  rpmfiles files;
2f13d7
+  inodeIndexHash inodeIndexes;
2f13d7
+};
2f13d7
+
2f13d7
+typedef struct reflink_state_s * reflink_state;
2f13d7
+
2f13d7
+static int inodeCmp(rpm_ino_t a, rpm_ino_t b)
2f13d7
+{
2f13d7
+    return (a != b);
2f13d7
+}
2f13d7
+
2f13d7
+static unsigned int inodeId(rpm_ino_t a)
2f13d7
+{
2f13d7
+    /* rpm_ino_t is uint32_t so maps safely to unsigned int */
2f13d7
+    return (unsigned int)a;
2f13d7
+}
2f13d7
+
2f13d7
+static rpmRC reflink_init(rpmPlugin plugin, rpmts ts) {
2f13d7
+  reflink_state state = rcalloc(1, sizeof(struct reflink_state_s));
2f13d7
+
2f13d7
+  /*
2f13d7
+  IOCTL-FICLONERANGE(2): ...Disk filesystems generally require the offset and
2f13d7
+  length arguments to be aligned to the fundamental block size.
2f13d7
+
2f13d7
+  The value of "fundamental block size" is directly related to the system's
2f13d7
+  page size, so we should use that.
2f13d7
+  */
2f13d7
+  state->fundamental_block_size = sysconf(_SC_PAGESIZE);
2f13d7
+  state->buffer = rcalloc(1, BUFFER_SIZE);
2f13d7
+  rpmPluginSetData(plugin, state);
2f13d7
+
2f13d7
+  return RPMRC_OK;
2f13d7
+}
2f13d7
+
2f13d7
+static void reflink_cleanup(rpmPlugin plugin) {
2f13d7
+  reflink_state state = rpmPluginGetData(plugin);
2f13d7
+  free(state->buffer);
2f13d7
+  free(state);
2f13d7
+}
2f13d7
+
2f13d7
+static rpmRC reflink_psm_pre(rpmPlugin plugin, rpmte te) {
2f13d7
+    reflink_state state = rpmPluginGetData(plugin);
2f13d7
+    state->fd = rpmteFd(te);
2f13d7
+    if (state->fd == 0) {
2f13d7
+      rpmlog(RPMLOG_DEBUG, _("reflink: fd = 0, no install\n"));
2f13d7
+      return RPMRC_OK;
2f13d7
+    }
2f13d7
+    rpm_loff_t current = Ftell(state->fd);
2f13d7
+    uint64_t magic;
2f13d7
+    if (Fseek(state->fd, -(sizeof(magic)), SEEK_END) < 0) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: failed to seek for magic\n"));
2f13d7
+      if (Fseek(state->fd, current, SEEK_SET) < 0) {
2f13d7
+        /* yes this gets a bit repetitive */
2f13d7
+        rpmlog(RPMLOG_ERR, _("reflink: unable to seek back to original location\n"));
2f13d7
+      }
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    size_t len = sizeof(magic);
2f13d7
+    if (Fread(&magic, len, 1, state->fd) != len) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: unable to read magic\n"));
2f13d7
+      if (Fseek(state->fd, current, SEEK_SET) < 0) {
2f13d7
+        rpmlog(RPMLOG_ERR, _("reflink: unable to seek back to original location\n"));
2f13d7
+      }
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    if (magic != MAGIC) {
2f13d7
+      rpmlog(RPMLOG_DEBUG, _("reflink: not transcoded\n"));
2f13d7
+      if (Fseek(state->fd, current, SEEK_SET) < 0) {
2f13d7
+        rpmlog(RPMLOG_ERR, _("reflink: unable to seek back to original location\n"));
2f13d7
+        return RPMRC_FAIL;
2f13d7
+      }
2f13d7
+      return RPMRC_OK;
2f13d7
+    }
2f13d7
+    rpmlog(RPMLOG_DEBUG, _("reflink: *is* transcoded\n"));
2f13d7
+    Header h = rpmteHeader(te);
2f13d7
+
2f13d7
+    /* replace/add header that main fsm.c can read */
2f13d7
+    headerDel(h, RPMTAG_PAYLOADFORMAT);
2f13d7
+    headerPutString(h, RPMTAG_PAYLOADFORMAT, "clon");
2f13d7
+    headerFree(h);
2f13d7
+    state->files = rpmteFiles(te);
2f13d7
+    /* tail of file contains offset_table, offset_checksums
2f13d7
+       then magic
2f13d7
+    */
2f13d7
+    if (Fseek(state->fd, -(sizeof(rpm_loff_t) * 2 + sizeof(magic)), SEEK_END) < 0) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: failed to seek for tail %p\n"), state->fd);
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    rpm_loff_t table_start;
2f13d7
+    len = sizeof(table_start);
2f13d7
+    if (Fread(&table_start, len, 1, state->fd) != len) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: unable to read table_start\n"));
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    if (Fseek(state->fd, table_start, SEEK_SET) < 0) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: unable to seek to table_start\n"));
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    len = sizeof(state->keys);
2f13d7
+    if (Fread(&state->keys, len, 1, state->fd) != len) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: unable to read number of keys\n"));
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    len = sizeof(state->keysize);
2f13d7
+    if (Fread(&state->keysize, len, 1, state->fd) != len) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: unable to read keysize\n"));
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    rpmlog(RPMLOG_DEBUG, _("reflink: table_start=0x%lx, keys=%d, keysize=%d\n"), table_start, state->keys, state->keysize);
2f13d7
+    // now get digest table if there is a reason to have one.
2f13d7
+    if (state->keys == 0 || state->keysize == 0) {
2f13d7
+      // no files (or no digests(!))
2f13d7
+      state->table = NULL;
2f13d7
+    } else {
2f13d7
+      int table_size = state->keys * (state->keysize + sizeof(rpm_loff_t));
2f13d7
+      state->table = rcalloc(1, table_size);
2f13d7
+      if (Fread(state->table, table_size, 1, state->fd) != table_size) {
2f13d7
+        rpmlog(RPMLOG_ERR, _("reflink: unable to read table\n"));
2f13d7
+        return RPMRC_FAIL;
2f13d7
+      }
2f13d7
+      state->inodeIndexes = inodeIndexHashCreate(state->keys, inodeId, inodeCmp, NULL, NULL);
2f13d7
+    }
2f13d7
+
2f13d7
+    // seek back to original location
2f13d7
+    // might not be needed if we seek to offset immediately
2f13d7
+    if (Fseek(state->fd, current, SEEK_SET) < 0) {
2f13d7
+      rpmlog(RPMLOG_ERR, _("reflink: unable to seek back to original location\n"));
2f13d7
+      return RPMRC_FAIL;
2f13d7
+    }
2f13d7
+    return RPMRC_OK;
2f13d7
+}
2f13d7
+
2f13d7
+static rpmRC reflink_psm_post(rpmPlugin plugin, rpmte te, int res)
2f13d7
+{
2f13d7
+    reflink_state state = rpmPluginGetData(plugin);
2f13d7
+    state->files = rpmfilesFree(state->files);
2f13d7
+    if (state->table) {
2f13d7
+      free(state->table);
2f13d7
+      state->table = NULL;
2f13d7
+    }
2f13d7
+    if (state->inodeIndexes) {
2f13d7
+      inodeIndexHashFree(state->inodeIndexes);
2f13d7
+      state->inodeIndexes = NULL;
2f13d7
+    }
2f13d7
+    return RPMRC_OK;
2f13d7
+}
2f13d7
+
2f13d7
+
2f13d7
+// have a prototype, warnings system
2f13d7
+rpm_loff_t find(const unsigned char *digest, reflink_state state);
2f13d7
+
2f13d7
+rpm_loff_t find(const unsigned char *digest, reflink_state state) {
2f13d7
+# if defined(__GNUC__)
2f13d7
+  /* GCC nested function because bsearch's comparison function can't access
2f13d7
+     state-keysize otherwise
2f13d7
+  */
2f13d7
+  int cmpdigest(const void *k1, const void *k2) {
2f13d7
+    rpmlog(RPMLOG_DEBUG, _("reflink: cmpdigest k1=%p k2=%p\n"), k1, k2);
2f13d7
+    return memcmp(k1, k2, state->keysize);
2f13d7
+  }
2f13d7
+# endif
2f13d7
+  rpmlog(RPMLOG_DEBUG, _("reflink: bsearch(key=%p, base=%p, nmemb=%d, size=%lu)\n"), digest, state->table, state->keys, state->keysize + sizeof(rpm_loff_t));
2f13d7
+  char *entry = bsearch(digest, state->table, state->keys, state->keysize + sizeof(rpm_loff_t), cmpdigest);
2f13d7
+  if (entry == NULL) {
2f13d7
+    return NOT_FOUND;
2f13d7
+  }
2f13d7
+  rpm_loff_t offset = *(rpm_loff_t *)(entry + state->keysize);
2f13d7
+  return offset;
2f13d7
+}
2f13d7
+
2f13d7
+static rpmRC reflink_fsm_file_pre(rpmPlugin plugin, rpmfi fi, const char* path, mode_t file_mode, rpmFsmOp op)
2f13d7
+{
2f13d7
+    struct file_clone_range fcr;
2f13d7
+    rpm_loff_t size;
2f13d7
+    int dst, rc;
2f13d7
+    int *hlix;
2f13d7
+
2f13d7
+    reflink_state state = rpmPluginGetData(plugin);
2f13d7
+    if (state->table == NULL) {
2f13d7
+        // no table means rpm is not in reflink format, so leave. Now.
2f13d7
+        return RPMRC_OK;
2f13d7
+    }
2f13d7
+    if (op == FA_TOUCH) {
2f13d7
+        // we're not overwriting an existing file
2f13d7
+        return RPMRC_OK;
2f13d7
+    }
2f13d7
+    fcr.dest_offset = 0;
2f13d7
+    if (S_ISREG(file_mode) && !(rpmfiFFlags(fi) & RPMFILE_GHOST)) {
2f13d7
+      rpm_ino_t inode = rpmfiFInode(fi);
2f13d7
+      /* check for hard link entry in table. GetEntry overwrites hlix with the address of the first match */
2f13d7
+      if (inodeIndexHashGetEntry(state->inodeIndexes, inode, &hlix, NULL, NULL)) {
2f13d7
+        // entry is in table, use hard link
2f13d7
+        char *fn = rpmfilesFN(state->files, hlix[0]);
2f13d7
+        if (link(fn, path) != 0) {
2f13d7
+          rpmlog(RPMLOG_ERR, _("reflink: Unable to hard link %s -> %s due to %s\n"), fn, path, strerror(errno));
2f13d7
+          free(fn);
2f13d7
+          return RPMRC_FAIL;
2f13d7
+        }
2f13d7
+        free(fn);
2f13d7
+        return RPMRC_PLUGIN_CONTENTS;
2f13d7
+      }
2f13d7
+      /* if we didn't hard link, then we'll track this inode as being created soon */
2f13d7
+      if (rpmfiFNlink(fi) > 1) {
2f13d7
+        /* minor optimization: only store files with more than one link */
2f13d7
+        inodeIndexHashAddEntry(state->inodeIndexes, inode, rpmfiFX(fi));
2f13d7
+      }
2f13d7
+      /* derived from wfd_open in fsm.c */
2f13d7
+      mode_t old_umask = umask(0577);
2f13d7
+      dst = open(path, O_WRONLY | O_CREAT | O_EXCL, S_IRUSR);
2f13d7
+      umask(old_umask);
2f13d7
+      if (dst == -1) {
2f13d7
+          rpmlog(RPMLOG_ERR, _("reflink: Unable to open %s for writing due to %s, flags = %x\n"), path, strerror(errno), rpmfiFFlags(fi));
2f13d7
+          return RPMRC_FAIL;
2f13d7
+      }
2f13d7
+      size = rpmfiFSize(fi);
2f13d7
+      if (size > 0) {
2f13d7
+          /* round src_length down to fundamental_block_size multiple */
2f13d7
+          fcr.src_length = size / state->fundamental_block_size * state->fundamental_block_size;
2f13d7
+          if ((size % state->fundamental_block_size) > 0) {
2f13d7
+              /* round up to next fundamental_block_size. We expect the data in the rpm to be similarly padded */
2f13d7
+              fcr.src_length += state->fundamental_block_size;
2f13d7
+          }
2f13d7
+          fcr.src_fd = Fileno(state->fd);
2f13d7
+          if (fcr.src_fd == -1) {
2f13d7
+            close(dst);
2f13d7
+            rpmlog(RPMLOG_ERR, _("reflink: src fd lookup failed\n"));
2f13d7
+            return RPMRC_FAIL;
2f13d7
+          }
2f13d7
+          fcr.src_offset = find(rpmfiFDigest(fi, NULL, NULL), state);
2f13d7
+          if (fcr.src_offset == NOT_FOUND) {
2f13d7
+            close(dst);
2f13d7
+            rpmlog(RPMLOG_ERR, _("reflink: digest not found\n"));
2f13d7
+            return RPMRC_FAIL;
2f13d7
+          }
2f13d7
+          rpmlog(RPMLOG_DEBUG, _("reflink: Reflinking %lu bytes at %lu to %s orig size=%lu, file=%ld\n"), fcr.src_length, fcr.src_offset, path, size, fcr.src_fd);
2f13d7
+          rc = ioctl(dst, FICLONERANGE, &fcr;;
2f13d7
+          if (rc) {
2f13d7
+            rpmlog(RPMLOG_WARNING, _("reflink: falling back to copying bits for %s due to %d, %d = %s\n"), path, rc, errno, strerror(errno));
2f13d7
+            if (Fseek(state->fd, fcr.src_offset, SEEK_SET) < 0) {
2f13d7
+                close(dst);
2f13d7
+                rpmlog(RPMLOG_ERR, _("reflink: unable to seek on copying bits\n"));
2f13d7
+                return RPMRC_FAIL;
2f13d7
+            }
2f13d7
+            rpm_loff_t left = size;
2f13d7
+            size_t len, read, written;
2f13d7
+            while (left) {
2f13d7
+              len = (left > BUFFER_SIZE ? BUFFER_SIZE : left);
2f13d7
+              read = Fread(state->buffer, len, 1, state->fd);
2f13d7
+              if (read != len) {
2f13d7
+                close(dst);
2f13d7
+                rpmlog(RPMLOG_ERR, _("reflink: short read on copying bits\n"));
2f13d7
+                return RPMRC_FAIL;
2f13d7
+              }
2f13d7
+              written = write(dst, state->buffer, len);
2f13d7
+              if (read != written) {
2f13d7
+                close(dst);
2f13d7
+                rpmlog(RPMLOG_ERR, _("reflink: short write on copying bits\n"));
2f13d7
+                return RPMRC_FAIL;
2f13d7
+              }
2f13d7
+              left -= len;
2f13d7
+            }
2f13d7
+          } else {
2f13d7
+            /* reflink worked, so truncate */
2f13d7
+            rc = ftruncate(dst, size);
2f13d7
+            if (rc) {
2f13d7
+                rpmlog(RPMLOG_ERR, _("reflink: Unable to truncate %s to %ld due to %s\n"), path, size, strerror(errno));
2f13d7
+                return RPMRC_FAIL;
2f13d7
+            }
2f13d7
+          }
2f13d7
+      }
2f13d7
+      close(dst);
2f13d7
+      return RPMRC_PLUGIN_CONTENTS;
2f13d7
+    }
2f13d7
+    return RPMRC_OK;
2f13d7
+}
2f13d7
+
2f13d7
+struct rpmPluginHooks_s reflink_hooks = {
2f13d7
+    .init = reflink_init,
2f13d7
+    .cleanup = reflink_cleanup,
2f13d7
+    .psm_pre = reflink_psm_pre,
2f13d7
+    .psm_post = reflink_psm_post,
2f13d7
+    .fsm_file_pre = reflink_fsm_file_pre,
2f13d7
+};
2f13d7
diff --git a/rpm2extents.c b/rpm2extents.c
2f13d7
new file mode 100644
2f13d7
index 000000000..5662b86a6
2f13d7
--- /dev/null
2f13d7
+++ b/rpm2extents.c
2f13d7
@@ -0,0 +1,519 @@
2f13d7
+/* rpm2extents: convert payload to inline extents */
2f13d7
+
2f13d7
+#include "system.h"
2f13d7
+
2f13d7
+#include <rpm/rpmlib.h>		/* rpmReadPackageFile .. */
2f13d7
+#include <rpm/rpmfi.h>
2f13d7
+#include <rpm/rpmtag.h>
2f13d7
+#include <rpm/rpmio.h>
2f13d7
+#include <rpm/rpmpgp.h>
2f13d7
+
2f13d7
+#include <rpm/rpmts.h>
2f13d7
+#include "lib/rpmlead.h"
2f13d7
+#include "lib/signature.h"
2f13d7
+#include "lib/header_internal.h"
2f13d7
+#include "rpmio/rpmio_internal.h"
2f13d7
+
2f13d7
+#include <unistd.h>
2f13d7
+#include <sys/types.h>
2f13d7
+#include <sys/wait.h>
2f13d7
+#include <signal.h>
2f13d7
+#include <errno.h>
2f13d7
+#include <string.h>
2f13d7
+
2f13d7
+#include "debug.h"
2f13d7
+
2f13d7
+/* hash of void * (pointers) to file digests to offsets within output.
2f13d7
+   The length of the key depends on what the FILEDIGESTALGO is.
2f13d7
+ */
2f13d7
+#undef HASHTYPE
2f13d7
+#undef HTKEYTYPE
2f13d7
+#undef HTDATATYPE
2f13d7
+#define HASHTYPE digestSet
2f13d7
+#define HTKEYTYPE const unsigned char *
2f13d7
+#include "lib/rpmhash.H"
2f13d7
+#include "lib/rpmhash.C"
2f13d7
+
2f13d7
+/* magic value at end of file (64 bits) that indicates this is a transcoded rpm */
2f13d7
+#define MAGIC 3472329499408095051
2f13d7
+
2f13d7
+struct digestoffset {
2f13d7
+    const unsigned char * digest;
2f13d7
+    rpm_loff_t pos;
2f13d7
+};
2f13d7
+
2f13d7
+rpm_loff_t pad_to(rpm_loff_t pos, rpm_loff_t unit);
2f13d7
+
2f13d7
+rpm_loff_t pad_to(rpm_loff_t pos, rpm_loff_t unit)
2f13d7
+{
2f13d7
+    return (unit - (pos % unit)) % unit;
2f13d7
+}
2f13d7
+
2f13d7
+static int digestor(
2f13d7
+    FD_t fdi,
2f13d7
+    FD_t fdo,
2f13d7
+    FD_t validationo,
2f13d7
+    uint8_t algos[],
2f13d7
+    uint32_t algos_len
2f13d7
+)
2f13d7
+{
2f13d7
+    ssize_t fdilength;
2f13d7
+    const char *filedigest, *algo_name;
2f13d7
+    size_t filedigest_len, len;
2f13d7
+    uint32_t algo_name_len, algo_digest_len;
2f13d7
+    int algo;
2f13d7
+    rpmRC rc = RPMRC_FAIL;
2f13d7
+
2f13d7
+    for (algo = 0; algo < algos_len; algo++)
2f13d7
+    {
2f13d7
+        fdInitDigest(fdi, algos[algo], 0);
2f13d7
+    }
2f13d7
+    fdilength = ufdCopy(fdi, fdo);
2f13d7
+    if (fdilength == -1)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("digest cat failed\n"));
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+
2f13d7
+    len = sizeof(fdilength);
2f13d7
+    if (Fwrite(&fdilength, len, 1, validationo) != len)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write input length %zd\n"), fdilength);
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    len = sizeof(algos_len);
2f13d7
+    if (Fwrite(&algos_len, len, 1, validationo) != len)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write number of validation digests\n"));
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    for (algo = 0; algo < algos_len; algo++)
2f13d7
+    {
2f13d7
+        fdFiniDigest(fdi, algos[algo], (void **)&filedigest, &filedigest_len, 0);
2f13d7
+
2f13d7
+        algo_name = pgpValString(PGPVAL_HASHALGO, algos[algo]);
2f13d7
+        algo_name_len = (uint32_t)strlen(algo_name);
2f13d7
+        algo_digest_len = (uint32_t)filedigest_len;
2f13d7
+
2f13d7
+        len = sizeof(algo_name_len);
2f13d7
+        if (Fwrite(&algo_name_len, len, 1, validationo) != len)
2f13d7
+        {
2f13d7
+            fprintf(
2f13d7
+                stderr,
2f13d7
+                _("Unable to write validation algo name length\n")
2f13d7
+            );
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+        len = sizeof(algo_digest_len);
2f13d7
+        if (Fwrite(&algo_digest_len, len, 1, validationo) != len)
2f13d7
+        {
2f13d7
+            fprintf(
2f13d7
+                stderr,
2f13d7
+                _("Unable to write number of bytes for validation digest\n")
2f13d7
+            );
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+        if (Fwrite(algo_name, algo_name_len, 1, validationo) != algo_name_len)
2f13d7
+        {
2f13d7
+            fprintf(stderr, _("Unable to write validation algo name\n"));
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+        if (
2f13d7
+            Fwrite(
2f13d7
+                filedigest,
2f13d7
+                algo_digest_len,
2f13d7
+                1,
2f13d7
+                validationo
2f13d7
+            ) != algo_digest_len
2f13d7
+        )
2f13d7
+        {
2f13d7
+            fprintf(
2f13d7
+                stderr,
2f13d7
+                _("Unable to write validation digest value %u, %zu\n"),
2f13d7
+                algo_digest_len,
2f13d7
+                filedigest_len
2f13d7
+            );
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+    }
2f13d7
+    rc = RPMRC_OK;
2f13d7
+exit:
2f13d7
+    return rc;
2f13d7
+}
2f13d7
+
2f13d7
+static rpmRC process_package(FD_t fdi, FD_t validationi)
2f13d7
+{
2f13d7
+    uint32_t diglen;
2f13d7
+    /* GNU C extension: can use diglen from outer context */
2f13d7
+    int digestSetCmp(const unsigned char * a, const unsigned char * b)
2f13d7
+    {
2f13d7
+        return memcmp(a, b, diglen);
2f13d7
+    }
2f13d7
+
2f13d7
+    unsigned int digestSetHash(const unsigned char * digest)
2f13d7
+    {
2f13d7
+        /* assumes sizeof(unsigned int) < diglen */
2f13d7
+        return *(unsigned int *)digest;
2f13d7
+    }
2f13d7
+
2f13d7
+    int digestoffsetCmp(const void * a, const void * b)
2f13d7
+    {
2f13d7
+        return digestSetCmp(
2f13d7
+            ((struct digestoffset *)a)->digest,
2f13d7
+            ((struct digestoffset *)b)->digest
2f13d7
+        );
2f13d7
+    }
2f13d7
+
2f13d7
+    FD_t fdo;
2f13d7
+    FD_t gzdi;
2f13d7
+    Header h, sigh;
2f13d7
+    long fundamental_block_size = sysconf(_SC_PAGESIZE);
2f13d7
+    rpmRC rc = RPMRC_OK;
2f13d7
+    rpm_mode_t mode;
2f13d7
+    char *rpmio_flags = NULL, *zeros;
2f13d7
+    const unsigned char *digest;
2f13d7
+    rpm_loff_t pos, size, pad, validation_pos;
2f13d7
+    uint32_t offset_ix = 0;
2f13d7
+    size_t len;
2f13d7
+    int next = 0;
2f13d7
+
2f13d7
+    fdo = fdDup(STDOUT_FILENO);
2f13d7
+
2f13d7
+    if (rpmReadPackageRaw(fdi, &sigh, &h))
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Error reading package\n"));
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+
2f13d7
+    if (rpmLeadWrite(fdo, h))
2f13d7
+    {
2f13d7
+        fprintf(
2f13d7
+            stderr,
2f13d7
+            _("Unable to write package lead: %s\n"),
2f13d7
+            Fstrerror(fdo)
2f13d7
+        );
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+
2f13d7
+    if (rpmWriteSignature(fdo, sigh))
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write signature: %s\n"), Fstrerror(fdo));
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+
2f13d7
+    if (headerWrite(fdo, h, HEADER_MAGIC_YES))
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write headers: %s\n"), Fstrerror(fdo));
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+
2f13d7
+    /* Retrieve payload size and compression type. */
2f13d7
+    {	const char *compr = headerGetString(h, RPMTAG_PAYLOADCOMPRESSOR);
2f13d7
+        rpmio_flags = rstrscat(NULL, "r.", compr ? compr : "gzip", NULL);
2f13d7
+    }
2f13d7
+
2f13d7
+    gzdi = Fdopen(fdi, rpmio_flags);	/* XXX gzdi == fdi */
2f13d7
+    free(rpmio_flags);
2f13d7
+
2f13d7
+    if (gzdi == NULL)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("cannot re-open payload: %s\n"), Fstrerror(gzdi));
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+
2f13d7
+    rpmfiles files = rpmfilesNew(NULL, h, 0, RPMFI_KEEPHEADER);
2f13d7
+    rpmfi fi = rpmfiNewArchiveReader(
2f13d7
+        gzdi,
2f13d7
+        files,
2f13d7
+        RPMFI_ITER_READ_ARCHIVE_CONTENT_FIRST
2f13d7
+    );
2f13d7
+
2f13d7
+    /* this is encoded in the file format, so needs to be fixed size (for
2f13d7
+        now?)
2f13d7
+    */
2f13d7
+    diglen = (uint32_t)rpmDigestLength(rpmfiDigestAlgo(fi));
2f13d7
+    digestSet ds = digestSetCreate(
2f13d7
+        rpmfiFC(fi),
2f13d7
+        digestSetHash,
2f13d7
+        digestSetCmp,
2f13d7
+        NULL
2f13d7
+    );
2f13d7
+    struct digestoffset offsets[rpmfiFC(fi)];
2f13d7
+    pos = RPMLEAD_SIZE + headerSizeof(sigh, HEADER_MAGIC_YES);
2f13d7
+
2f13d7
+    /* main headers are aligned to 8 byte boundry */
2f13d7
+    pos += pad_to(pos, 8);
2f13d7
+    pos += headerSizeof(h, HEADER_MAGIC_YES);
2f13d7
+
2f13d7
+    zeros = xcalloc(fundamental_block_size, 1);
2f13d7
+
2f13d7
+    while (next >= 0)
2f13d7
+    {
2f13d7
+        next = rpmfiNext(fi);
2f13d7
+        if (next == RPMERR_ITER_END)
2f13d7
+        {
2f13d7
+            rc = RPMRC_OK;
2f13d7
+            break;
2f13d7
+        }
2f13d7
+        mode = rpmfiFMode(fi);
2f13d7
+        if (!S_ISREG(mode) || !rpmfiArchiveHasContent(fi))
2f13d7
+        {
2f13d7
+            /* not a regular file, or the archive doesn't contain any content for
2f13d7
+               this entry
2f13d7
+            */
2f13d7
+            continue;
2f13d7
+        }
2f13d7
+        digest = rpmfiFDigest(fi, NULL, NULL);
2f13d7
+        if (digestSetGetEntry(ds, digest, NULL))
2f13d7
+        {
2f13d7
+            /* This specific digest has already been included, so skip it */
2f13d7
+            continue;
2f13d7
+        }
2f13d7
+        pad = pad_to(pos, fundamental_block_size);
2f13d7
+        if (Fwrite(zeros, sizeof(char), pad, fdo) != pad)
2f13d7
+        {
2f13d7
+            fprintf(stderr, _("Unable to write padding\n"));
2f13d7
+            rc = RPMRC_FAIL;
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+        /* round up to next fundamental_block_size */
2f13d7
+        pos += pad;
2f13d7
+        digestSetAddEntry(ds, digest);
2f13d7
+        offsets[offset_ix].digest = digest;
2f13d7
+        offsets[offset_ix].pos = pos;
2f13d7
+        offset_ix++;
2f13d7
+        size = rpmfiFSize(fi);
2f13d7
+        rc = rpmfiArchiveReadToFile(fi, fdo, 0);
2f13d7
+        if (rc != RPMRC_OK)
2f13d7
+        {
2f13d7
+            fprintf(stderr, _("rpmfiArchiveReadToFile failed with %d\n"), rc);
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+        pos += size;
2f13d7
+    }
2f13d7
+    Fclose(gzdi);	/* XXX gzdi == fdi */
2f13d7
+
2f13d7
+    qsort(
2f13d7
+        offsets,
2f13d7
+        (size_t)offset_ix,
2f13d7
+        sizeof(struct digestoffset),
2f13d7
+        digestoffsetCmp
2f13d7
+    );
2f13d7
+
2f13d7
+    len = sizeof(offset_ix);
2f13d7
+    if (Fwrite(&offset_ix, len, 1, fdo) != len)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write length of table\n"));
2f13d7
+        rc = RPMRC_FAIL;
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    len = sizeof(diglen);
2f13d7
+    if (Fwrite(&diglen, len, 1, fdo) != len)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write length of digest\n"));
2f13d7
+        rc = RPMRC_FAIL;
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    len = sizeof(rpm_loff_t);
2f13d7
+    for (int x = 0; x < offset_ix; x++)
2f13d7
+    {
2f13d7
+        if (Fwrite(offsets[x].digest, diglen, 1, fdo) != diglen)
2f13d7
+        {
2f13d7
+            fprintf(stderr, _("Unable to write digest\n"));
2f13d7
+            rc = RPMRC_FAIL;
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+        if (Fwrite(&offsets[x].pos, len, 1, fdo) != len)
2f13d7
+        {
2f13d7
+            fprintf(stderr, _("Unable to write offset\n"));
2f13d7
+            rc = RPMRC_FAIL;
2f13d7
+            goto exit;
2f13d7
+        }
2f13d7
+    }
2f13d7
+    validation_pos = (
2f13d7
+        pos + sizeof(offset_ix) + sizeof(diglen) +
2f13d7
+        offset_ix * (diglen + sizeof(rpm_loff_t))
2f13d7
+    );
2f13d7
+
2f13d7
+    ssize_t validation_len = ufdCopy(validationi, fdo);
2f13d7
+    if (validation_len == -1)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("digest table ufdCopy failed\n"));
2f13d7
+        rc = RPMRC_FAIL;
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    /* add more padding so the last file can be cloned. It doesn't matter that
2f13d7
+       the table and validation etc are in this space. In fact, it's pretty
2f13d7
+       efficient if it is
2f13d7
+    */
2f13d7
+
2f13d7
+    pad = pad_to(
2f13d7
+        (
2f13d7
+            validation_pos + validation_len + 2 * sizeof(rpm_loff_t) +
2f13d7
+            sizeof(uint64_t)
2f13d7
+        ),
2f13d7
+        fundamental_block_size
2f13d7
+    );
2f13d7
+    if (Fwrite(zeros, sizeof(char), pad, fdo) != pad)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write final padding\n"));
2f13d7
+        rc = RPMRC_FAIL;
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    zeros = _free(zeros);
2f13d7
+    if (Fwrite(&pos, len, 1, fdo) != len)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write offset of digest table\n"));
2f13d7
+        rc = RPMRC_FAIL;
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    if (Fwrite(&validation_pos, len, 1, fdo) != len)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write offset of validation table\n"));
2f13d7
+        rc = RPMRC_FAIL;
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+    uint64_t magic = MAGIC;
2f13d7
+    len = sizeof(magic);
2f13d7
+    if (Fwrite(&magic, len, 1, fdo) != len)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Unable to write magic\n"));
2f13d7
+        rc = RPMRC_FAIL;
2f13d7
+        goto exit;
2f13d7
+    }
2f13d7
+
2f13d7
+exit:
2f13d7
+    rpmfilesFree(files);
2f13d7
+    rpmfiFree(fi);
2f13d7
+    headerFree(h);
2f13d7
+    return rc;
2f13d7
+}
2f13d7
+
2f13d7
+int main(int argc, char *argv[])
2f13d7
+{
2f13d7
+    rpmRC rc;
2f13d7
+    int cprc = 0;
2f13d7
+    uint8_t algos[argc - 1];
2f13d7
+    int mainpipefd[2];
2f13d7
+    int metapipefd[2];
2f13d7
+    pid_t cpid, w;
2f13d7
+    int wstatus;
2f13d7
+
2f13d7
+    xsetprogname(argv[0]);	/* Portability call -- see system.h */
2f13d7
+    rpmReadConfigFiles(NULL, NULL);
2f13d7
+
2f13d7
+    if (argc > 1 && (rstreq(argv[1], "-h") || rstreq(argv[1], "--help")))
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Usage: %s [DIGESTALGO]...\n"), argv[0]);
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+
2f13d7
+    if (argc == 1)
2f13d7
+    {
2f13d7
+        fprintf(
2f13d7
+            stderr,
2f13d7
+            _("Need at least one DIGESTALGO parameter, e.g. 'SHA256'\n")
2f13d7
+        );
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+
2f13d7
+    for (int x = 0; x < (argc - 1); x++)
2f13d7
+    {
2f13d7
+        if (pgpStringVal(PGPVAL_HASHALGO, argv[x + 1], &algos[x]) != 0)
2f13d7
+        {
2f13d7
+            fprintf(
2f13d7
+                stderr,
2f13d7
+                _("Unable to resolve '%s' as a digest algorithm, exiting\n"),
2f13d7
+                argv[x + 1]
2f13d7
+            );
2f13d7
+            exit(EXIT_FAILURE);
2f13d7
+        }
2f13d7
+    }
2f13d7
+
2f13d7
+
2f13d7
+    if (pipe(mainpipefd) == -1)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Main pipe failure\n"));
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+    if (pipe(metapipefd) == -1)
2f13d7
+    {
2f13d7
+        fprintf(stderr, _("Meta pipe failure\n"));
2f13d7
+        exit(EXIT_FAILURE);
2f13d7
+    }
2f13d7
+    cpid = fork();
2f13d7
+    if (cpid == 0)
2f13d7
+    {
2f13d7
+        /* child: digestor */
2f13d7
+        close(mainpipefd[0]);
2f13d7
+        close(metapipefd[0]);
2f13d7
+        FD_t fdi = fdDup(STDIN_FILENO);
2f13d7
+        FD_t fdo = fdDup(mainpipefd[1]);
2f13d7
+        FD_t validationo = fdDup(metapipefd[1]);
2f13d7
+        rc = digestor(fdi, fdo, validationo, algos, argc - 1);
2f13d7
+        Fclose(validationo);
2f13d7
+        Fclose(fdo);
2f13d7
+        Fclose(fdi);
2f13d7
+    } else {
2f13d7
+        /* parent: main program */
2f13d7
+        close(mainpipefd[1]);
2f13d7
+        close(metapipefd[1]);
2f13d7
+        FD_t fdi = fdDup(mainpipefd[0]);
2f13d7
+        FD_t validationi = fdDup(metapipefd[0]);
2f13d7
+        rc = process_package(fdi, validationi);
2f13d7
+        Fclose(validationi);
2f13d7
+        /* fdi is normally closed through the stacked file gzdi in the function. */
2f13d7
+        /* wait for child process (digestor for stdin) to complete. */
2f13d7
+        if (rc != RPMRC_OK)
2f13d7
+        {
2f13d7
+            if (kill(cpid, SIGTERM) != 0)
2f13d7
+            {
2f13d7
+                fprintf(
2f13d7
+                    stderr,
2f13d7
+                    _("Failed to kill digest process when main process failed: %s\n"),
2f13d7
+                    strerror(errno)
2f13d7
+                );
2f13d7
+            }
2f13d7
+        }
2f13d7
+        w = waitpid(cpid, &wstatus, 0);
2f13d7
+        if (w == -1)
2f13d7
+        {
2f13d7
+            fprintf(stderr, _("waitpid failed\n"));
2f13d7
+            cprc = EXIT_FAILURE;
2f13d7
+        } else if (WIFEXITED(wstatus))
2f13d7
+        {
2f13d7
+            cprc = WEXITSTATUS(wstatus);
2f13d7
+            if (cprc != 0)
2f13d7
+            {
2f13d7
+                fprintf(
2f13d7
+                    stderr,
2f13d7
+                    _("Digest process non-zero exit code %d\n"),
2f13d7
+                    cprc
2f13d7
+                );
2f13d7
+            }
2f13d7
+        } else if (WIFSIGNALED(wstatus))
2f13d7
+        {
2f13d7
+            fprintf(
2f13d7
+                stderr,
2f13d7
+                _("Digest process was terminated with a signal: %d\n"),
2f13d7
+                WTERMSIG(wstatus)
2f13d7
+            );
2f13d7
+            cprc = EXIT_FAILURE;
2f13d7
+        } else
2f13d7
+        {
2f13d7
+            /* don't think this can happen, but covering all bases */
2f13d7
+            fprintf(stderr, _("Unhandled circumstance in waitpid\n"));
2f13d7
+            cprc = EXIT_FAILURE;
2f13d7
+        }
2f13d7
+        if (cprc != EXIT_SUCCESS)
2f13d7
+        {
2f13d7
+            rc = RPMRC_FAIL;
2f13d7
+        }
2f13d7
+    }
2f13d7
+    if (rc != RPMRC_OK)
2f13d7
+    {
2f13d7
+        /* translate rpmRC into generic failure return code. */
2f13d7
+        return EXIT_FAILURE;
2f13d7
+    }
2f13d7
+    return EXIT_SUCCESS;
2f13d7
+}
2f13d7
diff --git a/rpmio/rpmpgp.c b/rpmio/rpmpgp.c
2f13d7
index 015c15a5c..7b972b4a6 100644
2f13d7
--- a/rpmio/rpmpgp.c
2f13d7
+++ b/rpmio/rpmpgp.c
2f13d7
@@ -283,6 +283,16 @@ int pgpValTok(pgpValTbl vs, const char * s, const char * se)
2f13d7
     return vs->val;
2f13d7
 }
2f13d7
 
2f13d7
+int pgpStringVal(pgpValType type, const char *str, uint8_t *val)
2f13d7
+{
2f13d7
+    pgpValTbl tbl = pgpValTable(type);
2f13d7
+    if (tbl == NULL) return -1;
2f13d7
+    int v = pgpValTok(tbl, str, str + strlen(str));
2f13d7
+    if (v == -1) return -1;
2f13d7
+    *val = (uint8_t)v;
2f13d7
+    return 0;
2f13d7
+}
2f13d7
+
2f13d7
 /** \ingroup rpmpgp
2f13d7
  * Decode length from 1, 2, or 5 octet body length encoding, used in
2f13d7
  * new format packet headers and V4 signature subpackets.
2f13d7
diff --git a/rpmio/rpmpgp.h b/rpmio/rpmpgp.h
2f13d7
index c53e29b01..2b57318ba 100644
2f13d7
--- a/rpmio/rpmpgp.h
2f13d7
+++ b/rpmio/rpmpgp.h
2f13d7
@@ -973,6 +973,15 @@ typedef rpmFlags rpmDigestFlags;
2f13d7
  */
2f13d7
 const char * pgpValString(pgpValType type, uint8_t val);
2f13d7
 
2f13d7
+/** \ingroup rpmpgp
2f13d7
+ * Return  OpenPGP value for a string.
2f13d7
+ * @param type		type of value
2f13d7
+ * @param str		string to lookup
2f13d7
+ * @param[out] val  byte value associated with string
2f13d7
+ * @return		0 on success else -1
2f13d7
+ */
2f13d7
+int pgpStringVal(pgpValType type, const char *str, uint8_t *val);
2f13d7
+
2f13d7
 /** \ingroup rpmpgp
2f13d7
  * Return (native-endian) integer from big-endian representation.
2f13d7
  * @param s		pointer to big-endian integer
2f13d7
-- 
2f13d7
2.35.1
2f13d7