Original pull request on github that was decline as centos 8 is EOL and 4.14.x branch is not going to have new releases https://github.com/rpm-software-management/rpm/pull/2130
Disclaimer: first time patching RPMs and working with mock/chroot, I may be doing something dumb. Let me know. any help is appreciated.
zstd supports multi-threaded compression and is already available in rpm 4.17.
Unfortunately, this feature is not available on centos 8, that has 4.14.x only.
Adding this feature requires upgrading dependency on zstd lib and backporting the intergration.
Changes:
- Backport of the Support threading for zstd compression PR
- Backport of bug fix fa97556
I couldn't backport autodetection for the number of threads as the getncpus macro is not supported in 4.14.x. I replaced it with sysconf(_SC_NPROCESSORS_ONLN)
that is already used in xz.
- 4.14.x will require libzstd version >= 1.3.8.
I ran mock with -r centos-stream-hyperscale-experimental-8-x86_64
as the main branch wasn't working and was missing tons of dependencies. the experimental repo has zstd 1.4.4, which doesn't have multithreading (MT) enabled by default. The library should be rebuild with a macro to enable MT. zst 1.5.0 has MT enabled by default tho and it's available in centos-stream-hyperscale-8-x86_64
, so I decided to include the main repo in my mock configs and install zst 1.5.1 instead.
➜ rpm git:(c8s-sig-hyperscale) ✗ cat /etc/mock/centos-stream-hyperscale-experimental-8-x86_64.cfg include('templates/centos-stream-8.tpl') include('templates/epel-8.tpl') include('templates/epel-next-8.tpl') include('templates/centos-stream-hyperscale-8.tpl') include('templates/centos-stream-hyperscale-experimental-8.tpl') config_opts['root'] = 'centos-stream-hyperscale-experimental-8-x86_64' config_opts['target_arch'] = 'x86_64' config_opts['legal_host_arches'] = ('x86_64',) config_opts['macros']['%dist'] = ".hsx.el8" config_opts['macros']['%centos_hs'] = "1"
Then I updated the SPECS/rpm.spec
locally to require zstd 1.5.0+
Requires: zstd >= 1.3.8
I also tested the PR without these changes, MT wasn't working and I was getting an warning that zstd doesn't not support multi-threading
and falling back to the single-threaded mode.
What I tested:
- Ran mock -r centos-stream-hyperscale-experimental-8-x86_64 --sources ./SOURCES --spec ./SPECS/*.spec --postinstall
with my patch that finished successfully
- Ran mock -r centos-stream-hyperscale-experimental-8-x86_64 --sources ./SOURCES --spec ./SPECS/*.spec --shell
to get inside the chroot
- Set up a basic rpm build inside the chroot with one 200MB file in SOURCES generated with head -c 200MB </dev/urandom >myfile
and put /usr/bin/podman
in the test rpm too to add more weigh (300 MB). So a test rpm build ~500MB.
- Ran with default compressor
<mock-chroot> sh-4.4# time -v rpmbuild -bb -v SPECS/hello.spec
...
Wrote: /builddir/build/RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.Lc2gUG
+ umask 022
+ cd /builddir/build/BUILD
+ cd hello-1.0
+ rm -rf /builddir/build/BUILDROOT/hello-1.0-1.hsx.el8.x86_64
+ exit 0
Command being timed: "rpmbuild -bb -v SPECS/hello.spec"
User time (seconds): 124.53
System time (seconds): 2.66
Percent of CPU this job got: 96%
Elapsed (wall clock) time (h🇲🇲ss or m:ss): 2:12.12
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 41896
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 3
Minor (reclaiming a frame) page faults: 55327
Voluntary context switches: 7890
Involuntary context switches: 17473
Swaps: 0
File system inputs: 0
File system outputs: 1500824
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
<mock-chroot> sh-4.4# du -sh RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
202M RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
w19T0.zstdio
<mock-chroot> sh-4.4# time -v rpmbuild --define '_binary_payload w19T0.zstdio' -bb -v SPECS/hello.spec
...
Wrote: /builddir/build/RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.l3HgpH
+ umask 022
+ cd /builddir/build/BUILD
+ cd hello-1.0
+ rm -rf /builddir/build/BUILDROOT/hello-1.0-1.hsx.el8.x86_64
+ exit 0
Command being timed: "rpmbuild --define _binary_payload w19T0.zstdio -bb -v SPECS/hello.spec"
User time (seconds): 130.39
System time (seconds): 2.86
Percent of CPU this job got: 158%
Elapsed (wall clock) time (h🇲🇲ss or m:ss): 1:23.94
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 449096
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 158357
Voluntary context switches: 8363
Involuntary context switches: 28262
Swaps: 0
File system inputs: 0
File system outputs: 1499936
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
<mock-chroot> sh-4.4# du -sh RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
202M RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
<mock-chroot> sh-4.4# time -v rpmbuild --define '_binary_payload w19.zstdio' -bb -v SPECS/hello.spec
...
+ rm -rf /builddir/build/BUILDROOT/hello-1.0-1.hsx.el8.x86_64
+ exit 0
Command being timed: "rpmbuild --define _binary_payload w19.zstdio -bb -v SPECS/hello.spec"
User time (seconds): 106.24
System time (seconds): 2.23
Percent of CPU this job got: 94%
Elapsed (wall clock) time (h🇲🇲ss or m:ss): 1:54.48
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 112264
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 74180
Voluntary context switches: 7649
Involuntary context switches: 15383
Swaps: 0
File system inputs: 0
File system outputs: 1499872
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
<mock-chroot> sh-4.4# du -sh RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
202M RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
<mock-chroot> sh-4.4# time -v rpmbuild --define '_binary_payload w19T1.zstdio' -bb -v SPECS/hello.spec
...
Wrote: /builddir/build/RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.Hyyqzy
+ umask 022
+ cd /builddir/build/BUILD
+ cd hello-1.0
+ rm -rf /builddir/build/BUILDROOT/hello-1.0-1.hsx.el8.x86_64
+ exit 0
Command being timed: "rpmbuild --define _binary_payload w19T1.zstdio -bb -v SPECS/hello.spec"
User time (seconds): 118.46
System time (seconds): 2.42
Percent of CPU this job got: 96%
Elapsed (wall clock) time (h🇲🇲ss or m:ss): 2:04.70
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 300452
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 121284
Voluntary context switches: 8408
Involuntary context switches: 15759
Swaps: 0
File system inputs: 0
File system outputs: 1499944
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
<mock-chroot> sh-4.4# du -sh RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
202M RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
<mock-chroot> sh-4.4# time -v rpmbuild --define '_binary_payload w19T4.zstdio' -bb -v SPECS/hello.spec
...
Wrote: /builddir/build/RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.hgAtHo
+ umask 022
+ cd /builddir/build/BUILD
+ cd hello-1.0
+ rm -rf /builddir/build/BUILDROOT/hello-1.0-1.hsx.el8.x86_64
+ exit 0
Command being timed: "rpmbuild --define _binary_payload w19T4.zstdio -bb -v SPECS/hello.spec"
User time (seconds): 132.98
System time (seconds): 3.33
Percent of CPU this job got: 159%
Elapsed (wall clock) time (h🇲🇲ss or m:ss): 1:25.38
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 745580
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 3
Minor (reclaiming a frame) page faults: 232525
Voluntary context switches: 8169
Involuntary context switches: 43975
Swaps: 0
File system inputs: 53208
File system outputs: 1499944
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
<mock-chroot> sh-4.4# du -sh RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
202M RPMS/hello-1.0-1.hsx.el8.x86_64.rpm
I also unpacked the rpm and compared the files' md5s with the original files' md5s - they are the same
# unpacking <mock-chroot> sh-4.4# rpm2cpio hello-1.0-1.hsx.el8.x86_64.rpm | cpio -idmv ./usr/bin/hello-world.sh ./usr/bin/myfile ./usr/bin/podman ./usr/lib/.build-id ./usr/lib/.build-id/ad ./usr/lib/.build-id/ad/f500a8beab5bcb6b66a5fa6d7497517b79468b 467002 blocks <mock-chroot> sh-4.4# md5sum ./usr/bin/myfile 88ebe22a347d1c3e99603cb6811b5881 ./usr/bin/myfile <mock-chroot> sh-4.4# md5sum ./usr/bin/podman 4520b424e2c11b78973ff2b8249c46f6 ./usr/bin/podman ... # original files ➜ rpmbuild pwd /home/alex/rpmbuild ➜ rpmbuild cd SOURCES/hello-1.0 ➜ hello-1.0 md5sum * b37bd98e4077dc920344c0dcacc8ed4d hello-world.sh 88ebe22a347d1c3e99603cb6811b5881 myfile 4520b424e2c11b78973ff2b8249c46f6 podman
new api to use multi-threaded compression since this version