===============================================================================


July 2022
===============================================================================


Reference
=========


 - Intel® Communications Chipset 4xxx Series Software for Linux*
   Getting Started Guide

===============================================================================


Overview
=========

Intel® QuickAssist Technology provides security and compression acceleration
capabilities used to improve performance and efficiency across the data center.

Previously existing build system has been replaced by Autotools solution
in order to provide with the widely known and easy to use tool to build and
install source packages.

===============================================================================


Licensing
=========

This product is released under the BSD-3-Clause License.

Files within this project have various inbound licences, listed below:
        - Dual BSD/GPLv2 License
        - BSD License

This package also links against files with the following licenses:
        - OpenSSL License
        - ZLIB License

For BSD-3-Clause license, please see the file LICENSE contained in the top
level folder.
For Dual BSD/GPLv2 please see the file headers of the relevant files.

===============================================================================


Using the QATlib package
========================

    Note: Instructions assume running on Fedora and as non-root user who can
    get root access using sudo, i.e. user is in sudoers group. Command-lines
    may be different in other Linux distributions.


Check System Prerequisites
==========================

    There are some prerequisites to running the software in this package. If
    running from a distro many of these are taken care of, if not here they
    are:

     * platform must have an Intel® Communications Chipset 4xxx Series device
        Use "lspci -d 8086:4940" to check PF devices exist.
        Use "lspci -d 8086:4941" to check VF devices exist.
     * OR platform must have an Intel® 401xx series device
        Use "lspci -d 8086:4942" to check PF devices exist.
        Use "lspci -d 8086:4943" to check VF devices exist.
        VFs are only visible if all prerequisites below are satisfied.
     * firmware must be available
        Check that these files exist, if not contact qat-linux@intel.com
        /lib/firmware/qat_4xxx.bin
        /lib/firmware/qat_4xxx_mmp.bin
     * kernel driver must be running
        Use "lsmod | grep qat" to check that these kernel modules are running:
        intel_qat
        qat_4xxx
        They should load by default if using any of the following:
         * Linux kernel v5.11+ (This is for crypto, for compression use v5.17+)
         * Fedora 34+ (for compression use 36+)
         * RHEL 8.4+ (for compression use 9.0+)
     * BIOS settings
        Intel VT-d and SR-IOV must be enabled in the platform BIOS.
        Consult your platform guide on how to do this.
        If using an Intel BKC these usually default to on, you can verify by
        rebooting, entering F2 on the console to get to the BIOS menus and
        checking these are enabled:
        EDKII Menu
          -> Socket Configuration
           -> IIO Configuration
            -> Intel VT for Directed I/O (VT-d)
             -> Intel VT for Directed I/O
        EDKII Menu
         -> Platform Configuration
          -> Miscellaneous Configuration
           -> SR-IOV Support
     * Grub settings
        set intel_iommu on in grub file, e.g. in Fedora:
         * sudo vi /etc/default/grub
         * to GRUB_CMDLINE_LINUX add "intel_iommu=on"
         * sudo grub2-mkconfig -o /etc/grub2-efi.cfg


===============================================================================


Compilation and installation - quickstart instructions
======================================================

    Note, more detailed instructions in following section.

    ---------------------------------------------------------------------------

    Summary if running from Fedora 34+:
        # Install qatlib
        sudo dnf install -y qatlib

        # Add your user to qat group
        sudo usermod -a -G qat `whoami`

        # Enable qat service and make persistent after reboot
        sudo systemctl enable qat
        sudo systemctl start qat

        # Increase amount of locked memory for your user
        # Note: cpa_sample_code requires a mimimum of 500MB to run compression tests.
        # To add 500MB:
        sudo cp /etc/security/limits.conf /etc/security/limits.conf.qatlib_bak
        echo `whoami` - memlock 500000  | sudo tee -a /etc/security/limits.conf > /dev/null

        # Re-login in order to update the user's group
        sudo su -l $USER

        # If you want to run sample code you need to build from sources, these
        # steps will download qatlib source rpm, extract it and build sample code.
        # Install dependencies
        sudo dnf install -y gcc systemd-devel automake autoconf libtool
        sudo dnf install -y openssl-devel zlib-devel

        # Install qatlib-tests rpm
        sudo dnf install -y qatlib-tests

        # Run it! (takes several minutes to complete)
        cpa_sample_code

    ----------------------------------------------------------------------------

    Summary if building from sources.

        # Install dependencies
        sudo dnf install -y gcc systemd-devel automake autoconf libtool
        sudo dnf install -y openssl-devel zlib-devel

        # Clone QATlib into ~/qatlib, i.e. in your home dir
        cd ~
        git clone https://github.com/intel/qatlib.git

        # Build and install library
        cd qatlib
        ./autogen.sh
        ./configure --enable-service
        make -j
        sudo make install

        # Add your user to the "qat" group which was automatically
        # created by --enable-service
        sudo usermod -a -G qat `whoami`

        # Increase amount of locked memory for your user
        # Note: cpa_sample_code requires a mimimum of 500MB to run compression tests.
        # To add 500MB:
        sudo cp /etc/security/limits.conf /etc/security/limits.conf.qatlib_bak
        echo `whoami` - memlock 500000  | sudo tee -a /etc/security/limits.conf > /dev/null

        # Re-login in order to update the user group, this will move you back
        # into your home dir
        sudo su -l $USER

        # Compression sample code expects to find data files at a known location,
        # so call the samples-install target to put them there
        sudo make samples-install

        # Run! (takes several minutes to complete)
        cpa_sample_code

        # no need to leave the samples installed, so cleanup
        sudo make samples-uninstall

===============================================================================


Compilation and installation - detailed instructions
====================================================

    1) Install compilation dependencies

        If running from a distro most or all of QATlib depends are taken care
        of, if not use your OS-specific commands to install the following:

        tools:
            gcc
            make
            autotools (automake, autoconf, libtool)
            systemd-devel

        libraries:
            openssl-devel
            zlib-devel

    ----------------------------------------------------------------------------

    2) Download package and configure it

        To generate configure script - call:
            ./autogen.sh

        To setup the default build configuration - call:
            ./configure

        Full list of configuration options can be found under
        "Configuration options" section below or by calling:
            ./configure --help

        To setup configuration that will automatically enable qat service
        during installation - call:
        ./configure --enable-service

    ----------------------------------------------------------------------------

    3) Configure the system

        Create the "qat" group.
            qat.service requires "qat" group
            This is automatically created by configure --enable-service so this
            step can be skipped. To manually create the group:
                sudo groupadd qat

        Add your user to "qat" group.
            To be able to use QATlib functionalities as non root user, user must
            be part of the qat group. To add your user to the qat group
                sudo usermod -a -G qat <your user>

        Set maximum amount of locked memory for your user.
            The maximum amount of locked memory should be set correctly by
            defining memlock limit. For cpa_sample_code running crypto at least
            200MB are required, if running compression set at least 500MB.
            To set 500MB add a line like this
            <your user> - memlock 500000
            in the file /etc/security/limits.conf.

        After making user changes it's necessary to re-login for them to
        take effect
            sudo su -l $USER

    ----------------------------------------------------------------------------

    4) Compile and install the library

        Build and install:
            make -j
            sudo make install

        If the service was not configured to start automatically, by using
        --enable-service then it will need to be started. To start it
        and make persistent after reboot:
            systemctl enable qat
            systemctl start qat

    ----------------------------------------------------------------------------

    5) Run sample application

        Sample code is built by default.
        Information on running the performance sample-code is at
        ./quickassist/lookaside/access_layer/src/sample_code/README.txt

        Examples:
            Compression sample code expects to find data files at a known
            location, so call the samples-install target to put them there:
            sudo make samples-install

            Run all performance tests:
            ./cpa_sample_code

            Symmetric crypto tests only:
            ./cpa_sample_code runTests=1

            Asymmetric crypto RSA tests only:
            ./cpa_sample_code runTests=2

            Compression tests only:
            ./cpa_sample_code runTests=32

        Examples of other samples for a specific functionality:
            Run just one symmetric cipher operation:
            ./cipher_sample

            Run just one compress/decompress operation:
            ./dc_stateless_sample

    ----------------------------------------------------------------------------

    6) Remove the libraries and cleanup

        Uninstall:
            sudo make samples-uninstall #if samples installed
            sudo systemctl stop qat
            sudo make uninstall

        Clean up:
            make clean
            make distclean

===============================================================================


Full list of Configuration options
==================================

    Typical examples of how to update configuration options are as follows:

                ./configure ICP_ANY_FLAG=value or
                ./configure --enable-something

    If it is required to use more than one flag at once:

                ./configure ICP_ANY_PATH=path ICP_ANY_NAME=name --enable-something

    Features flags:
        Enables or disables the additional features supported by 4xxx package

        --disable-option-checking
                Ignores unrecognized configure options when run along with it.

        --disable-FEATURE
                Does not include FEATURE (same as --enable-FEATURE=no).

        --enable-FEATURE[=ARG]
                Includes FEATURE [ARG=yes].

        --enable-silent-rules
                Less verbose build output (undo: "make V=1").

        --disable-silent-rules
                Verbose build output (undo: "make V=0").

        --enable-maintainer-mode
                Enables make rules and dependencies not useful (and
                sometimes confusing) to the casual installer.

        --enable-dependency-tracking
                Does not reject slow dependency extractors.

        --disable-dependency-tracking
                Speeds up one-time build

        --enable-icp-debug
                Enables debugging.

        --disable-param-check
                Disables parameters checking in the top-level APIs
                (Use for performance optimization).

        --disable-stats
                Disables statistic collection (Use for performance optimization).

        --enable-icp-log-syslog
                Enables debugging messages to be outputted to the
                system log instead of standard output.

        --enable-icp-trace
                Enables tracing for the Cryptography API.

        --enable-dc-error-simulation
                Enables Data Compression Error Simulation.

        --enable-legacy-lib-names
                Enables legacy names for libraries.

        --enable-service
                Automatically creates "qat" group and enables systemd service
                during installation.

        MAX_MR
                Number of Miller Rabin rounds for prime operations. Setting this
                to a smaller value reduces the memory usage required by the
                driver.

                Type: value
                Default value: 50


===============================================================================


Common issues
=============

    Issue: errors like these are seen in system logs:
         * 4xxx 0000:6b:00.0: IOMMU should be enabled for SR-IOV to work
         * vfio-pci: probe of 0000:6b:00.1 failed with error -22
    Likely cause: Check VT-d, SR-IOV and intel_iommu settings are correct as
        described above. One way to check that is to run:
        dmesg | grep "Virtualization Technology"
        If you get output similar to below, everything is ready, otherwise
        there is something misconfigured.
        "[57.503644] DMAR: Intel(R) Virtualization Technology for Directed I/O"

    Issue: On running sample code:
        qaeMemInit started
        Open failed on /dev/vfio/15
        No device found
        ADF_UIO_PROXY err: icp_adf_userProcessToStart: Failed to start SSL...
    Likely cause: Incorrect permissions. Use "id <your user>" to verify the
        user is in the qat group and logout/login to the shell to ensure group
        changes take effect.

    Issue: On running sample code:
        qaeMemInit started
        No device found
        ADF_UIO_PROXY err: icp_adf_userProcessToStart: Failed to start SSL...
    Likely cause: (1) No PF driver available. Check that PFs are available and
        bound to qat_4xxx:
        sudo lspci -vvd:4940 | grep "Kernel driver in use".
        sudo lspci -vvd:4942 | grep "Kernel driver in use"
        If no driver in use, upgrade to Linux kernel 5.11 or greater.
        (2) No VFs available. Check VFs are available and bound to vfio-pci
        sudo lspci -vvd:4941 | grep "Kernel driver in use"
        sudo lspci -vvd:4943 | grep "Kernel driver in use"

    Issue: On running ./autogen.sh following warning appears:
        aclocal: warning: couldn't open directory 'm4': No such file or dir...
    Likely cause: This warning can be ignored, as is resolved subsequently

    Issue: After upgrading to new version following error may be received:
         This qatlib v21.8 received response from incompatible qatmgr v0.0
    Likely cause: if qat service was already running during upgrade, it won't
        be restarted automatically.
        After upgrading ensure that qat service is restarted.

    Issue: "DC Instances are not present" error when trying to run
        compression operations, e.g. using "cpa_sample_code runTests=32"
    Likely cause: QAT driver in Linux kernel before v5.17 doesn't support
        compression service. Upgrade to a later kernel.

    Issue: "Could not open corpus file: /usr/local/share/qat/calgary"
        seen when running compression sample code.
    Likely cause: sample code data files not installed at the expected location
        Run: sudo make samples-install

    Issue: Error "memFreeNUMA:1313 Address to be freed cannot be NULL"
        seen when running cpa_sample_code.
        Also dmesg | grep MEMLOCK shows messages like this:
        "hvfio_pin_pages_remote: RLIMIT_MEMLOCK (204800000) exceeded"
    Likely cause: Not enough locked memory. Increase the amount of locked
        memory for your user – see instructions above.
        After updating re-login so the changes take effect:
        sudo su -l $USER

    Issue: error on make install
        "Job for qat.service failed because the control process exited with
        error code"
        System logs (dmesg) show QAT kernel module failed with error like:
        "4xxx 0000:6b:00.0: enabling device (0140 -> 0142)
         QAT: authentication error (FCU_STATUS = 0x3),retry = 0
         4xxx 0000:6b:00.0: Failed to load MMP
         4xxx 0000:6b:00.0: Failed to load acceleration FW
         4xxx 0000:6b:00.0: Resetting device qat_dev0
         4xxx: probe of 0000:6b:00.0 failed with error -14"
    Likely cause: firmware files not present in initramfs on module loading
        early in boot process.
    Fix: First make sure you have the firmware installed in /lib/firmware,
        see pre-requisites section above.
        In order to load the firmware, the driver must be reloaded, i.e. run:
        "sudo rmmod qat_4xxx; sudo modprobe qat_4xxx; sudo systemctl start qat".
        For a persistent change (fix for future reboots) run "dracut --force".

    Issue: X kernel taint flag seen on SUSE from SLES15-SP4 onwards
        "intel_qat: externally supported module,setting X kernel taint flag."
        "qat_4xxx: externally supported module,setting X kernel taint flag."
    Likely cause: This is normal on load of the kernel driver in a SUSE distro.
        See explanation for X flag here:
        https://www.suse.com/support/kb/doc/?id=000016321
        It indicates that the modules are supported by external parties, in
        this case Intel.
    Fix: None necessary, qatlib can be used as normal despite this message.

    Issue: make: *** No rule to make target 'sample-uninstall'.  Stop.
    Likely cause: There was a name change between 21.11 and 22.07, note the
        extra 's'. Sorry if it causes confusion.
    Fix: sudo make samples-uninstall

===============================================================================


Legal/Disclaimers
===================

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL(R) PRODUCTS.
NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL
PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S
TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY
WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO
SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING
TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY
PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel products are
not intended for use in medical, life saving, life sustaining, critical control
 or safety systems, or in nuclear facility applications.

Intel may make changes to specifications and product descriptions at any time,
without notice.

(C) Intel Corporation 2021

* Other names and brands may be claimed as the property of others.

===============================================================================
