
================================================================================
README for AppQoS

December 2020
================================================================================

CONTENTS
========

- Introduction
- Requirements and Installation
- Configuration
- Usage
- REST API
- Usage Scenario
- How To Generate self-signed SSL certificate
- Legal Disclaimer


INTRODUCTION
============

AppQoS is a proof-of-concept software created to demonstrate the use of
Intel(R) RDT technologies (CAT, MBA) and Intel(R) SST technologies
Base Frequency (BF), Core Power (CP) to improve QoS for applications
via partitioning system resources.
AppQoS allows to assign Apps to Pools with different resources.
In current version each of the Pools (group of cores) has fixed amount of
cache and memory bandwidth allocated. Thanks to Intel(R) SST BF and CP support,
AppQoS allows the CPU to be deployed with an asymmetric core frequency
configuration. Amount of resources allocated and cores frequency
configuration could depend on Pool's priority.
Intel(R) RDT CAT and MBA configuration is done via "libpqos" library and
Intel(R) SST BF and CP configuration via external "Intel pwr" library.

AppQoS provides simple, local, REST API management interface secured
with HTTPS and "Basic HTTP Auth". REST API management interface allows
user to manage Apps and Pools. An App controlled by AppQoS could be
a process, a container or a VNF.

AppQoS is reference code written in Python. It is fully configurable and
can be easily modified to suit other use cases and allow remote management.


REQUIREMENTS AND INSTALLATION
=============================

AppQoS requires minimum Linux kernel v4.20 (for Intel(R) RDT L3 CAT and MBA)
however, for support of Intel(R) SST BF and CP Linux kernel v5.2 is required.
AppQoS is a part of Intel(R) RDT Software Package, it is located in "appqos"
directory.

AppQoS requires Python v3.x and depends on the following Python modules
and external components:
 - pqos (libqos Python wrapper)
 - flask
 - flask_httpauth
 - flask_restful
 - jsonschema
 - Intel pwr library (github.com, CommsPowerManagement)

NOTE:
To install mentioned dependencies in virtualenv please use "make setup" command.

Intel(R) RDT CAT and MBA configuration is done via libpqos using MSR
or OS/resctrl interfaces (user configurable), for more information please see
"Software Compatibility" section of README for Intel(R) RDT Software Package.
For hardware requirements please see "Hardware Support" section of
mentioned README file.

For more information about "Intel pwr" library used to configure Intel(R) SST BF and CP
please see https://github.com/intel/CommsPowerManagement .


CONFIGURATION
=============

AppQoS supports Pools of Apps, each with defined cores, cache and memory
bandwidth allocation. Each App describes a single app, with one or more PIDs.

Cache Allocation, Memory Bandwidth Allocation (via CAT and MBA), Base Frequency,
Core Power (via SST BF and CP) and affinity are configured by AppQoS on per core
basis.

NOTE: make sure that cores are isolated via "isolcpu=" kernel param.

AppQoS config is stored in JSON format in file "./appqos.conf" for which
JSON Schema file is available in "./schema" directory.

Configuration file is "Read-Only", no runtime changes are saved.

If there is no "Default" pool (with "id" equal to 0) defined in config file,
AppQoS will dynamically create one on start-up.
All unassigned cores will be assigned to "Default" pool,
MBA will be configured to "no throttling" (100% (default) or ~2^32MBps (MBA CTRL enabled))
and CAT CBM to all cache ways.

EXAMPLE CONFIG

{
"rdt_iface": {"interface": "msr"},
"mba_ctrl": {"enabled": false},
"apps": [{"id": 1, "name": "App1", "pids": [1979, 1980]},
         {"id": 2, "cores": [4], "name": "App2", "pids": [1592, 1593, 1594]},
         {"id": 3, "cores": [5], "name": "App3", "pids": [1576, 1577, 1578]}],
"pools": [{"id": 1, "cores": [1, 2, 3], "name": "HP", "mba": 100, "cbm": "0xFF0", "apps": [1], "power_profile": 1},
          {"id": 2, "cores": [4, 5, 6], "name": "MP", "mba": 50, "cbm": "0xC", "apps": [2, 3]},
          {"id": 3, "cores": [7, 8, 9], "name": "LP", "mba": 10, "cbm": "0x3", "apps": []}],
"power_profiles": [{"id": 1, "min_freq": 2500, "max_freq": 2500, "epp": "performance", "name": "HP"}],
"auth": {"username": "admin", "password": "password"},
"sstbf": {"configured": false},
"power_profiles_expert_mode": false,
"power_profiles_verify": true
}

"rdt_iface" section, Intel(R) RDT configuration.
 - "interface" - requested libpqos interface ("msr" or "os")

"mba_ctrl" section, Intel(R) RDT configuration.
 - "enabled" - MBA CTRL requested state (requires "os" interface)

"apps" section, Apps being managed by App QoS.
 - "id" - App's ID
 - "name" - App's name (optional)
 - "cores" - cores being used by App (optional)
 - "pids" - list of App's PIDs

"pools" section, Pools of Apps.
 - "id" - Pool's ID
 - "name" - Pool's name
 - "apps" - list of Apps being part of the Pool
 - "cbm" - Intel RDT CAT CacheWayBitmask assigned to Pool
 - MBA related configuration:
    - "mba" - Intel RDT MBA rate [%] assigned to Pool (default, 1 - 100 [%])
   OR
    - "mba_bw" - Intel RDT MBA rate [MBps] assigned to Pool (1 - 2^32-1 [MBps])
      (requires MBA CTRL to be enabled, please see config's "mba_ctrl" section)
 - "cores" - cores being assigned to Pool
 - "power_profile" - Power Profile ID to be applied on pool's cores

"power_profiles" section, Power Profiles/SST-CP.
 - "id" - Profile's ID
 - "name" - Profiles's name
 - "min_freq" - min. frequency [MHz]
 - "max_freq" - max. frequency [MHz]
 - "epp" - Energy Performance Preference

"auth" section, REST API username and password.
 - "username" - username
 - "password" - password

"sstbf" section, Intel(R) SST-BF configuration.
 - "configured" - SST-BF requested state

NOTE:
"power profiles" are ignored when SST-BF is configured.

Global options:
 - "power_profiles_expert_mode" - make power profiles editable via REST API
   (Default: False)

 - "power_profiles_verify" - Admission Control feature for config file content,
   verifies Power Profiles and Pools configuration (Default: True)

USAGE
=====

AppQoS is a Python script accepting the following, optional,
command line options:
 - -h, --help, show this help message and exit
 - -c PATH, --config PATH, Configuration file path
 - --port PORT, REST API port (default: 5000)
 - -V, --verbose, Verbose mode

NOTE: AppQoS requires root privileges.

By default it will attempt to read from "appqos.conf" file from current folder.
Example command line to run AppQoS in verbose mode and config file in
non-default location:

sudo ./appqos.py -c /tmp/appqos.conf -V

NOTE:
To run AppQoS in virtualenv please use "make run" command.

For more information about configuration file please see CONFIGURATION paragraph,
for information about runtime configuration please see REST API paragraph.
For more info please see USAGE SCENARIO paragraph.


REST API
========

The REST API is a local (but can be easily modified to allow remote access),
secured interface that allows the user to control AppQoS.
The REST API allows to:
 - add, remove or move App between Pools,
 - add, remove or modify Pools,
 - configure Intel(R) SST BF and CP technologies
 - configure Intel(R) RDT interface type and MBA mode

NOTE: None of configuration changes made via REST API are saved to configuration file.

As REST API uses HTTPS, it requires SSL certificate,
please see HOW TO GENERATE SELF-SIGNED SSL CERTIFICATE paragraph for more info.

The REST API uses "Basic HTTP Auth", username and password are stored in appqos.conf.
To set your own username and password please modify the following section of appqos.conf:

"auth": {"username": "admin", "password": "password"}

NOTE: If using curl to do REST API calls, add --user admin:password to command line

REST API URIs
-------------

JSON Schema files for REST API commands and responses are available in "./schema" directory.

- GET /apps - get all/collection of apps

- POST /apps - create new app
 Example request:
  {"pool_id": 2,
  "name": "hello",
  "cores": [1,2],
  "pids": [1]}

 Result:
  {"id": 5}

- GET /apps/{id} - get app for given id
 Example response:
  {"id": 6,
  "cores": [2, 3, 11],
  "name": "App",
  "pids": [1748, 1749, 1750],
  "pool_id": 3
  }

- PUT /apps/{id} - update app (e.g.:move between pools) for given id
 Example request:
  {"pool_id": 2}

- DELETE /apps/{id} - delete app for given id


- GET /pools - get all/collection of pools

- POST /pools - create new pool

- GET /pools/{id} - get pool for given id

- PUT /pools/{id} - modify pool for given id

- DELETE /pools/{id} - delete empty pool for given id


- GET /power_profiles - get all/collection of power profiles

- POST /power_profiles - create new power profile

- GET /power_profiles/{id} - get power profile for given id

- PUT /power_profiles/{id} - modify power profile for given id

- DELETE /power_profiles/{id} - delete power profile for given id


- GET /stats - get stats


- GET /caps - get system capabilities
 Example response:
  {"capabilities": ["cat","mba","sstbf","power"]
  }


- GET /caps/sstbf - get Intel(R) SST-BF details
 Example response:
  {"configured": true,
   "hp_cores": [2,...,88],
   "std_cores": [0,...,95]
  }

- PUT /caps/sstbf - configure Intel(R) SST-BF
 Example request:
  {"configured": true}


- GET /caps/rdt_iface - get Intel(R) RDT interface type
 Example response:
  {"interface": "msr",
   "interface_supported": ["msr", "os"]
  }

- PUT /caps/rdt_iface - set Intel(R) RDT interface type
 Example request:
  {"interface": "os"}
 NOTE: Request will be processed only when there are no pools (except "default" #0) configured in the system.
 "Default" pool will be reset to default values (full cache and MEM BW access).


- GET /caps/mba - get Intel(R) RDT MBA details
 Example response:
  {"mba_enabled": true,
  "mba_bw_enabled": false
  }


- GET /caps/mba_ctrl - get Intel(R) RDT MBA CTRL state
 Example response:
  {"enabled": false,
  "supported": true
  }

- PUT /caps/mba_ctrl - set Intel(R) RDT MBA CTRL state
 Example request:
  {"enabled": true}
 NOTE: Request will be processed only when there are no pools (except "default" #0) configured in the system.
 "Default" pool will be reset to default values (full cache and MEM BW access).


USAGE SCENARIO
==============

AppQoS, leveraging Intel(R) RDT (CAT, MBA) and SST (BF, CP) Technologies,
is to be used to prioritize and protect performance of high priority applications.

Please see below for complete step-by-step guide to configure AppQoS to
leverage Intel(R) RDT (CAT, MBA) and SST-BF Technologies to prioritize and
protect performance of high priority application.

SETUP

Get self-signed SSL certificate (for more info please see
HOW TO GENERATE SELF-SIGNED SSL CERTIFICATE paragraph).

Create appqos.conf file (for more info please see Configuration).

cat /tmp/appqos.conf
{
    "apps": [
    ],

    "auth": {
        "password": "password",
        "username": "admin"
    },

    "sstbf": {
        "configured": true
    },

    "pools": [
    ],

    "power_profiles_expert_mode": true,

    "rdt_iface": {
        "interface": "msr"},
    "mba_ctrl": {
        "enabled": false
    }
}

Empty config file with essential configuration only:
 - REST API Authentication details,
 - RDT interface and MBA mode configuration
 - SST-BF configuration
 - option to enable Power Profile "Expert Mode" to allow us to
   add power profiles via REST API

AppQoS command line parameters (for more info please see USAGE paragraph)

./appqos.py --help
usage: appqos.py [-h] [-c PATH] [--port PORT] [-V]

optional arguments:
  -h, --help            show this help message and exit
  -c PATH, --config PATH
                        Configuration file path
  --port PORT           REST API port
  -V, --verbose         Verbose mode

NOTE: REST API port (5000 by default) can be set via "--port"

START APPQOS

sudo make run

INFO  Interface MSR, MBA BW: unsupported.
INFO  Interface OS, MBA BW: supported.
INFO  Supported RDT interfaces: ['msr', 'os']
INFO  RDT initialized with 'msr' interface
INFO  Supported capabilities:
INFO  ['cat', 'mba', 'sstbf', 'power']
 * Serving Flask app "rest.rest_server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO   * Running on https://127.0.0.1:5000/ (Press CTRL+C to quit)
INFO  Configuring SST-BF
INFO  SST-BF enabled, configured.
INFO  SST-BF HP cores: [2, 10, 11, 12, 13, 14, 15, 16, 25, 34, 35, 36, 37, 38, 39, 40]
INFO  SST-BF STD cores: [0, 1, 3, 4, 5, 6, 7, 8, 9, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 41, 42, 43, 44, 45, 46, 47]
INFO  Power Profiles/EPP enabled, not configured, SST-BF is configured
INFO  Configuring RDT
INFO  RDT MBA CTRL disabled
INFO  Configuration changed, processing new config...

VERIFY INITIAL CONFIGURATION

List all Pools
curl https://localhost:5000/pools -X GET --user admin:password -k | jq

  {
    "id": 0,
    "mba": 100,
    "cbm": 2047,
    "name": "Default",
    "cores": [
      0,
      1,
      2,
      3,
...
      43,
      44,
      45,
      46,
      47
    ]
  }
]

NOTE: "Default" pool automatically created (id=0).
Pool takes all cores (48), 100% of MBA and full CAT CBM (all cache ways).

List all Apps
curl https://localhost:5000/apps -X GET --user admin:password -k | jq

{
    "message": "No apps in config file"
}

NOTE: No Apps defined.

List capabilities
curl https://localhost:5000/caps -X GET --user admin:password -k | jq

{
    "capabilities": [
        "cat",
        "mba",
        "sstbf",
        "power"
    ]
}

NOTE: Detected capabilities listed,
Intel(R) RDT CAT, MBA and Intel(R) SST BF, CP technologies supported.

List RDT interface configuration details
curl https://localhost:5000/caps/rdt_iface -X GET --user admin:password -k | jq -c

{"interface": "msr",
 "interface_supported": ["msr", "os"]
}
NOTE: MSR interface is used (as per config file). System supports both MSR and OS interfaces.

List MBA CTRL/MBA details
curl https://localhost:5000/caps/mba_ctrl -X GET --user admin:password -k | jq -c

{"enabled": false,
 "supported": false
}
NOTE: MBA CTRL not supported for MSR interface.

curl https://localhost:5000/caps/mba -X GET --user admin:password -k | jq -c

{"mba_enabled": true,
 "mba_bw_enabled": false}

NOTE: MBA CTRL is NOT enabled (as per config file) so "mba_bw" is not available,
user should use "mba" field to configure "pools" MBA in %.
If MBA CTRL would be enabled, user would use "mba_bw" to configure "pools" MBA in MBps.
Error 404 will be returned when MBA is not supported.

List Intel(R) SST-BF details
curl https://localhost:5000/caps/sstbf -X GET --user admin:password -k | jq -c

{"configured":true
 "hp_cores": [2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40],
 "std_cores": [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47]}

NOTE: SST-BF enabled (as per config file), list of HP and STD cores provided.

Verify frequencies settings

sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))'

        Freq.   Freq.
Core#   Min     Max     EPP
0       2100    2100    balance_performance
1       2100    2100    balance_performance
2       2800    2800    balance_performance
3       2100    2100    balance_performance
...
45      2100    2100    balance_performance
46      2100    2100    balance_performance
47      2100    2100    balance_performance

NOTE: HP cores' frequencies are set to 2800MHz, STD cores' to 2100MHz


Unconfigure SST-BF

curl https://localhost:5000/caps/sstbf -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"configured": false}'

{
    "message": "SST-BF caps modified"
}

Verify frequencies settings

sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))'

        Freq.   Freq.
Core#   Min     Max     EPP
0       2300    2300    balance_performance
1       2300    2300    balance_performance
2       2300    2300    balance_performance
3       2300    2300    balance_performance
...
45      2300    2300    balance_performance
46      2300    2300    balance_performance
47      2300    2300    balance_performance

NOTE: HP and STD cores' frequencies are set to 2300MHz


CREATE APPS

Create multiple processes, list their PIDs and core affinity

killall sleep
for n in {1..5}; do bash -c "sleep 1d &"; done
for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0-47
pid 13892's current affinity list: 0-47
pid 13890's current affinity list: 0-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

Create App, specify cores

curl https://localhost:5000/apps -X POST --user admin:password -k -H "Content-Type: application/json" -d '{"name": "APP1", "cores": [0,1,2,3], "pids": [13894, 13892]}'

{
    "id": 1
}

Verify App's creation (list ALL apps)

curl https://localhost:5000/apps -X GET --user admin:password -k
[
    {
        "name": " APP1",
        "cores": [
            0,
            1,
            2,
            3
        ],
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    }
]

NOTE: App was assigned to Pool #0 as cores 0,1,2,3 are part of that Pool

Verify PIDs' core affinity

for i in `pidof sleep`; do taskset -p -c $i; done
pid 13894's current affinity list: 0-3
pid 13892's current affinity list: 0-3
pid 13890's current affinity list: 0-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

Create App, specify destination Pool ID

curl https://localhost:5000/apps -X POST --user admin:password -k -H "Content-Type: application/json" -d '{"name": "APP2", "pool_id": 0, "pids": [13890]}'

{
    "id": 2
}

Verify App's creation (get specific App)

curl https://localhost:5000/apps/2 -X GET --user admin:password -k

{
    "name": "APP2",
    "pids": [
        13890
    ],
    "id": 2,
    "pool_id": 0
}

NOTE: App was assigned to Pool #0 as requested via "pool_id" param

Verify PIDs' core affinity

for i in `pidof sleep`; do taskset -p -c $i; done
pid 13894's current affinity list: 0-3
pid 13892's current affinity list: 0-3
pid 13890's current affinity list: 0-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

NOTE: App2 PID's core affinity is 0-47 as those are "Default" pool cores.

MODIFY "DEFAULT" POOL

List SST-BF's STD cores

curl https://localhost:5000/caps/sstbf -X GET --user admin:password -k | jq .std_cores -c
 [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47]

Modify "Default" pool to exclude HP cores, assign 50% of MBA and 4 LLC CWs (CBM: 0xF/15)

curl https://localhost:5000/pools/0 -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"cores": [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47], "mba": 50, "cbm": "0xf"}'

"POOL 0 updated"

Check new "Default" pool configuration

curl https://localhost:5000/pools/0 -X GET --user admin:password -k | jq -c

{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1,2]}

Check how does "Default" pool modification has affected Apps in that pool

curl https://localhost:5000/apps -X GET --user admin:password -k

[
    {
        "name": "APP1",
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    },
    {
        "name": "APP2",
        "pids": [
            13890
        ],
        "id": 2,
        "pool_id": 0
    }
]

for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

NOTE: As some of the cores that were specified for APP1 are not part of
"Default" pool anymore, core affinity for APP1 PIDs' was reset to
new "Default" pool cores.

CREATE HP POOL

Get list of HP cores

curl https://localhost:5000/caps/sstbf -X GET --user admin:password -k | jq .hp_cores -c
[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40]

Create HP Pool, with 7 (isolated) LLC CWs allocated and 100% of MBA

curl https://localhost:5000/pools -X POST --user admin:password -k -H "Content-Type: application/json" -d '{"name": "HP", "cores": [2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40], "cbm": "0x7F0", "mba": 100}'

{
    "id": 7
}

NOTE: Pool with "id=7" was created.

Get All Pools to verify creation of new HP Pool(#7)

curl https://localhost:5000/pools -X GET --user admin:password -k | jq -c

[{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1,2]},
{"name":"HP",
"cores":[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40],
"cbm":2032,
"mba":100,
"id":7}]

DEFINE AND APPLY POWER PROFILES

Define HP power profile

curl https://localhost:5000/power_profiles -X POST --user admin:password -k -H "Content-Type: application/json" -d '{"name": "PP HP", "min_freq": 3000, "max_freq": 3000, "epp": "performance"}'

{
  "id": 0,
  "message": "New POWER PROFILE 0 added"
}

Define STD power profile

curl https://localhost:5000/power_profiles -X POST --user admin:password -k -H "Content-Type: application/json" -d '{"name": "PP STD", "min_freq": 1500, "max_freq": 1500, "epp": "power"}'

{
  "id": 1,
  "message": "New POWER PROFILE 1 added"
}

Get All Power Profiles to verify creation of new Power Profiles

curl https://localhost:5000/power_profiles -X GET --user admin:password -k | jq

[
  {
    "name": "PP HP",
    "min_freq": 3000,
    "max_freq": 3000,
    "epp": "performance",
    "id": 0
  },
  {
    "name": "PP STD",
    "min_freq": 1500,
    "max_freq": 1500,
    "epp": "power",
    "id": 1
  }
]

Apply power profiles

curl https://localhost:5000/pools/7 -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"power_profile": 0}'

{
  "message": "POOL 7 not updated, Power Profiles configuration would cause CPU to be oversubscribed."
}

NOTE:
AppQoS supports Admission Control feature, which protects CPU from being oversubscribed,
as a result of that, order in which power profile are applied is important.
(Configuration change can be forced by setting "verify" flag to "false" for a request)

Lets apply the STD power profile first.

curl https://localhost:5000/pools/0 -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"power_profile": 1}'

{
  "message": "POOL 0 updated"
}

curl https://localhost:5000/pools/7 -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"power_profile": 0}'

{
  "message": "POOL 7 updated"
}

Verify power profiles configuration in HW

sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))'

        Freq.   Freq.
Core#   Min     Max     EPP
0       1500    1500    power
1       1500    1500    power
2       3000    3000    performance
3       1500    1500    power
...
45      1500    1500    power
46      1500    1500    power
47      1500    1500    power

NOTE:
Cores from HP pool have frequency fixed to 3000MHz and EPP set to "performance",
STD pool's cores have frequency fixed to 1500MHz and EPP set to "power".

MOVE APP#2

Move App#2 to HP Pool(#7)

curl https://localhost:5000/apps/2 -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"pool_id": 7}'

"APP 2 moved to new pool"

Verify new configuration, list all apps

curl https://localhost:5000/apps -X GET --user admin:password -k

[
    {
        "name": "APP1",
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    },
    {
        "name": "APP2",
        "pids": [
            13890
        ],
        "id": 2,
        "pool_id": 7
    }
]

Verify new configuration, list pools

curl https://localhost:5000/pools -X GET --user admin:password -k | jq -c

[{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1]},
{"name":"HP",
"cores":[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40],
"cbm":2032,
"mba":100,
"id":7,
"apps":[2]}]

Verify new configuration, check core affinity

for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 2,10-16,25,34-40
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

MODIFY APP#2

Modify App core affinity and name

curl https://localhost:5000/apps/2 -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"cores": [2,11,13], "name": "APP2 HP"}'

"APP 2 moved to new pool"

Verify new configuration, get App#2 details

curl https://localhost:5000/apps/2 -X GET --user admin:password -k

{
    "name": "APP2 HP",
    "pids": [
        13890
    ],
    "id": 2,
    "cores": [
        2,
        11,
        13
    ],
    "pool_id": 7
}

Verify new configuration, check core affinity

for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 2,11,13
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

REMOVE APP#2

Remove App#2

curl https://localhost:5000/apps/2 -X DELETE --user admin:password -k

"APP 2 deleted"

Verify new configuration, get App#2 and all Apps details

curl https://localhost:5000/apps/2 -X GET --user admin:password -k

{
    "message": "APP 2 not found in config"
}

NOTE: App#2 not found

curl https://localhost:5000/apps -X GET --user admin:password -k
[
    {
        "name": "stress-ng",
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    }
]

NOTE: No App#2 listed

Verify new configuration, check core affinity

for i in `pidof sleep`; do taskset -p -c $i; done
pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

NOTE: App#2's PID core affinity set to "Default" pool cores

REMOVE HP POOL

Remove HP Pool (#7)

curl https://localhost:5000/pools/7 -X DELETE --user admin:password -k

"POOL 7 deleted"

Verify new configuration, get Pool#7 and all  pools details

curl https://localhost:5000/pools/7 -X GET --user admin:password -k

{
    "message": "POOL 7 not found in config"
}

NOTE: Pool#7 not found

curl https://localhost:5000/pools -X GET --user admin:password -k | jq -c

[{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1]}]

NOTE: No Pool#7 listed

RESET CONFIGURATION

Perform configuration reset.

curl https://localhost:5000/reset -X POST --user admin:password -k

"Reset performed. Configuration reloaded."

Verify configuration after reset

curl https://localhost:5000/apps -X GET --user admin:password -k

{
    "message": "No apps in config file"
}

NOTE: No Apps configured

curl https://localhost:5000/pools -X GET --user admin:password -k | jq -c

[{"id":0,
"mba":100,
"cbm":2047,
"name":"Default",
"cores":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]}]

NOTE: "Default" pool reconfigured, takes all cores (48),
100% of MBA and full CAT CBM (all cache ways).

ENABLE RDT OS INTERFACE

Verify that RDT OS interface is supported.

curl https://localhost:5000/caps/rdt_iface -X GET --user admin:password -k | jq -c

{"interface": "msr",
 "interface_supported": ["msr", "os"]
}

NOTE: RDT "os" interface is supported but NOT enabled.

Enable RDT OS interface.

curl https://localhost:5000/caps/rdt_iface -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"interface": "os"}'

{"message": "RDT Interface modified"}

NOTE: Can be done ONLY when there are no Pools configured (other than "default" Pool #0)
(e.g.: all non-default, configured Pools were removed).
"Default" pool will be reset to default values (full cache and MEM BW access).

List RDT configuration details

curl https://localhost:5000/caps/rdt_iface -X GET --user admin:password -k | jq -c

{"interface": "os",
 "interface_supported": ["msr", "os"]
}

NOTE: RDT OS interface is now enabled.

ENABLE MBA CTRL

Verify MBA CTRL status.

curl https://localhost:5000/caps/mba_ctrl -X GET --user admin:password -k | jq -c

{"enabled": false,
 "supported": true
}

NOTE: MBA CTRL is supported but NOT enabled.

Enable RDT MBA CTRL 

curl https://localhost:5000/caps/mba_ctrl -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"enabled": true}'

{"message": "MBA CTRL status changed."}

NOTE: Can be done ONLY when there are no Pools configured (other than "default" Pool #0)
(e.g.: all non-default, configured Pools were removed).
"Default" pool will be reset to default values (full cache and MEM BW access).

Verify MBA CTRL status

mount | grep resctrl

resctrl on /sys/fs/resctrl type resctrl (rw,relatime,mba_MBps)

NOTE: resctrl fs mounted with "mba_MBps" option, MBA CTRL is enabled.

curl https://localhost:5000/caps/mba_ctrl -X GET --user admin:password -k | jq -c

{"enabled": true,
 "supported": true
}

curl https://localhost:5000/caps/mba -X GET --user admin:password -k | jq -c

{"mba_bw_enabled": true
 "mba_enabled": false}

NOTE: MBA CTRL is now enabled.
User can use "mba_bw" to configure pool's MBA allocation.

Verify MBA configuration for "Default" Pool

curl https://localhost:5000/pools -X GET --user admin:password -k | jq

[
  {
    "id": 0,
    "mba_bw": 4294967295,
    "cbm": 2047,
    "name": "Default",
    "cores": [
      0,
      1,
...
      45,
      46,
      47
    ]
  }
]

NOTE: MBA for Pool #0 set to maximum value 4294967295MBps (~2^32MBps).

Limit Pool #0 MBA to ~5GBps

curl https://localhost:5000/pools/0 -X PUT --user admin:password -k -H "Content-Type: application/json" -d '{"mba_bw": 5000}'

{"message": "POOL 0 updated"}

Verify new Pool #0 MBA configuration.

curl https://localhost:5000/pools/0 -X GET --user admin:password -k | jq

{
  "id": 0,
  "mba_bw": 5000,
  "cbm": 2047,
  "name": "Default",
  "cores": [
    0,
    1,
...
    45,
    46,
    47
  ]
}

NOTE: MBA for Pool #0 is set to 5000MBps

cat /sys/fs/resctrl/schemata

    L3:0=7ff;1=7ff
    MB:0=5000;1=5000

NOTE: Resctrl shows new MBA configuration for Pool #0.


HOW TO GENERATE SELF-SIGNED SSL CERTIFICATE
===========================================

NOTE: just for example purposes, please obtain proper certificate !

It is needed to generate self-signed SSL certificate
(e.g.: using the OpenSSL toolkit) to enable HTTPS connections.

To generate a self-signed SSL certificate using the OpenSSL, run:

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout appqos.key -out appqos.crt

NOTE: The filenames are hardcoded to "appqos.key" and "appqos.crt"


Legal Disclaimer
================

THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR
IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS
ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND
CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL
DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR
USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO
FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT
OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
