diff --git a/SOURCES/ci-Change-netifaces-dependency-to-0.10.4-965.patch b/SOURCES/ci-Change-netifaces-dependency-to-0.10.4-965.patch new file mode 100644 index 0000000..02b5deb --- /dev/null +++ b/SOURCES/ci-Change-netifaces-dependency-to-0.10.4-965.patch @@ -0,0 +1,47 @@ +From 67d62f2c0df1fcb5cd86be73cba6064075aa61e3 Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito +Date: Fri, 14 Jan 2022 16:39:46 +0100 +Subject: [PATCH 3/6] Change netifaces dependency to 0.10.4 (#965) + +RH-Author: Emanuele Giuseppe Esposito +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [3/6] 81f0638e62841bab09b423d9cb5d340026ee87c2 +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy +RH-Acked-by: Eduardo Otubo + +commit b9d308b4d61d22bacc05bcae59819755975631f8 +Author: Andrew Kutz <101085+akutz@users.noreply.github.com> +Date: Tue Aug 10 15:10:44 2021 -0500 + + Change netifaces dependency to 0.10.4 (#965) + + Change netifaces dependency to 0.10.4 + + Currently versions Ubuntu <=20.10 use netifaces 0.10.4 By requiring + netifaces 0.10.9, the VMware datasource omitted itself from cloud-init + on Ubuntu <=20.10. + + This patch changes the netifaces dependency to 0.10.4. While it is true + there are patches to netifaces post 0.10.4 that are desirable, testing + against the most common network configuration was performed to verify + the VMware datasource will still function with netifaces 0.10.4. + +Signed-off-by: Emanuele Giuseppe Esposito +--- + requirements.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/requirements.txt b/requirements.txt +index 41d01d62..c4adc455 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -40,4 +40,4 @@ jsonschema + # and still participate in instance-data by gathering the network in detail at + # runtime and merge that information into the metadata and repersist that to + # disk. +-netifaces>=0.10.9 ++netifaces>=0.10.4 +-- +2.27.0 + diff --git a/SOURCES/ci-Datasource-for-VMware-953.patch b/SOURCES/ci-Datasource-for-VMware-953.patch new file mode 100644 index 0000000..4e24674 --- /dev/null +++ b/SOURCES/ci-Datasource-for-VMware-953.patch @@ -0,0 +1,2198 @@ +From 697152978b1194aa10ab39597802bb2b4041773c Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito +Date: Fri, 14 Jan 2022 16:37:42 +0100 +Subject: [PATCH 2/6] Datasource for VMware (#953) + +RH-Author: Emanuele Giuseppe Esposito +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [2/6] a0999fa63b8117959839f62bd470f9fe632b31cc +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy +RH-Acked-by: Eduardo Otubo + +commit 8b4a9bc7b81e61943af873bad92e2133f8275b0b +Author: Andrew Kutz <101085+akutz@users.noreply.github.com> +Date: Mon Aug 9 21:24:07 2021 -0500 + + Datasource for VMware (#953) + + This patch finally introduces the Cloud-Init Datasource for VMware + GuestInfo as a part of cloud-init proper. This datasource has existed + since 2018, and rapidly became the de facto datasource for developers + working with Packer, Terraform, for projects like kube-image-builder, + and the de jure datasource for Photon OS. + + The major change to the datasource from its previous incarnation is + the name. Now named DatasourceVMware, this new version of the + datasource will allow multiple transport types in addition to + GuestInfo keys. + + This datasource includes several unique features developed to address + real-world situations: + + * Support for reading any key (metadata, userdata, vendordata) both + from the guestinfo table when running on a VM in vSphere as well as + from an environment variable when running inside of a container, + useful for rapid dev/test. + + * Allows booting with DHCP while still providing full participation + in Cloud-Init instance data and Jinja queries. The netifaces library + provides the ability to inspect the network after it is online, + and the runtime network configuration is then merged into the + existing metadata and persisted to disk. + + * Advertises the local_ipv4 and local_ipv6 addresses via guestinfo + as well. This is useful as Guest Tools is not always able to + identify what would be considered the local address. + + The primary author and current steward of this datasource spoke at + Cloud-Init Con 2020 where there was interest in contributing this datasource + to the Cloud-Init codebase. + + The datasource currently lives in its own GitHub repository at + https://github.com/vmware/cloud-init-vmware-guestinfo. Once the datasource + is merged into Cloud-Init, the old repository will be deprecated. + +Signed-off-by: Emanuele Giuseppe Esposito +--- + README.md | 2 +- + cloudinit/settings.py | 1 + + cloudinit/sources/DataSourceVMware.py | 871 ++++++++++++++++++ + doc/rtd/topics/availability.rst | 1 + + doc/rtd/topics/datasources.rst | 2 +- + doc/rtd/topics/datasources/vmware.rst | 359 ++++++++ + requirements.txt | 9 + + .../unittests/test_datasource/test_common.py | 3 + + .../unittests/test_datasource/test_vmware.py | 377 ++++++++ + tests/unittests/test_ds_identify.py | 279 +++++- + tools/.github-cla-signers | 1 + + tools/ds-identify | 76 +- + 12 files changed, 1977 insertions(+), 4 deletions(-) + create mode 100644 cloudinit/sources/DataSourceVMware.py + create mode 100644 doc/rtd/topics/datasources/vmware.rst + create mode 100644 tests/unittests/test_datasource/test_vmware.py + +diff --git a/README.md b/README.md +index 435405da..b98f61d3 100644 +--- a/README.md ++++ b/README.md +@@ -39,7 +39,7 @@ get in contact with that distribution and send them our way! + + | Supported OSes | Supported Public Clouds | Supported Private Clouds | + | --- | --- | --- | +-| Alpine Linux
ArchLinux
Debian
Fedora
FreeBSD
Gentoo Linux
NetBSD
OpenBSD
RHEL/CentOS
SLES/openSUSE
Ubuntu










| Amazon Web Services
Microsoft Azure
Google Cloud Platform
Oracle Cloud Infrastructure
Softlayer
Rackspace Public Cloud
IBM Cloud
Digital Ocean
Bigstep
Hetzner
Joyent
CloudSigma
Alibaba Cloud
OVH
OpenNebula
Exoscale
Scaleway
CloudStack
AltCloud
SmartOS
HyperOne
Rootbox
| Bare metal installs
OpenStack
LXD
KVM
Metal-as-a-Service (MAAS)















| ++| Alpine Linux
ArchLinux
Debian
Fedora
FreeBSD
Gentoo Linux
NetBSD
OpenBSD
RHEL/CentOS
SLES/openSUSE
Ubuntu










| Amazon Web Services
Microsoft Azure
Google Cloud Platform
Oracle Cloud Infrastructure
Softlayer
Rackspace Public Cloud
IBM Cloud
Digital Ocean
Bigstep
Hetzner
Joyent
CloudSigma
Alibaba Cloud
OVH
OpenNebula
Exoscale
Scaleway
CloudStack
AltCloud
SmartOS
HyperOne
Rootbox
| Bare metal installs
OpenStack
LXD
KVM
Metal-as-a-Service (MAAS)

VMware














| + + ## To start developing cloud-init + +diff --git a/cloudinit/settings.py b/cloudinit/settings.py +index 2acf2615..d5f32dbb 100644 +--- a/cloudinit/settings.py ++++ b/cloudinit/settings.py +@@ -42,6 +42,7 @@ CFG_BUILTIN = { + 'Exoscale', + 'RbxCloud', + 'UpCloud', ++ 'VMware', + # At the end to act as a 'catch' when none of the above work... + 'None', + ], +diff --git a/cloudinit/sources/DataSourceVMware.py b/cloudinit/sources/DataSourceVMware.py +new file mode 100644 +index 00000000..22ca63de +--- /dev/null ++++ b/cloudinit/sources/DataSourceVMware.py +@@ -0,0 +1,871 @@ ++# Cloud-Init DataSource for VMware ++# ++# Copyright (c) 2018-2021 VMware, Inc. All Rights Reserved. ++# ++# Authors: Anish Swaminathan ++# Andrew Kutz ++# ++# This file is part of cloud-init. See LICENSE file for license information. ++ ++"""Cloud-Init DataSource for VMware ++ ++This module provides a cloud-init datasource for VMware systems and supports ++multiple transports types, including: ++ ++ * EnvVars ++ * GuestInfo ++ ++Netifaces (https://github.com/al45tair/netifaces) ++ ++ Please note this module relies on the netifaces project to introspect the ++ runtime, network configuration of the host on which this datasource is ++ running. This is in contrast to the rest of cloud-init which uses the ++ cloudinit/netinfo module. ++ ++ The reasons for using netifaces include: ++ ++ * Netifaces is built in C and is more portable across multiple systems ++ and more deterministic than shell exec'ing local network commands and ++ parsing their output. ++ ++ * Netifaces provides a stable way to determine the view of the host's ++ network after DHCP has brought the network online. Unlike most other ++ datasources, this datasource still provides support for JINJA queries ++ based on networking information even when the network is based on a ++ DHCP lease. While this does not tie this datasource directly to ++ netifaces, it does mean the ability to consistently obtain the ++ correct information is paramount. ++ ++ * It is currently possible to execute this datasource on macOS ++ (which many developers use today) to print the output of the ++ get_host_info function. This function calls netifaces to obtain ++ the same runtime network configuration that the datasource would ++ persist to the local system's instance data. ++ ++ However, the netinfo module fails on macOS. The result is either a ++ hung operation that requires a SIGINT to return control to the user, ++ or, if brew is used to install iproute2mac, the ip commands are used ++ but produce output the netinfo module is unable to parse. ++ ++ While macOS is not a target of cloud-init, this feature is quite ++ useful when working on this datasource. ++ ++ For more information about this behavior, please see the following ++ PR comment, https://bit.ly/3fG7OVh. ++ ++ The authors of this datasource are not opposed to moving away from ++ netifaces. The goal may be to eventually do just that. This proviso was ++ added to the top of this module as a way to remind future-us and others ++ why netifaces was used in the first place in order to either smooth the ++ transition away from netifaces or embrace it further up the cloud-init ++ stack. ++""" ++ ++import collections ++import copy ++from distutils.spawn import find_executable ++import ipaddress ++import json ++import os ++import socket ++import time ++ ++from cloudinit import dmi, log as logging ++from cloudinit import sources ++from cloudinit import util ++from cloudinit.subp import subp, ProcessExecutionError ++ ++import netifaces ++ ++ ++PRODUCT_UUID_FILE_PATH = "/sys/class/dmi/id/product_uuid" ++ ++LOG = logging.getLogger(__name__) ++NOVAL = "No value found" ++ ++DATA_ACCESS_METHOD_ENVVAR = "envvar" ++DATA_ACCESS_METHOD_GUESTINFO = "guestinfo" ++ ++VMWARE_RPCTOOL = find_executable("vmware-rpctool") ++REDACT = "redact" ++CLEANUP_GUESTINFO = "cleanup-guestinfo" ++VMX_GUESTINFO = "VMX_GUESTINFO" ++GUESTINFO_EMPTY_YAML_VAL = "---" ++ ++LOCAL_IPV4 = "local-ipv4" ++LOCAL_IPV6 = "local-ipv6" ++WAIT_ON_NETWORK = "wait-on-network" ++WAIT_ON_NETWORK_IPV4 = "ipv4" ++WAIT_ON_NETWORK_IPV6 = "ipv6" ++ ++ ++class DataSourceVMware(sources.DataSource): ++ """ ++ Setting the hostname: ++ The hostname is set by way of the metadata key "local-hostname". ++ ++ Setting the instance ID: ++ The instance ID may be set by way of the metadata key "instance-id". ++ However, if this value is absent then the instance ID is read ++ from the file /sys/class/dmi/id/product_uuid. ++ ++ Configuring the network: ++ The network is configured by setting the metadata key "network" ++ with a value consistent with Network Config Versions 1 or 2, ++ depending on the Linux distro's version of cloud-init: ++ ++ Network Config Version 1 - http://bit.ly/cloudinit-net-conf-v1 ++ Network Config Version 2 - http://bit.ly/cloudinit-net-conf-v2 ++ ++ For example, CentOS 7's official cloud-init package is version ++ 0.7.9 and does not support Network Config Version 2. However, ++ this datasource still supports supplying Network Config Version 2 ++ data as long as the Linux distro's cloud-init package is new ++ enough to parse the data. ++ ++ The metadata key "network.encoding" may be used to indicate the ++ format of the metadata key "network". Valid encodings are base64 ++ and gzip+base64. ++ """ ++ ++ dsname = "VMware" ++ ++ def __init__(self, sys_cfg, distro, paths, ud_proc=None): ++ sources.DataSource.__init__(self, sys_cfg, distro, paths, ud_proc) ++ ++ self.data_access_method = None ++ self.vmware_rpctool = VMWARE_RPCTOOL ++ ++ def _get_data(self): ++ """ ++ _get_data loads the metadata, userdata, and vendordata from one of ++ the following locations in the given order: ++ ++ * envvars ++ * guestinfo ++ ++ Please note when updating this function with support for new data ++ transports, the order should match the order in the dscheck_VMware ++ function from the file ds-identify. ++ """ ++ ++ # Initialize the locally scoped metadata, userdata, and vendordata ++ # variables. They are assigned below depending on the detected data ++ # access method. ++ md, ud, vd = None, None, None ++ ++ # First check to see if there is data via env vars. ++ if os.environ.get(VMX_GUESTINFO, ""): ++ md = guestinfo_envvar("metadata") ++ ud = guestinfo_envvar("userdata") ++ vd = guestinfo_envvar("vendordata") ++ ++ if md or ud or vd: ++ self.data_access_method = DATA_ACCESS_METHOD_ENVVAR ++ ++ # At this point, all additional data transports are valid only on ++ # a VMware platform. ++ if not self.data_access_method: ++ system_type = dmi.read_dmi_data("system-product-name") ++ if system_type is None: ++ LOG.debug("No system-product-name found") ++ return False ++ if "vmware" not in system_type.lower(): ++ LOG.debug("Not a VMware platform") ++ return False ++ ++ # If no data was detected, check the guestinfo transport next. ++ if not self.data_access_method: ++ if self.vmware_rpctool: ++ md = guestinfo("metadata", self.vmware_rpctool) ++ ud = guestinfo("userdata", self.vmware_rpctool) ++ vd = guestinfo("vendordata", self.vmware_rpctool) ++ ++ if md or ud or vd: ++ self.data_access_method = DATA_ACCESS_METHOD_GUESTINFO ++ ++ if not self.data_access_method: ++ LOG.error("failed to find a valid data access method") ++ return False ++ ++ LOG.info("using data access method %s", self._get_subplatform()) ++ ++ # Get the metadata. ++ self.metadata = process_metadata(load_json_or_yaml(md)) ++ ++ # Get the user data. ++ self.userdata_raw = ud ++ ++ # Get the vendor data. ++ self.vendordata_raw = vd ++ ++ # Redact any sensitive information. ++ self.redact_keys() ++ ++ # get_data returns true if there is any available metadata, ++ # userdata, or vendordata. ++ if self.metadata or self.userdata_raw or self.vendordata_raw: ++ return True ++ else: ++ return False ++ ++ def setup(self, is_new_instance): ++ """setup(is_new_instance) ++ ++ This is called before user-data and vendor-data have been processed. ++ ++ Unless the datasource has set mode to 'local', then networking ++ per 'fallback' or per 'network_config' will have been written and ++ brought up the OS at this point. ++ """ ++ ++ host_info = wait_on_network(self.metadata) ++ LOG.info("got host-info: %s", host_info) ++ ++ # Reflect any possible local IPv4 or IPv6 addresses in the guest ++ # info. ++ advertise_local_ip_addrs(host_info) ++ ++ # Ensure the metadata gets updated with information about the ++ # host, including the network interfaces, default IP addresses, ++ # etc. ++ self.metadata = util.mergemanydict([self.metadata, host_info]) ++ ++ # Persist the instance data for versions of cloud-init that support ++ # doing so. This occurs here rather than in the get_data call in ++ # order to ensure that the network interfaces are up and can be ++ # persisted with the metadata. ++ self.persist_instance_data() ++ ++ def _get_subplatform(self): ++ get_key_name_fn = None ++ if self.data_access_method == DATA_ACCESS_METHOD_ENVVAR: ++ get_key_name_fn = get_guestinfo_envvar_key_name ++ elif self.data_access_method == DATA_ACCESS_METHOD_GUESTINFO: ++ get_key_name_fn = get_guestinfo_key_name ++ else: ++ return sources.METADATA_UNKNOWN ++ ++ return "%s (%s)" % ( ++ self.data_access_method, ++ get_key_name_fn("metadata"), ++ ) ++ ++ @property ++ def network_config(self): ++ if "network" in self.metadata: ++ LOG.debug("using metadata network config") ++ else: ++ LOG.debug("using fallback network config") ++ self.metadata["network"] = { ++ "config": self.distro.generate_fallback_config(), ++ } ++ return self.metadata["network"]["config"] ++ ++ def get_instance_id(self): ++ # Pull the instance ID out of the metadata if present. Otherwise ++ # read the file /sys/class/dmi/id/product_uuid for the instance ID. ++ if self.metadata and "instance-id" in self.metadata: ++ return self.metadata["instance-id"] ++ with open(PRODUCT_UUID_FILE_PATH, "r") as id_file: ++ self.metadata["instance-id"] = str(id_file.read()).rstrip().lower() ++ return self.metadata["instance-id"] ++ ++ def get_public_ssh_keys(self): ++ for key_name in ( ++ "public-keys-data", ++ "public_keys_data", ++ "public-keys", ++ "public_keys", ++ ): ++ if key_name in self.metadata: ++ return sources.normalize_pubkey_data(self.metadata[key_name]) ++ return [] ++ ++ def redact_keys(self): ++ # Determine if there are any keys to redact. ++ keys_to_redact = None ++ if REDACT in self.metadata: ++ keys_to_redact = self.metadata[REDACT] ++ elif CLEANUP_GUESTINFO in self.metadata: ++ # This is for backwards compatibility. ++ keys_to_redact = self.metadata[CLEANUP_GUESTINFO] ++ ++ if self.data_access_method == DATA_ACCESS_METHOD_GUESTINFO: ++ guestinfo_redact_keys(keys_to_redact, self.vmware_rpctool) ++ ++ ++def decode(key, enc_type, data): ++ """ ++ decode returns the decoded string value of data ++ key is a string used to identify the data being decoded in log messages ++ """ ++ LOG.debug("Getting encoded data for key=%s, enc=%s", key, enc_type) ++ ++ raw_data = None ++ if enc_type in ["gzip+base64", "gz+b64"]: ++ LOG.debug("Decoding %s format %s", enc_type, key) ++ raw_data = util.decomp_gzip(util.b64d(data)) ++ elif enc_type in ["base64", "b64"]: ++ LOG.debug("Decoding %s format %s", enc_type, key) ++ raw_data = util.b64d(data) ++ else: ++ LOG.debug("Plain-text data %s", key) ++ raw_data = data ++ ++ return util.decode_binary(raw_data) ++ ++ ++def get_none_if_empty_val(val): ++ """ ++ get_none_if_empty_val returns None if the provided value, once stripped ++ of its trailing whitespace, is empty or equal to GUESTINFO_EMPTY_YAML_VAL. ++ ++ The return value is always a string, regardless of whether the input is ++ a bytes class or a string. ++ """ ++ ++ # If the provided value is a bytes class, convert it to a string to ++ # simplify the rest of this function's logic. ++ val = util.decode_binary(val) ++ val = val.rstrip() ++ if len(val) == 0 or val == GUESTINFO_EMPTY_YAML_VAL: ++ return None ++ return val ++ ++ ++def advertise_local_ip_addrs(host_info): ++ """ ++ advertise_local_ip_addrs gets the local IP address information from ++ the provided host_info map and sets the addresses in the guestinfo ++ namespace ++ """ ++ if not host_info: ++ return ++ ++ # Reflect any possible local IPv4 or IPv6 addresses in the guest ++ # info. ++ local_ipv4 = host_info.get(LOCAL_IPV4) ++ if local_ipv4: ++ guestinfo_set_value(LOCAL_IPV4, local_ipv4) ++ LOG.info("advertised local ipv4 address %s in guestinfo", local_ipv4) ++ ++ local_ipv6 = host_info.get(LOCAL_IPV6) ++ if local_ipv6: ++ guestinfo_set_value(LOCAL_IPV6, local_ipv6) ++ LOG.info("advertised local ipv6 address %s in guestinfo", local_ipv6) ++ ++ ++def handle_returned_guestinfo_val(key, val): ++ """ ++ handle_returned_guestinfo_val returns the provided value if it is ++ not empty or set to GUESTINFO_EMPTY_YAML_VAL, otherwise None is ++ returned ++ """ ++ val = get_none_if_empty_val(val) ++ if val: ++ return val ++ LOG.debug("No value found for key %s", key) ++ return None ++ ++ ++def get_guestinfo_key_name(key): ++ return "guestinfo." + key ++ ++ ++def get_guestinfo_envvar_key_name(key): ++ return ("vmx." + get_guestinfo_key_name(key)).upper().replace(".", "_", -1) ++ ++ ++def guestinfo_envvar(key): ++ val = guestinfo_envvar_get_value(key) ++ if not val: ++ return None ++ enc_type = guestinfo_envvar_get_value(key + ".encoding") ++ return decode(get_guestinfo_envvar_key_name(key), enc_type, val) ++ ++ ++def guestinfo_envvar_get_value(key): ++ env_key = get_guestinfo_envvar_key_name(key) ++ return handle_returned_guestinfo_val(key, os.environ.get(env_key, "")) ++ ++ ++def guestinfo(key, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ guestinfo returns the guestinfo value for the provided key, decoding ++ the value when required ++ """ ++ val = guestinfo_get_value(key, vmware_rpctool) ++ if not val: ++ return None ++ enc_type = guestinfo_get_value(key + ".encoding", vmware_rpctool) ++ return decode(get_guestinfo_key_name(key), enc_type, val) ++ ++ ++def guestinfo_get_value(key, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ Returns a guestinfo value for the specified key. ++ """ ++ LOG.debug("Getting guestinfo value for key %s", key) ++ ++ try: ++ (stdout, stderr) = subp( ++ [ ++ vmware_rpctool, ++ "info-get " + get_guestinfo_key_name(key), ++ ] ++ ) ++ if stderr == NOVAL: ++ LOG.debug("No value found for key %s", key) ++ elif not stdout: ++ LOG.error("Failed to get guestinfo value for key %s", key) ++ return handle_returned_guestinfo_val(key, stdout) ++ except ProcessExecutionError as error: ++ if error.stderr == NOVAL: ++ LOG.debug("No value found for key %s", key) ++ else: ++ util.logexc( ++ LOG, ++ "Failed to get guestinfo value for key %s: %s", ++ key, ++ error, ++ ) ++ except Exception: ++ util.logexc( ++ LOG, ++ "Unexpected error while trying to get " ++ + "guestinfo value for key %s", ++ key, ++ ) ++ ++ return None ++ ++ ++def guestinfo_set_value(key, value, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ Sets a guestinfo value for the specified key. Set value to an empty string ++ to clear an existing guestinfo key. ++ """ ++ ++ # If value is an empty string then set it to a single space as it is not ++ # possible to set a guestinfo key to an empty string. Setting a guestinfo ++ # key to a single space is as close as it gets to clearing an existing ++ # guestinfo key. ++ if value == "": ++ value = " " ++ ++ LOG.debug("Setting guestinfo key=%s to value=%s", key, value) ++ ++ try: ++ subp( ++ [ ++ vmware_rpctool, ++ ("info-set %s %s" % (get_guestinfo_key_name(key), value)), ++ ] ++ ) ++ return True ++ except ProcessExecutionError as error: ++ util.logexc( ++ LOG, ++ "Failed to set guestinfo key=%s to value=%s: %s", ++ key, ++ value, ++ error, ++ ) ++ except Exception: ++ util.logexc( ++ LOG, ++ "Unexpected error while trying to set " ++ + "guestinfo key=%s to value=%s", ++ key, ++ value, ++ ) ++ ++ return None ++ ++ ++def guestinfo_redact_keys(keys, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ guestinfo_redact_keys redacts guestinfo of all of the keys in the given ++ list. each key will have its value set to "---". Since the value is valid ++ YAML, cloud-init can still read it if it tries. ++ """ ++ if not keys: ++ return ++ if not type(keys) in (list, tuple): ++ keys = [keys] ++ for key in keys: ++ key_name = get_guestinfo_key_name(key) ++ LOG.info("clearing %s", key_name) ++ if not guestinfo_set_value( ++ key, GUESTINFO_EMPTY_YAML_VAL, vmware_rpctool ++ ): ++ LOG.error("failed to clear %s", key_name) ++ LOG.info("clearing %s.encoding", key_name) ++ if not guestinfo_set_value(key + ".encoding", "", vmware_rpctool): ++ LOG.error("failed to clear %s.encoding", key_name) ++ ++ ++def load_json_or_yaml(data): ++ """ ++ load first attempts to unmarshal the provided data as JSON, and if ++ that fails then attempts to unmarshal the data as YAML. If data is ++ None then a new dictionary is returned. ++ """ ++ if not data: ++ return {} ++ try: ++ return util.load_json(data) ++ except (json.JSONDecodeError, TypeError): ++ return util.load_yaml(data) ++ ++ ++def process_metadata(data): ++ """ ++ process_metadata processes metadata and loads the optional network ++ configuration. ++ """ ++ network = None ++ if "network" in data: ++ network = data["network"] ++ del data["network"] ++ ++ network_enc = None ++ if "network.encoding" in data: ++ network_enc = data["network.encoding"] ++ del data["network.encoding"] ++ ++ if network: ++ if isinstance(network, collections.abc.Mapping): ++ LOG.debug("network data copied to 'config' key") ++ network = {"config": copy.deepcopy(network)} ++ else: ++ LOG.debug("network data to be decoded %s", network) ++ dec_net = decode("metadata.network", network_enc, network) ++ network = { ++ "config": load_json_or_yaml(dec_net), ++ } ++ ++ LOG.debug("network data %s", network) ++ data["network"] = network ++ ++ return data ++ ++ ++# Used to match classes to dependencies ++datasources = [ ++ (DataSourceVMware, (sources.DEP_FILESYSTEM,)), # Run at init-local ++ (DataSourceVMware, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)), ++] ++ ++ ++def get_datasource_list(depends): ++ """ ++ Return a list of data sources that match this set of dependencies ++ """ ++ return sources.list_from_depends(depends, datasources) ++ ++ ++def get_default_ip_addrs(): ++ """ ++ Returns the default IPv4 and IPv6 addresses based on the device(s) used for ++ the default route. Please note that None may be returned for either address ++ family if that family has no default route or if there are multiple ++ addresses associated with the device used by the default route for a given ++ address. ++ """ ++ # TODO(promote and use netifaces in cloudinit.net* modules) ++ gateways = netifaces.gateways() ++ if "default" not in gateways: ++ return None, None ++ ++ default_gw = gateways["default"] ++ if ( ++ netifaces.AF_INET not in default_gw ++ and netifaces.AF_INET6 not in default_gw ++ ): ++ return None, None ++ ++ ipv4 = None ++ ipv6 = None ++ ++ gw4 = default_gw.get(netifaces.AF_INET) ++ if gw4: ++ _, dev4 = gw4 ++ addr4_fams = netifaces.ifaddresses(dev4) ++ if addr4_fams: ++ af_inet4 = addr4_fams.get(netifaces.AF_INET) ++ if af_inet4: ++ if len(af_inet4) > 1: ++ LOG.warning( ++ "device %s has more than one ipv4 address: %s", ++ dev4, ++ af_inet4, ++ ) ++ elif "addr" in af_inet4[0]: ++ ipv4 = af_inet4[0]["addr"] ++ ++ # Try to get the default IPv6 address by first seeing if there is a default ++ # IPv6 route. ++ gw6 = default_gw.get(netifaces.AF_INET6) ++ if gw6: ++ _, dev6 = gw6 ++ addr6_fams = netifaces.ifaddresses(dev6) ++ if addr6_fams: ++ af_inet6 = addr6_fams.get(netifaces.AF_INET6) ++ if af_inet6: ++ if len(af_inet6) > 1: ++ LOG.warning( ++ "device %s has more than one ipv6 address: %s", ++ dev6, ++ af_inet6, ++ ) ++ elif "addr" in af_inet6[0]: ++ ipv6 = af_inet6[0]["addr"] ++ ++ # If there is a default IPv4 address but not IPv6, then see if there is a ++ # single IPv6 address associated with the same device associated with the ++ # default IPv4 address. ++ if ipv4 and not ipv6: ++ af_inet6 = addr4_fams.get(netifaces.AF_INET6) ++ if af_inet6: ++ if len(af_inet6) > 1: ++ LOG.warning( ++ "device %s has more than one ipv6 address: %s", ++ dev4, ++ af_inet6, ++ ) ++ elif "addr" in af_inet6[0]: ++ ipv6 = af_inet6[0]["addr"] ++ ++ # If there is a default IPv6 address but not IPv4, then see if there is a ++ # single IPv4 address associated with the same device associated with the ++ # default IPv6 address. ++ if not ipv4 and ipv6: ++ af_inet4 = addr6_fams.get(netifaces.AF_INET) ++ if af_inet4: ++ if len(af_inet4) > 1: ++ LOG.warning( ++ "device %s has more than one ipv4 address: %s", ++ dev6, ++ af_inet4, ++ ) ++ elif "addr" in af_inet4[0]: ++ ipv4 = af_inet4[0]["addr"] ++ ++ return ipv4, ipv6 ++ ++ ++# patched socket.getfqdn() - see https://bugs.python.org/issue5004 ++ ++ ++def getfqdn(name=""): ++ """Get fully qualified domain name from name. ++ An empty argument is interpreted as meaning the local host. ++ """ ++ # TODO(may want to promote this function to util.getfqdn) ++ # TODO(may want to extend util.get_hostname to accept fqdn=True param) ++ name = name.strip() ++ if not name or name == "0.0.0.0": ++ name = util.get_hostname() ++ try: ++ addrs = socket.getaddrinfo( ++ name, None, 0, socket.SOCK_DGRAM, 0, socket.AI_CANONNAME ++ ) ++ except socket.error: ++ pass ++ else: ++ for addr in addrs: ++ if addr[3]: ++ name = addr[3] ++ break ++ return name ++ ++ ++def is_valid_ip_addr(val): ++ """ ++ Returns false if the address is loopback, link local or unspecified; ++ otherwise true is returned. ++ """ ++ # TODO(extend cloudinit.net.is_ip_addr exclude link_local/loopback etc) ++ # TODO(migrate to use cloudinit.net.is_ip_addr)# ++ ++ addr = None ++ try: ++ addr = ipaddress.ip_address(val) ++ except ipaddress.AddressValueError: ++ addr = ipaddress.ip_address(str(val)) ++ except Exception: ++ return None ++ ++ if addr.is_link_local or addr.is_loopback or addr.is_unspecified: ++ return False ++ return True ++ ++ ++def get_host_info(): ++ """ ++ Returns host information such as the host name and network interfaces. ++ """ ++ # TODO(look to promote netifices use up in cloud-init netinfo funcs) ++ host_info = { ++ "network": { ++ "interfaces": { ++ "by-mac": collections.OrderedDict(), ++ "by-ipv4": collections.OrderedDict(), ++ "by-ipv6": collections.OrderedDict(), ++ }, ++ }, ++ } ++ hostname = getfqdn(util.get_hostname()) ++ if hostname: ++ host_info["hostname"] = hostname ++ host_info["local-hostname"] = hostname ++ host_info["local_hostname"] = hostname ++ ++ default_ipv4, default_ipv6 = get_default_ip_addrs() ++ if default_ipv4: ++ host_info[LOCAL_IPV4] = default_ipv4 ++ if default_ipv6: ++ host_info[LOCAL_IPV6] = default_ipv6 ++ ++ by_mac = host_info["network"]["interfaces"]["by-mac"] ++ by_ipv4 = host_info["network"]["interfaces"]["by-ipv4"] ++ by_ipv6 = host_info["network"]["interfaces"]["by-ipv6"] ++ ++ ifaces = netifaces.interfaces() ++ for dev_name in ifaces: ++ addr_fams = netifaces.ifaddresses(dev_name) ++ af_link = addr_fams.get(netifaces.AF_LINK) ++ af_inet4 = addr_fams.get(netifaces.AF_INET) ++ af_inet6 = addr_fams.get(netifaces.AF_INET6) ++ ++ mac = None ++ if af_link and "addr" in af_link[0]: ++ mac = af_link[0]["addr"] ++ ++ # Do not bother recording localhost ++ if mac == "00:00:00:00:00:00": ++ continue ++ ++ if mac and (af_inet4 or af_inet6): ++ key = mac ++ val = {} ++ if af_inet4: ++ af_inet4_vals = [] ++ for ip_info in af_inet4: ++ if not is_valid_ip_addr(ip_info["addr"]): ++ continue ++ af_inet4_vals.append(ip_info) ++ val["ipv4"] = af_inet4_vals ++ if af_inet6: ++ af_inet6_vals = [] ++ for ip_info in af_inet6: ++ if not is_valid_ip_addr(ip_info["addr"]): ++ continue ++ af_inet6_vals.append(ip_info) ++ val["ipv6"] = af_inet6_vals ++ by_mac[key] = val ++ ++ if af_inet4: ++ for ip_info in af_inet4: ++ key = ip_info["addr"] ++ if not is_valid_ip_addr(key): ++ continue ++ val = copy.deepcopy(ip_info) ++ del val["addr"] ++ if mac: ++ val["mac"] = mac ++ by_ipv4[key] = val ++ ++ if af_inet6: ++ for ip_info in af_inet6: ++ key = ip_info["addr"] ++ if not is_valid_ip_addr(key): ++ continue ++ val = copy.deepcopy(ip_info) ++ del val["addr"] ++ if mac: ++ val["mac"] = mac ++ by_ipv6[key] = val ++ ++ return host_info ++ ++ ++def wait_on_network(metadata): ++ # Determine whether we need to wait on the network coming online. ++ wait_on_ipv4 = False ++ wait_on_ipv6 = False ++ if WAIT_ON_NETWORK in metadata: ++ wait_on_network = metadata[WAIT_ON_NETWORK] ++ if WAIT_ON_NETWORK_IPV4 in wait_on_network: ++ wait_on_ipv4_val = wait_on_network[WAIT_ON_NETWORK_IPV4] ++ if isinstance(wait_on_ipv4_val, bool): ++ wait_on_ipv4 = wait_on_ipv4_val ++ else: ++ wait_on_ipv4 = util.translate_bool(wait_on_ipv4_val) ++ if WAIT_ON_NETWORK_IPV6 in wait_on_network: ++ wait_on_ipv6_val = wait_on_network[WAIT_ON_NETWORK_IPV6] ++ if isinstance(wait_on_ipv6_val, bool): ++ wait_on_ipv6 = wait_on_ipv6_val ++ else: ++ wait_on_ipv6 = util.translate_bool(wait_on_ipv6_val) ++ ++ # Get information about the host. ++ host_info = None ++ while host_info is None: ++ # This loop + sleep results in two logs every second while waiting ++ # for either ipv4 or ipv6 up. Do we really need to log each iteration ++ # or can we log once and log on successful exit? ++ host_info = get_host_info() ++ ++ network = host_info.get("network") or {} ++ interfaces = network.get("interfaces") or {} ++ by_ipv4 = interfaces.get("by-ipv4") or {} ++ by_ipv6 = interfaces.get("by-ipv6") or {} ++ ++ if wait_on_ipv4: ++ ipv4_ready = len(by_ipv4) > 0 if by_ipv4 else False ++ if not ipv4_ready: ++ host_info = None ++ ++ if wait_on_ipv6: ++ ipv6_ready = len(by_ipv6) > 0 if by_ipv6 else False ++ if not ipv6_ready: ++ host_info = None ++ ++ if host_info is None: ++ LOG.debug( ++ "waiting on network: wait4=%s, ready4=%s, wait6=%s, ready6=%s", ++ wait_on_ipv4, ++ ipv4_ready, ++ wait_on_ipv6, ++ ipv6_ready, ++ ) ++ time.sleep(1) ++ ++ LOG.debug("waiting on network complete") ++ return host_info ++ ++ ++def main(): ++ """ ++ Executed when this file is used as a program. ++ """ ++ try: ++ logging.setupBasicLogging() ++ except Exception: ++ pass ++ metadata = { ++ "wait-on-network": {"ipv4": True, "ipv6": "false"}, ++ "network": {"config": {"dhcp": True}}, ++ } ++ host_info = wait_on_network(metadata) ++ metadata = util.mergemanydict([metadata, host_info]) ++ print(util.json_dumps(metadata)) ++ ++ ++if __name__ == "__main__": ++ main() ++ ++# vi: ts=4 expandtab +diff --git a/doc/rtd/topics/availability.rst b/doc/rtd/topics/availability.rst +index f58b2b38..6606367c 100644 +--- a/doc/rtd/topics/availability.rst ++++ b/doc/rtd/topics/availability.rst +@@ -64,5 +64,6 @@ Additionally, cloud-init is supported on these private clouds: + - LXD + - KVM + - Metal-as-a-Service (MAAS) ++- VMware + + .. vi: textwidth=79 +diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst +index 228173d2..8afed470 100644 +--- a/doc/rtd/topics/datasources.rst ++++ b/doc/rtd/topics/datasources.rst +@@ -49,7 +49,7 @@ The following is a list of documents for each supported datasource: + datasources/smartos.rst + datasources/upcloud.rst + datasources/zstack.rst +- ++ datasources/vmware.rst + + Creation + ======== +diff --git a/doc/rtd/topics/datasources/vmware.rst b/doc/rtd/topics/datasources/vmware.rst +new file mode 100644 +index 00000000..996eb61f +--- /dev/null ++++ b/doc/rtd/topics/datasources/vmware.rst +@@ -0,0 +1,359 @@ ++.. _datasource_vmware: ++ ++VMware ++====== ++ ++This datasource is for use with systems running on a VMware platform such as ++vSphere and currently supports the following data transports: ++ ++ ++* `GuestInfo `_ keys ++ ++Configuration ++------------- ++ ++The configuration method is dependent upon the transport: ++ ++GuestInfo Keys ++^^^^^^^^^^^^^^ ++ ++One method of providing meta, user, and vendor data is by setting the following ++key/value pairs on a VM's ``extraConfig`` `property `_ : ++ ++.. list-table:: ++ :header-rows: 1 ++ ++ * - Property ++ - Description ++ * - ``guestinfo.metadata`` ++ - A YAML or JSON document containing the cloud-init metadata. ++ * - ``guestinfo.metadata.encoding`` ++ - The encoding type for ``guestinfo.metadata``. ++ * - ``guestinfo.userdata`` ++ - A YAML document containing the cloud-init user data. ++ * - ``guestinfo.userdata.encoding`` ++ - The encoding type for ``guestinfo.userdata``. ++ * - ``guestinfo.vendordata`` ++ - A YAML document containing the cloud-init vendor data. ++ * - ``guestinfo.vendordata.encoding`` ++ - The encoding type for ``guestinfo.vendordata``. ++ ++ ++All ``guestinfo.*.encoding`` values may be set to ``base64`` or ++``gzip+base64``. ++ ++Features ++-------- ++ ++This section reviews several features available in this datasource, regardless ++of how the meta, user, and vendor data was discovered. ++ ++Instance data and lazy networks ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++One of the hallmarks of cloud-init is `its use of instance-data and JINJA ++queries <../instancedata.html#using-instance-data>`_ ++-- the ability to write queries in user and vendor data that reference runtime ++information present in ``/run/cloud-init/instance-data.json``. This works well ++when the metadata provides all of the information up front, such as the network ++configuration. For systems that rely on DHCP, however, this information may not ++be available when the metadata is persisted to disk. ++ ++This datasource ensures that even if the instance is using DHCP to configure ++networking, the same details about the configured network are available in ++``/run/cloud-init/instance-data.json`` as if static networking was used. This ++information collected at runtime is easy to demonstrate by executing the ++datasource on the command line. From the root of this repository, run the ++following command: ++ ++.. code-block:: bash ++ ++ PYTHONPATH="$(pwd)" python3 cloudinit/sources/DataSourceVMware.py ++ ++The above command will result in output similar to the below JSON: ++ ++.. code-block:: json ++ ++ { ++ "hostname": "akutz.localhost", ++ "local-hostname": "akutz.localhost", ++ "local-ipv4": "192.168.0.188", ++ "local_hostname": "akutz.localhost", ++ "network": { ++ "config": { ++ "dhcp": true ++ }, ++ "interfaces": { ++ "by-ipv4": { ++ "172.0.0.2": { ++ "netmask": "255.255.255.255", ++ "peer": "172.0.0.2" ++ }, ++ "192.168.0.188": { ++ "broadcast": "192.168.0.255", ++ "mac": "64:4b:f0:18:9a:21", ++ "netmask": "255.255.255.0" ++ } ++ }, ++ "by-ipv6": { ++ "fd8e:d25e:c5b6:1:1f5:b2fd:8973:22f2": { ++ "flags": 208, ++ "mac": "64:4b:f0:18:9a:21", ++ "netmask": "ffff:ffff:ffff:ffff::/64" ++ } ++ }, ++ "by-mac": { ++ "64:4b:f0:18:9a:21": { ++ "ipv4": [ ++ { ++ "addr": "192.168.0.188", ++ "broadcast": "192.168.0.255", ++ "netmask": "255.255.255.0" ++ } ++ ], ++ "ipv6": [ ++ { ++ "addr": "fd8e:d25e:c5b6:1:1f5:b2fd:8973:22f2", ++ "flags": 208, ++ "netmask": "ffff:ffff:ffff:ffff::/64" ++ } ++ ] ++ }, ++ "ac:de:48:00:11:22": { ++ "ipv6": [] ++ } ++ } ++ } ++ }, ++ "wait-on-network": { ++ "ipv4": true, ++ "ipv6": "false" ++ } ++ } ++ ++ ++Redacting sensitive information ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++Sometimes the cloud-init userdata might contain sensitive information, and it ++may be desirable to have the ``guestinfo.userdata`` key (or other guestinfo ++keys) redacted as soon as its data is read by the datasource. This is possible ++by adding the following to the metadata: ++ ++.. code-block:: yaml ++ ++ redact: # formerly named cleanup-guestinfo, which will also work ++ - userdata ++ - vendordata ++ ++When the above snippet is added to the metadata, the datasource will iterate ++over the elements in the ``redact`` array and clear each of the keys. For ++example, when the guestinfo transport is used, the above snippet will cause ++the following commands to be executed: ++ ++.. code-block:: shell ++ ++ vmware-rpctool "info-set guestinfo.userdata ---" ++ vmware-rpctool "info-set guestinfo.userdata.encoding " ++ vmware-rpctool "info-set guestinfo.vendordata ---" ++ vmware-rpctool "info-set guestinfo.vendordata.encoding " ++ ++Please note that keys are set to the valid YAML string ``---`` as it is not ++possible remove an existing key from the guestinfo key-space. A key's analogous ++encoding property will be set to a single white-space character, causing the ++datasource to treat the actual key value as plain-text, thereby loading it as ++an empty YAML doc (hence the aforementioned ``---``\ ). ++ ++Reading the local IP addresses ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++This datasource automatically discovers the local IPv4 and IPv6 addresses for ++a guest operating system based on the default routes. However, when inspecting ++a VM externally, it's not possible to know what the *default* IP address is for ++the guest OS. That's why this datasource sets the discovered, local IPv4 and ++IPv6 addresses back in the guestinfo namespace as the following keys: ++ ++ ++* ``guestinfo.local-ipv4`` ++* ``guestinfo.local-ipv6`` ++ ++It is possible that a host may not have any default, local IP addresses. It's ++also possible the reported, local addresses are link-local addresses. But these ++two keys may be used to discover what this datasource determined were the local ++IPv4 and IPv6 addresses for a host. ++ ++Waiting on the network ++^^^^^^^^^^^^^^^^^^^^^^ ++ ++Sometimes cloud-init may bring up the network, but it will not finish coming ++online before the datasource's ``setup`` function is called, resulting in an ++``/var/run/cloud-init/instance-data.json`` file that does not have the correct ++network information. It is possible to instruct the datasource to wait until an ++IPv4 or IPv6 address is available before writing the instance data with the ++following metadata properties: ++ ++.. code-block:: yaml ++ ++ wait-on-network: ++ ipv4: true ++ ipv6: true ++ ++If either of the above values are true, then the datasource will sleep for a ++second, check the network status, and repeat until one or both addresses from ++the specified families are available. ++ ++Walkthrough ++----------- ++ ++The following series of steps is a demonstration on how to configure a VM with ++this datasource: ++ ++ ++#. Create the metadata file for the VM. Save the following YAML to a file named ++ ``metadata.yaml``\ : ++ ++ .. code-block:: yaml ++ ++ instance-id: cloud-vm ++ local-hostname: cloud-vm ++ network: ++ version: 2 ++ ethernets: ++ nics: ++ match: ++ name: ens* ++ dhcp4: yes ++ ++#. Create the userdata file ``userdata.yaml``\ : ++ ++ .. code-block:: yaml ++ ++ #cloud-config ++ ++ users: ++ - default ++ - name: akutz ++ primary_group: akutz ++ sudo: ALL=(ALL) NOPASSWD:ALL ++ groups: sudo, wheel ++ ssh_import_id: None ++ lock_passwd: true ++ ssh_authorized_keys: ++ - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDE0c5FczvcGSh/tG4iw+Fhfi/O5/EvUM/96js65tly4++YTXK1d9jcznPS5ruDlbIZ30oveCBd3kT8LLVFwzh6hepYTf0YmCTpF4eDunyqmpCXDvVscQYRXyasEm5olGmVe05RrCJSeSShAeptv4ueIn40kZKOghinGWLDSZG4+FFfgrmcMCpx5YSCtX2gvnEYZJr0czt4rxOZuuP7PkJKgC/mt2PcPjooeX00vAj81jjU2f3XKrjjz2u2+KIt9eba+vOQ6HiC8c2IzRkUAJ5i1atLy8RIbejo23+0P4N2jjk17QySFOVHwPBDTYb0/0M/4ideeU74EN/CgVsvO6JrLsPBR4dojkV5qNbMNxIVv5cUwIy2ThlLgqpNCeFIDLCWNZEFKlEuNeSQ2mPtIO7ETxEL2Cz5y/7AIuildzYMc6wi2bofRC8HmQ7rMXRWdwLKWsR0L7SKjHblIwarxOGqLnUI+k2E71YoP7SZSlxaKi17pqkr0OMCF+kKqvcvHAQuwGqyumTEWOlH6TCx1dSPrW+pVCZSHSJtSTfDW2uzL6y8k10MT06+pVunSrWo5LHAXcS91htHV1M1UrH/tZKSpjYtjMb5+RonfhaFRNzvj7cCE1f3Kp8UVqAdcGBTtReoE8eRUT63qIxjw03a7VwAyB2w+9cu1R9/vAo8SBeRqw== sakutz@gmail.com ++ ++#. Please note this step requires that the VM be powered off. All of the ++ commands below use the VMware CLI tool, `govc `_. ++ ++ Go ahead and assign the path to the VM to the environment variable ``VM``\ : ++ ++ .. code-block:: shell ++ ++ export VM="/inventory/path/to/the/vm" ++ ++#. Power off the VM: ++ ++ .. raw:: html ++ ++
++ ++ ⚠️ First Boot Mode ++ ++ To ensure the next power-on operation results in a first-boot scenario for ++ cloud-init, it may be necessary to run the following command just before ++ powering off the VM: ++ ++ .. code-block:: bash ++ ++ cloud-init clean ++ ++ Otherwise cloud-init may not run in first-boot mode. For more information ++ on how the boot mode is determined, please see the ++ `First Boot Documentation <../boot.html#first-boot-determination>`_. ++ ++ .. raw:: html ++ ++
++ ++ .. code-block:: shell ++ ++ govc vm.power -off "${VM}" ++ ++#. ++ Export the environment variables that contain the cloud-init metadata and ++ userdata: ++ ++ .. code-block:: shell ++ ++ export METADATA=$(gzip -c9 /dev/null || base64; }) \ ++ USERDATA=$(gzip -c9 /dev/null || base64; }) ++ ++#. ++ Assign the metadata and userdata to the VM: ++ ++ .. code-block:: shell ++ ++ govc vm.change -vm "${VM}" \ ++ -e guestinfo.metadata="${METADATA}" \ ++ -e guestinfo.metadata.encoding="gzip+base64" \ ++ -e guestinfo.userdata="${USERDATA}" \ ++ -e guestinfo.userdata.encoding="gzip+base64" ++ ++ Please note the above commands include specifying the encoding for the ++ properties. This is important as it informs the datasource how to decode ++ the data for cloud-init. Valid values for ``metadata.encoding`` and ++ ``userdata.encoding`` include: ++ ++ ++ * ``base64`` ++ * ``gzip+base64`` ++ ++#. ++ Power on the VM: ++ ++ .. code-block:: shell ++ ++ govc vm.power -vm "${VM}" -on ++ ++If all went according to plan, the CentOS box is: ++ ++* Locked down, allowing SSH access only for the user in the userdata ++* Configured for a dynamic IP address via DHCP ++* Has a hostname of ``cloud-vm`` ++ ++Examples ++-------- ++ ++This section reviews common configurations: ++ ++Setting the hostname ++^^^^^^^^^^^^^^^^^^^^ ++ ++The hostname is set by way of the metadata key ``local-hostname``. ++ ++Setting the instance ID ++^^^^^^^^^^^^^^^^^^^^^^^ ++ ++The instance ID may be set by way of the metadata key ``instance-id``. However, ++if this value is absent then then the instance ID is read from the file ++``/sys/class/dmi/id/product_uuid``. ++ ++Providing public SSH keys ++^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++The public SSH keys may be set by way of the metadata key ``public-keys-data``. ++Each newline-terminated string will be interpreted as a separate SSH public ++key, which will be placed in distro's default user's ++``~/.ssh/authorized_keys``. If the value is empty or absent, then nothing will ++be written to ``~/.ssh/authorized_keys``. ++ ++Configuring the network ++^^^^^^^^^^^^^^^^^^^^^^^ ++ ++The network is configured by setting the metadata key ``network`` with a value ++consistent with Network Config Versions ++`1 <../network-config-format-v1.html>`_ or ++`2 <../network-config-format-v2.html>`_\ , depending on the Linux ++distro's version of cloud-init. ++ ++The metadata key ``network.encoding`` may be used to indicate the format of ++the metadata key "network". Valid encodings are ``base64`` and ``gzip+base64``. +diff --git a/requirements.txt b/requirements.txt +index 5817da3b..41d01d62 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -32,3 +32,12 @@ jsonpatch + + # For validating cloud-config sections per schema definitions + jsonschema ++ ++# Used by DataSourceVMware to inspect the host's network configuration during ++# the "setup()" function. ++# ++# This allows a host that uses DHCP to bring up the network during BootLocal ++# and still participate in instance-data by gathering the network in detail at ++# runtime and merge that information into the metadata and repersist that to ++# disk. ++netifaces>=0.10.9 +diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py +index 5912f7ee..475a2cf8 100644 +--- a/tests/unittests/test_datasource/test_common.py ++++ b/tests/unittests/test_datasource/test_common.py +@@ -28,6 +28,7 @@ from cloudinit.sources import ( + DataSourceScaleway as Scaleway, + DataSourceSmartOS as SmartOS, + DataSourceUpCloud as UpCloud, ++ DataSourceVMware as VMware, + ) + from cloudinit.sources import DataSourceNone as DSNone + +@@ -50,6 +51,7 @@ DEFAULT_LOCAL = [ + RbxCloud.DataSourceRbxCloud, + Scaleway.DataSourceScaleway, + UpCloud.DataSourceUpCloudLocal, ++ VMware.DataSourceVMware, + ] + + DEFAULT_NETWORK = [ +@@ -66,6 +68,7 @@ DEFAULT_NETWORK = [ + OpenStack.DataSourceOpenStack, + OVF.DataSourceOVFNet, + UpCloud.DataSourceUpCloud, ++ VMware.DataSourceVMware, + ] + + +diff --git a/tests/unittests/test_datasource/test_vmware.py b/tests/unittests/test_datasource/test_vmware.py +new file mode 100644 +index 00000000..597db7c8 +--- /dev/null ++++ b/tests/unittests/test_datasource/test_vmware.py +@@ -0,0 +1,377 @@ ++# Copyright (c) 2021 VMware, Inc. All Rights Reserved. ++# ++# Authors: Andrew Kutz ++# ++# This file is part of cloud-init. See LICENSE file for license information. ++ ++import base64 ++import gzip ++from cloudinit import dmi, helpers, safeyaml ++from cloudinit import settings ++from cloudinit.sources import DataSourceVMware ++from cloudinit.tests.helpers import ( ++ mock, ++ CiTestCase, ++ FilesystemMockingTestCase, ++ populate_dir, ++) ++ ++import os ++ ++PRODUCT_NAME_FILE_PATH = "/sys/class/dmi/id/product_name" ++PRODUCT_NAME = "VMware7,1" ++PRODUCT_UUID = "82343CED-E4C7-423B-8F6B-0D34D19067AB" ++REROOT_FILES = { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ PRODUCT_NAME_FILE_PATH: PRODUCT_NAME, ++} ++ ++VMW_MULTIPLE_KEYS = [ ++ "ssh-rsa AAAAB3NzaC1yc2EAAAA... test1@vmw.com", ++ "ssh-rsa AAAAB3NzaC1yc2EAAAA... test2@vmw.com", ++] ++VMW_SINGLE_KEY = "ssh-rsa AAAAB3NzaC1yc2EAAAA... test@vmw.com" ++ ++VMW_METADATA_YAML = """instance-id: cloud-vm ++local-hostname: cloud-vm ++network: ++ version: 2 ++ ethernets: ++ nics: ++ match: ++ name: ens* ++ dhcp4: yes ++""" ++ ++VMW_USERDATA_YAML = """## template: jinja ++#cloud-config ++users: ++- default ++""" ++ ++VMW_VENDORDATA_YAML = """## template: jinja ++#cloud-config ++runcmd: ++- echo "Hello, world." ++""" ++ ++ ++class TestDataSourceVMware(CiTestCase): ++ """ ++ Test common functionality that is not transport specific. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMware, self).setUp() ++ self.tmp = self.tmp_dir() ++ ++ def test_no_data_access_method(self): ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = None ++ ret = ds.get_data() ++ self.assertFalse(ret) ++ ++ def test_get_host_info(self): ++ host_info = DataSourceVMware.get_host_info() ++ self.assertTrue(host_info) ++ self.assertTrue(host_info["hostname"]) ++ self.assertTrue(host_info["local-hostname"]) ++ self.assertTrue(host_info["local_hostname"]) ++ self.assertTrue(host_info[DataSourceVMware.LOCAL_IPV4]) ++ ++ ++class TestDataSourceVMwareEnvVars(FilesystemMockingTestCase): ++ """ ++ Test the envvar transport. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMwareEnvVars, self).setUp() ++ self.tmp = self.tmp_dir() ++ os.environ[DataSourceVMware.VMX_GUESTINFO] = "1" ++ self.create_system_files() ++ ++ def tearDown(self): ++ del os.environ[DataSourceVMware.VMX_GUESTINFO] ++ return super(TestDataSourceVMwareEnvVars, self).tearDown() ++ ++ def create_system_files(self): ++ rootd = self.tmp_dir() ++ populate_dir( ++ rootd, ++ { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ }, ++ ) ++ self.assertTrue(self.reRoot(rootd)) ++ ++ def assert_get_data_ok(self, m_fn, m_fn_call_count=6): ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = None ++ ret = ds.get_data() ++ self.assertTrue(ret) ++ self.assertEqual(m_fn_call_count, m_fn.call_count) ++ self.assertEqual( ++ ds.data_access_method, DataSourceVMware.DATA_ACCESS_METHOD_ENVVAR ++ ) ++ return ds ++ ++ def assert_metadata(self, metadata, m_fn, m_fn_call_count=6): ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count) ++ assert_metadata(self, ds, metadata) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_subplatform(self, m_fn): ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ self.assertEqual( ++ ds.subplatform, ++ "%s (%s)" ++ % ( ++ DataSourceVMware.DATA_ACCESS_METHOD_ENVVAR, ++ DataSourceVMware.get_guestinfo_envvar_key_name("metadata"), ++ ), ++ ) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_only(self, m_fn): ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_userdata_only(self, m_fn): ++ m_fn.side_effect = ["", VMW_USERDATA_YAML, "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_vendordata_only(self, m_fn): ++ m_fn.side_effect = ["", "", VMW_VENDORDATA_YAML, ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_base64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_b64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_gzip_base64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gzip+base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_gz_b64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gz+b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_metadata_single_ssh_key(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_SINGLE_KEY ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_metadata_multiple_ssh_keys(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_MULTIPLE_KEYS ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ ++class TestDataSourceVMwareGuestInfo(FilesystemMockingTestCase): ++ """ ++ Test the guestinfo transport on a VMware platform. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMwareGuestInfo, self).setUp() ++ self.tmp = self.tmp_dir() ++ self.create_system_files() ++ ++ def create_system_files(self): ++ rootd = self.tmp_dir() ++ populate_dir( ++ rootd, ++ { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ PRODUCT_NAME_FILE_PATH: PRODUCT_NAME, ++ }, ++ ) ++ self.assertTrue(self.reRoot(rootd)) ++ ++ def assert_get_data_ok(self, m_fn, m_fn_call_count=6): ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = "vmware-rpctool" ++ ret = ds.get_data() ++ self.assertTrue(ret) ++ self.assertEqual(m_fn_call_count, m_fn.call_count) ++ self.assertEqual( ++ ds.data_access_method, ++ DataSourceVMware.DATA_ACCESS_METHOD_GUESTINFO, ++ ) ++ return ds ++ ++ def assert_metadata(self, metadata, m_fn, m_fn_call_count=6): ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count) ++ assert_metadata(self, ds, metadata) ++ ++ def test_ds_valid_on_vmware_platform(self): ++ system_type = dmi.read_dmi_data("system-product-name") ++ self.assertEqual(system_type, PRODUCT_NAME) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_subplatform(self, m_fn): ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ self.assertEqual( ++ ds.subplatform, ++ "%s (%s)" ++ % ( ++ DataSourceVMware.DATA_ACCESS_METHOD_GUESTINFO, ++ DataSourceVMware.get_guestinfo_key_name("metadata"), ++ ), ++ ) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_userdata_only(self, m_fn): ++ m_fn.side_effect = ["", VMW_USERDATA_YAML, "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_vendordata_only(self, m_fn): ++ m_fn.side_effect = ["", "", VMW_VENDORDATA_YAML, ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_metadata_single_ssh_key(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_SINGLE_KEY ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_metadata_multiple_ssh_keys(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_MULTIPLE_KEYS ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_base64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_b64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_gzip_base64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gzip+base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_gz_b64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gz+b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ ++class TestDataSourceVMwareGuestInfo_InvalidPlatform(FilesystemMockingTestCase): ++ """ ++ Test the guestinfo transport on a non-VMware platform. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMwareGuestInfo_InvalidPlatform, self).setUp() ++ self.tmp = self.tmp_dir() ++ self.create_system_files() ++ ++ def create_system_files(self): ++ rootd = self.tmp_dir() ++ populate_dir( ++ rootd, ++ { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ }, ++ ) ++ self.assertTrue(self.reRoot(rootd)) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_ds_invalid_on_non_vmware_platform(self, m_fn): ++ system_type = dmi.read_dmi_data("system-product-name") ++ self.assertEqual(system_type, None) ++ ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = "vmware-rpctool" ++ ret = ds.get_data() ++ self.assertFalse(ret) ++ ++ ++def assert_metadata(test_obj, ds, metadata): ++ test_obj.assertEqual(metadata.get("instance-id"), ds.get_instance_id()) ++ test_obj.assertEqual(metadata.get("local-hostname"), ds.get_hostname()) ++ ++ expected_public_keys = metadata.get("public_keys") ++ if not isinstance(expected_public_keys, list): ++ expected_public_keys = [expected_public_keys] ++ ++ test_obj.assertEqual(expected_public_keys, ds.get_public_ssh_keys()) ++ test_obj.assertIsInstance(ds.get_public_ssh_keys(), list) ++ ++ ++def get_ds(temp_dir): ++ ds = DataSourceVMware.DataSourceVMware( ++ settings.CFG_BUILTIN, None, helpers.Paths({"run_dir": temp_dir}) ++ ) ++ ds.vmware_rpctool = "vmware-rpctool" ++ return ds ++ ++ ++# vi: ts=4 expandtab +diff --git a/tests/unittests/test_ds_identify.py b/tests/unittests/test_ds_identify.py +index 1d8aaf18..8617d7bd 100644 +--- a/tests/unittests/test_ds_identify.py ++++ b/tests/unittests/test_ds_identify.py +@@ -649,6 +649,50 @@ class TestDsIdentify(DsIdentifyBase): + """EC2: bobrightbox.com in product_serial is not brightbox'""" + self._test_ds_not_found('Ec2-E24Cloud-negative') + ++ def test_vmware_no_valid_transports(self): ++ """VMware: no valid transports""" ++ self._test_ds_not_found('VMware-NoValidTransports') ++ ++ def test_vmware_envvar_no_data(self): ++ """VMware: envvar transport no data""" ++ self._test_ds_not_found('VMware-EnvVar-NoData') ++ ++ def test_vmware_envvar_no_virt_id(self): ++ """VMware: envvar transport success if no virt id""" ++ self._test_ds_found('VMware-EnvVar-NoVirtID') ++ ++ def test_vmware_envvar_activated_by_metadata(self): ++ """VMware: envvar transport activated by metadata""" ++ self._test_ds_found('VMware-EnvVar-Metadata') ++ ++ def test_vmware_envvar_activated_by_userdata(self): ++ """VMware: envvar transport activated by userdata""" ++ self._test_ds_found('VMware-EnvVar-Userdata') ++ ++ def test_vmware_envvar_activated_by_vendordata(self): ++ """VMware: envvar transport activated by vendordata""" ++ self._test_ds_found('VMware-EnvVar-Vendordata') ++ ++ def test_vmware_guestinfo_no_data(self): ++ """VMware: guestinfo transport no data""" ++ self._test_ds_not_found('VMware-GuestInfo-NoData') ++ ++ def test_vmware_guestinfo_no_virt_id(self): ++ """VMware: guestinfo transport fails if no virt id""" ++ self._test_ds_not_found('VMware-GuestInfo-NoVirtID') ++ ++ def test_vmware_guestinfo_activated_by_metadata(self): ++ """VMware: guestinfo transport activated by metadata""" ++ self._test_ds_found('VMware-GuestInfo-Metadata') ++ ++ def test_vmware_guestinfo_activated_by_userdata(self): ++ """VMware: guestinfo transport activated by userdata""" ++ self._test_ds_found('VMware-GuestInfo-Userdata') ++ ++ def test_vmware_guestinfo_activated_by_vendordata(self): ++ """VMware: guestinfo transport activated by vendordata""" ++ self._test_ds_found('VMware-GuestInfo-Vendordata') ++ + + class TestBSDNoSys(DsIdentifyBase): + """Test *BSD code paths +@@ -1136,7 +1180,240 @@ VALID_CFG = { + 'Ec2-E24Cloud-negative': { + 'ds': 'Ec2', + 'files': {P_SYS_VENDOR: 'e24cloudyday\n'}, +- } ++ }, ++ 'VMware-NoValidTransports': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-NoData': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-NoVirtID': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ ], ++ }, ++ 'VMware-EnvVar-Metadata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-Userdata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-Vendordata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 0, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-NoData': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-NoVirtID': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ ], ++ }, ++ 'VMware-GuestInfo-Metadata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-Userdata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-Vendordata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, + } + + # vi: ts=4 expandtab +diff --git a/tools/.github-cla-signers b/tools/.github-cla-signers +index 689d7902..cbfa883c 100644 +--- a/tools/.github-cla-signers ++++ b/tools/.github-cla-signers +@@ -1,5 +1,6 @@ + ader1990 + ajmyyra ++akutz + AlexBaranowski + Aman306 + andrewbogott +diff --git a/tools/ds-identify b/tools/ds-identify +index 2f2486f7..c01eae3d 100755 +--- a/tools/ds-identify ++++ b/tools/ds-identify +@@ -125,7 +125,7 @@ DI_DSNAME="" + # be searched if there is no setting found in config. + DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \ + CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \ +-OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud" ++OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud VMware" + DI_DSLIST="" + DI_MODE="" + DI_ON_FOUND="" +@@ -1350,6 +1350,80 @@ dscheck_IBMCloud() { + return ${DS_NOT_FOUND} + } + ++vmware_has_envvar_vmx_guestinfo() { ++ [ -n "${VMX_GUESTINFO:-}" ] ++} ++ ++vmware_has_envvar_vmx_guestinfo_metadata() { ++ [ -n "${VMX_GUESTINFO_METADATA:-}" ] ++} ++ ++vmware_has_envvar_vmx_guestinfo_userdata() { ++ [ -n "${VMX_GUESTINFO_USERDATA:-}" ] ++} ++ ++vmware_has_envvar_vmx_guestinfo_vendordata() { ++ [ -n "${VMX_GUESTINFO_VENDORDATA:-}" ] ++} ++ ++vmware_has_rpctool() { ++ command -v vmware-rpctool >/dev/null 2>&1 ++} ++ ++vmware_rpctool_guestinfo_metadata() { ++ vmware-rpctool "info-get guestinfo.metadata" ++} ++ ++vmware_rpctool_guestinfo_userdata() { ++ vmware-rpctool "info-get guestinfo.userdata" ++} ++ ++vmware_rpctool_guestinfo_vendordata() { ++ vmware-rpctool "info-get guestinfo.vendordata" ++} ++ ++dscheck_VMware() { ++ # Checks to see if there is valid data for the VMware datasource. ++ # The data transports are checked in the following order: ++ # ++ # * envvars ++ # * guestinfo ++ # ++ # Please note when updating this function with support for new data ++ # transports, the order should match the order in the _get_data ++ # function from the file DataSourceVMware.py. ++ ++ # Check to see if running in a container and the VMware ++ # datasource is configured via environment variables. ++ if vmware_has_envvar_vmx_guestinfo; then ++ if vmware_has_envvar_vmx_guestinfo_metadata || \ ++ vmware_has_envvar_vmx_guestinfo_userdata || \ ++ vmware_has_envvar_vmx_guestinfo_vendordata; then ++ return "${DS_FOUND}" ++ fi ++ fi ++ ++ # Do not proceed unless the detected platform is VMware. ++ if [ ! "${DI_VIRT}" = "vmware" ]; then ++ return "${DS_NOT_FOUND}" ++ fi ++ ++ # Do not proceed if the vmware-rpctool command is not present. ++ if ! vmware_has_rpctool; then ++ return "${DS_NOT_FOUND}" ++ fi ++ ++ # Activate the VMware datasource only if any of the fields used ++ # by the datasource are present in the guestinfo table. ++ if { vmware_rpctool_guestinfo_metadata || \ ++ vmware_rpctool_guestinfo_userdata || \ ++ vmware_rpctool_guestinfo_vendordata; } >/dev/null 2>&1; then ++ return "${DS_FOUND}" ++ fi ++ ++ return "${DS_NOT_FOUND}" ++} ++ + collect_info() { + read_uname_info + read_virt +-- +2.27.0 + diff --git a/SOURCES/ci-Revert-unnecesary-lcase-in-ds-identify-978.patch b/SOURCES/ci-Revert-unnecesary-lcase-in-ds-identify-978.patch new file mode 100644 index 0000000..2e5349c --- /dev/null +++ b/SOURCES/ci-Revert-unnecesary-lcase-in-ds-identify-978.patch @@ -0,0 +1,47 @@ +From 532a36edf0dea2b98835bd08e285bec9c50eb0f9 Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito +Date: Fri, 14 Jan 2022 16:42:41 +0100 +Subject: [PATCH 5/6] Revert unnecesary lcase in ds-identify (#978) + +RH-Author: Emanuele Giuseppe Esposito +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [5/6] 95634e4b42e3abfb91182b090c312eef29c63e54 +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy +RH-Acked-by: Eduardo Otubo + +commit f516a7d37c1654addc02485e681b4358d7e7c0db +Author: Andrew Kutz <101085+akutz@users.noreply.github.com> +Date: Fri Aug 13 14:30:55 2021 -0500 + + Revert unnecesary lcase in ds-identify (#978) + + This patch reverts an unnecessary lcase optimization in the + ds-identify script. SystemD documents the values produced by + the systemd-detect-virt command are lower case, and the mapping + table used by the FreeBSD check is also lower-case. + + The optimization added two new forked processes, needlessly + causing overhead. + +Signed-off-by: Emanuele Giuseppe Esposito +--- + tools/ds-identify | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/tools/ds-identify b/tools/ds-identify +index 0e12298f..7b782462 100755 +--- a/tools/ds-identify ++++ b/tools/ds-identify +@@ -449,7 +449,7 @@ detect_virt() { + read_virt() { + cached "$DI_VIRT" && return 0 + detect_virt +- DI_VIRT="$(echo "${_RET}" | tr '[:upper:]' '[:lower:]')" ++ DI_VIRT="${_RET}" + } + + is_container() { +-- +2.27.0 + diff --git a/SOURCES/ci-Update-dscheck_VMware-s-rpctool-check-970.patch b/SOURCES/ci-Update-dscheck_VMware-s-rpctool-check-970.patch new file mode 100644 index 0000000..492f1c6 --- /dev/null +++ b/SOURCES/ci-Update-dscheck_VMware-s-rpctool-check-970.patch @@ -0,0 +1,97 @@ +From cc79cb3958b943b755a9b11b3e87ce820058ccaa Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito +Date: Fri, 14 Jan 2022 16:41:47 +0100 +Subject: [PATCH 4/6] Update dscheck_VMware's rpctool check (#970) + +RH-Author: Emanuele Giuseppe Esposito +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [4/6] 6f4d732c55c521869210d8aeedfa1150ea5a92f8 +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy +RH-Acked-by: Eduardo Otubo + +commit 7781dec3306e9467f216cfcb36b7e10a8b38547a +Author: Shreenidhi Shedi <53473811+sshedi@users.noreply.github.com> +Date: Fri Aug 13 00:40:39 2021 +0530 + + Update dscheck_VMware's rpctool check (#970) + + This patch updates the dscheck_VMware function's use of "vmware-rpctool". + + When checking to see if a "guestinfo" property is set. + Because a successful exit code can occur even if there is an empty + string returned, it is possible that the VMware datasource will be + loaded as a false-positive. This patch ensures that in addition to + validating the exit code, the emitted output is also examined to ensure + a non-empty value is returned by rpctool before returning "${DS_FOUND}" + from "dscheck_VMware()". + +Signed-off-by: Emanuele Giuseppe Esposito +--- + tools/ds-identify | 15 +++++++++------ + 1 file changed, 9 insertions(+), 6 deletions(-) + +diff --git a/tools/ds-identify b/tools/ds-identify +index c01eae3d..0e12298f 100755 +--- a/tools/ds-identify ++++ b/tools/ds-identify +@@ -141,6 +141,7 @@ error() { + debug 0 "$@" + stderr "$@" + } ++ + warn() { + set -- "WARN:" "$@" + debug 0 "$@" +@@ -344,7 +345,6 @@ geom_label_status_as() { + return $ret + } + +- + read_fs_info_freebsd() { + local oifs="$IFS" line="" delim="," + local ret=0 labels="" dev="" label="" ftype="" isodevs="" +@@ -404,7 +404,6 @@ cached() { + [ -n "$1" ] && _RET="$1" && return || return 1 + } + +- + detect_virt() { + local virt="${UNAVAILABLE}" r="" out="" + if [ -d /run/systemd ]; then +@@ -450,7 +449,7 @@ detect_virt() { + read_virt() { + cached "$DI_VIRT" && return 0 + detect_virt +- DI_VIRT=${_RET} ++ DI_VIRT="$(echo "${_RET}" | tr '[:upper:]' '[:lower:]')" + } + + is_container() { +@@ -1370,16 +1369,20 @@ vmware_has_rpctool() { + command -v vmware-rpctool >/dev/null 2>&1 + } + ++vmware_rpctool_guestinfo() { ++ vmware-rpctool "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]" ++} ++ + vmware_rpctool_guestinfo_metadata() { +- vmware-rpctool "info-get guestinfo.metadata" ++ vmware_rpctool_guestinfo "metadata" + } + + vmware_rpctool_guestinfo_userdata() { +- vmware-rpctool "info-get guestinfo.userdata" ++ vmware_rpctool_guestinfo "userdata" + } + + vmware_rpctool_guestinfo_vendordata() { +- vmware-rpctool "info-get guestinfo.vendordata" ++ vmware_rpctool_guestinfo "vendordata" + } + + dscheck_VMware() { +-- +2.27.0 + diff --git a/SPECS/cloud-init.spec b/SPECS/cloud-init.spec index 425df39..d6ae704 100644 --- a/SPECS/cloud-init.spec +++ b/SPECS/cloud-init.spec @@ -6,7 +6,7 @@ Name: cloud-init Version: 21.1 -Release: 7%{?dist}.3 +Release: 7%{?dist}.4 Summary: Cloud instance init scripts Group: System Environment/Base @@ -40,6 +40,18 @@ Patch15: ci-cc_ssh.py-fix-private-key-group-owner-and-permission.patch Patch16: ci-fix-error-on-upgrade-caused-by-new-vendordata2-attri.patch # For bz#2028756 - [RHEL-8] Above 19.2 of cloud-init fails to configure routes when configuring static and default routes to the same destination IP [rhel-8.5.0.z] Patch17: ci-cloudinit-net-handle-two-different-routes-for-the-sa.patch +# For bz#2040690 - [RHEL8] [Azure] cloud-init fails to configure the system [rhel-8.5.0.z] +#Patch18: ci-Add-gdisk-and-openssl-as-deps-to-fix-UEFI-Azure-init.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch19: ci-Datasource-for-VMware-953.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch20: ci-Change-netifaces-dependency-to-0.10.4-965.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch21: ci-Update-dscheck_VMware-s-rpctool-check-970.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch22: ci-Revert-unnecesary-lcase-in-ds-identify-978.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +#Patch23: ci-Add-netifaces-package-as-a-Requires-in-cloud-init.sp.patch BuildArch: noarch @@ -88,6 +100,10 @@ Requires: shadow-utils Requires: util-linux Requires: xfsprogs Requires: dhcp-client +# https://bugzilla.redhat.com/show_bug.cgi?id=2040690 +Requires: gdisk +Requires: openssl +Requires: python3-netifaces %{?systemd_requires} @@ -231,6 +247,18 @@ fi %config(noreplace) %{_sysconfdir}/rsyslog.d/21-cloudinit.conf %changelog +* Wed Jan 19 2022 Jon Maloy - 21.1-7.el8_5.4 +- ci-Add-gdisk-and-openssl-as-deps-to-fix-UEFI-Azure-init.patch [bz#2040690] +- ci-Datasource-for-VMware-953.patch [bz#2040704] +- ci-Change-netifaces-dependency-to-0.10.4-965.patch [bz#2040704] +- ci-Update-dscheck_VMware-s-rpctool-check-970.patch [bz#2040704] +- ci-Revert-unnecesary-lcase-in-ds-identify-978.patch [bz#2040704] +- ci-Add-netifaces-package-as-a-Requires-in-cloud-init.sp.patch [bz#2040704] +- Resolves: bz#2040690 + ([RHEL8] [Azure] cloud-init fails to configure the system [rhel-8.5.0.z]) +- Resolves: bz#2040704 + ([cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z]) + * Wed Dec 08 2021 Jon Maloy - 21.1-7.el8_5.3 - ci-cloudinit-net-handle-two-different-routes-for-the-sa.patch [bz#2028756] - Resolves: bz#2028756