diff --git a/SOURCES/ci-Change-netifaces-dependency-to-0.10.4-965.patch b/SOURCES/ci-Change-netifaces-dependency-to-0.10.4-965.patch new file mode 100644 index 0000000..02b5deb --- /dev/null +++ b/SOURCES/ci-Change-netifaces-dependency-to-0.10.4-965.patch @@ -0,0 +1,47 @@ +From 67d62f2c0df1fcb5cd86be73cba6064075aa61e3 Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito <eesposit@redhat.com> +Date: Fri, 14 Jan 2022 16:39:46 +0100 +Subject: [PATCH 3/6] Change netifaces dependency to 0.10.4 (#965) + +RH-Author: Emanuele Giuseppe Esposito <eesposit@redhat.com> +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [3/6] 81f0638e62841bab09b423d9cb5d340026ee87c2 +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy <mmorsy@redhat.com> +RH-Acked-by: Eduardo Otubo <otubo@redhat.com> + +commit b9d308b4d61d22bacc05bcae59819755975631f8 +Author: Andrew Kutz <101085+akutz@users.noreply.github.com> +Date: Tue Aug 10 15:10:44 2021 -0500 + + Change netifaces dependency to 0.10.4 (#965) + + Change netifaces dependency to 0.10.4 + + Currently versions Ubuntu <=20.10 use netifaces 0.10.4 By requiring + netifaces 0.10.9, the VMware datasource omitted itself from cloud-init + on Ubuntu <=20.10. + + This patch changes the netifaces dependency to 0.10.4. While it is true + there are patches to netifaces post 0.10.4 that are desirable, testing + against the most common network configuration was performed to verify + the VMware datasource will still function with netifaces 0.10.4. + +Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> +--- + requirements.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/requirements.txt b/requirements.txt +index 41d01d62..c4adc455 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -40,4 +40,4 @@ jsonschema + # and still participate in instance-data by gathering the network in detail at + # runtime and merge that information into the metadata and repersist that to + # disk. +-netifaces>=0.10.9 ++netifaces>=0.10.4 +-- +2.27.0 + diff --git a/SOURCES/ci-Datasource-for-VMware-953.patch b/SOURCES/ci-Datasource-for-VMware-953.patch new file mode 100644 index 0000000..4e24674 --- /dev/null +++ b/SOURCES/ci-Datasource-for-VMware-953.patch @@ -0,0 +1,2198 @@ +From 697152978b1194aa10ab39597802bb2b4041773c Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito <eesposit@redhat.com> +Date: Fri, 14 Jan 2022 16:37:42 +0100 +Subject: [PATCH 2/6] Datasource for VMware (#953) + +RH-Author: Emanuele Giuseppe Esposito <eesposit@redhat.com> +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [2/6] a0999fa63b8117959839f62bd470f9fe632b31cc +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy <mmorsy@redhat.com> +RH-Acked-by: Eduardo Otubo <otubo@redhat.com> + +commit 8b4a9bc7b81e61943af873bad92e2133f8275b0b +Author: Andrew Kutz <101085+akutz@users.noreply.github.com> +Date: Mon Aug 9 21:24:07 2021 -0500 + + Datasource for VMware (#953) + + This patch finally introduces the Cloud-Init Datasource for VMware + GuestInfo as a part of cloud-init proper. This datasource has existed + since 2018, and rapidly became the de facto datasource for developers + working with Packer, Terraform, for projects like kube-image-builder, + and the de jure datasource for Photon OS. + + The major change to the datasource from its previous incarnation is + the name. Now named DatasourceVMware, this new version of the + datasource will allow multiple transport types in addition to + GuestInfo keys. + + This datasource includes several unique features developed to address + real-world situations: + + * Support for reading any key (metadata, userdata, vendordata) both + from the guestinfo table when running on a VM in vSphere as well as + from an environment variable when running inside of a container, + useful for rapid dev/test. + + * Allows booting with DHCP while still providing full participation + in Cloud-Init instance data and Jinja queries. The netifaces library + provides the ability to inspect the network after it is online, + and the runtime network configuration is then merged into the + existing metadata and persisted to disk. + + * Advertises the local_ipv4 and local_ipv6 addresses via guestinfo + as well. This is useful as Guest Tools is not always able to + identify what would be considered the local address. + + The primary author and current steward of this datasource spoke at + Cloud-Init Con 2020 where there was interest in contributing this datasource + to the Cloud-Init codebase. + + The datasource currently lives in its own GitHub repository at + https://github.com/vmware/cloud-init-vmware-guestinfo. Once the datasource + is merged into Cloud-Init, the old repository will be deprecated. + +Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> +--- + README.md | 2 +- + cloudinit/settings.py | 1 + + cloudinit/sources/DataSourceVMware.py | 871 ++++++++++++++++++ + doc/rtd/topics/availability.rst | 1 + + doc/rtd/topics/datasources.rst | 2 +- + doc/rtd/topics/datasources/vmware.rst | 359 ++++++++ + requirements.txt | 9 + + .../unittests/test_datasource/test_common.py | 3 + + .../unittests/test_datasource/test_vmware.py | 377 ++++++++ + tests/unittests/test_ds_identify.py | 279 +++++- + tools/.github-cla-signers | 1 + + tools/ds-identify | 76 +- + 12 files changed, 1977 insertions(+), 4 deletions(-) + create mode 100644 cloudinit/sources/DataSourceVMware.py + create mode 100644 doc/rtd/topics/datasources/vmware.rst + create mode 100644 tests/unittests/test_datasource/test_vmware.py + +diff --git a/README.md b/README.md +index 435405da..b98f61d3 100644 +--- a/README.md ++++ b/README.md +@@ -39,7 +39,7 @@ get in contact with that distribution and send them our way! + + | Supported OSes | Supported Public Clouds | Supported Private Clouds | + | --- | --- | --- | +-| Alpine Linux<br />ArchLinux<br />Debian<br />Fedora<br />FreeBSD<br />Gentoo Linux<br />NetBSD<br />OpenBSD<br />RHEL/CentOS<br />SLES/openSUSE<br />Ubuntu<br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /> | Amazon Web Services<br />Microsoft Azure<br />Google Cloud Platform<br />Oracle Cloud Infrastructure<br />Softlayer<br />Rackspace Public Cloud<br />IBM Cloud<br />Digital Ocean<br />Bigstep<br />Hetzner<br />Joyent<br />CloudSigma<br />Alibaba Cloud<br />OVH<br />OpenNebula<br />Exoscale<br />Scaleway<br />CloudStack<br />AltCloud<br />SmartOS<br />HyperOne<br />Rootbox<br /> | Bare metal installs<br />OpenStack<br />LXD<br />KVM<br />Metal-as-a-Service (MAAS)<br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />| ++| Alpine Linux<br />ArchLinux<br />Debian<br />Fedora<br />FreeBSD<br />Gentoo Linux<br />NetBSD<br />OpenBSD<br />RHEL/CentOS<br />SLES/openSUSE<br />Ubuntu<br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /> | Amazon Web Services<br />Microsoft Azure<br />Google Cloud Platform<br />Oracle Cloud Infrastructure<br />Softlayer<br />Rackspace Public Cloud<br />IBM Cloud<br />Digital Ocean<br />Bigstep<br />Hetzner<br />Joyent<br />CloudSigma<br />Alibaba Cloud<br />OVH<br />OpenNebula<br />Exoscale<br />Scaleway<br />CloudStack<br />AltCloud<br />SmartOS<br />HyperOne<br />Rootbox<br /> | Bare metal installs<br />OpenStack<br />LXD<br />KVM<br />Metal-as-a-Service (MAAS)<br /><br />VMware<br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />| + + ## To start developing cloud-init + +diff --git a/cloudinit/settings.py b/cloudinit/settings.py +index 2acf2615..d5f32dbb 100644 +--- a/cloudinit/settings.py ++++ b/cloudinit/settings.py +@@ -42,6 +42,7 @@ CFG_BUILTIN = { + 'Exoscale', + 'RbxCloud', + 'UpCloud', ++ 'VMware', + # At the end to act as a 'catch' when none of the above work... + 'None', + ], +diff --git a/cloudinit/sources/DataSourceVMware.py b/cloudinit/sources/DataSourceVMware.py +new file mode 100644 +index 00000000..22ca63de +--- /dev/null ++++ b/cloudinit/sources/DataSourceVMware.py +@@ -0,0 +1,871 @@ ++# Cloud-Init DataSource for VMware ++# ++# Copyright (c) 2018-2021 VMware, Inc. All Rights Reserved. ++# ++# Authors: Anish Swaminathan <anishs@vmware.com> ++# Andrew Kutz <akutz@vmware.com> ++# ++# This file is part of cloud-init. See LICENSE file for license information. ++ ++"""Cloud-Init DataSource for VMware ++ ++This module provides a cloud-init datasource for VMware systems and supports ++multiple transports types, including: ++ ++ * EnvVars ++ * GuestInfo ++ ++Netifaces (https://github.com/al45tair/netifaces) ++ ++ Please note this module relies on the netifaces project to introspect the ++ runtime, network configuration of the host on which this datasource is ++ running. This is in contrast to the rest of cloud-init which uses the ++ cloudinit/netinfo module. ++ ++ The reasons for using netifaces include: ++ ++ * Netifaces is built in C and is more portable across multiple systems ++ and more deterministic than shell exec'ing local network commands and ++ parsing their output. ++ ++ * Netifaces provides a stable way to determine the view of the host's ++ network after DHCP has brought the network online. Unlike most other ++ datasources, this datasource still provides support for JINJA queries ++ based on networking information even when the network is based on a ++ DHCP lease. While this does not tie this datasource directly to ++ netifaces, it does mean the ability to consistently obtain the ++ correct information is paramount. ++ ++ * It is currently possible to execute this datasource on macOS ++ (which many developers use today) to print the output of the ++ get_host_info function. This function calls netifaces to obtain ++ the same runtime network configuration that the datasource would ++ persist to the local system's instance data. ++ ++ However, the netinfo module fails on macOS. The result is either a ++ hung operation that requires a SIGINT to return control to the user, ++ or, if brew is used to install iproute2mac, the ip commands are used ++ but produce output the netinfo module is unable to parse. ++ ++ While macOS is not a target of cloud-init, this feature is quite ++ useful when working on this datasource. ++ ++ For more information about this behavior, please see the following ++ PR comment, https://bit.ly/3fG7OVh. ++ ++ The authors of this datasource are not opposed to moving away from ++ netifaces. The goal may be to eventually do just that. This proviso was ++ added to the top of this module as a way to remind future-us and others ++ why netifaces was used in the first place in order to either smooth the ++ transition away from netifaces or embrace it further up the cloud-init ++ stack. ++""" ++ ++import collections ++import copy ++from distutils.spawn import find_executable ++import ipaddress ++import json ++import os ++import socket ++import time ++ ++from cloudinit import dmi, log as logging ++from cloudinit import sources ++from cloudinit import util ++from cloudinit.subp import subp, ProcessExecutionError ++ ++import netifaces ++ ++ ++PRODUCT_UUID_FILE_PATH = "/sys/class/dmi/id/product_uuid" ++ ++LOG = logging.getLogger(__name__) ++NOVAL = "No value found" ++ ++DATA_ACCESS_METHOD_ENVVAR = "envvar" ++DATA_ACCESS_METHOD_GUESTINFO = "guestinfo" ++ ++VMWARE_RPCTOOL = find_executable("vmware-rpctool") ++REDACT = "redact" ++CLEANUP_GUESTINFO = "cleanup-guestinfo" ++VMX_GUESTINFO = "VMX_GUESTINFO" ++GUESTINFO_EMPTY_YAML_VAL = "---" ++ ++LOCAL_IPV4 = "local-ipv4" ++LOCAL_IPV6 = "local-ipv6" ++WAIT_ON_NETWORK = "wait-on-network" ++WAIT_ON_NETWORK_IPV4 = "ipv4" ++WAIT_ON_NETWORK_IPV6 = "ipv6" ++ ++ ++class DataSourceVMware(sources.DataSource): ++ """ ++ Setting the hostname: ++ The hostname is set by way of the metadata key "local-hostname". ++ ++ Setting the instance ID: ++ The instance ID may be set by way of the metadata key "instance-id". ++ However, if this value is absent then the instance ID is read ++ from the file /sys/class/dmi/id/product_uuid. ++ ++ Configuring the network: ++ The network is configured by setting the metadata key "network" ++ with a value consistent with Network Config Versions 1 or 2, ++ depending on the Linux distro's version of cloud-init: ++ ++ Network Config Version 1 - http://bit.ly/cloudinit-net-conf-v1 ++ Network Config Version 2 - http://bit.ly/cloudinit-net-conf-v2 ++ ++ For example, CentOS 7's official cloud-init package is version ++ 0.7.9 and does not support Network Config Version 2. However, ++ this datasource still supports supplying Network Config Version 2 ++ data as long as the Linux distro's cloud-init package is new ++ enough to parse the data. ++ ++ The metadata key "network.encoding" may be used to indicate the ++ format of the metadata key "network". Valid encodings are base64 ++ and gzip+base64. ++ """ ++ ++ dsname = "VMware" ++ ++ def __init__(self, sys_cfg, distro, paths, ud_proc=None): ++ sources.DataSource.__init__(self, sys_cfg, distro, paths, ud_proc) ++ ++ self.data_access_method = None ++ self.vmware_rpctool = VMWARE_RPCTOOL ++ ++ def _get_data(self): ++ """ ++ _get_data loads the metadata, userdata, and vendordata from one of ++ the following locations in the given order: ++ ++ * envvars ++ * guestinfo ++ ++ Please note when updating this function with support for new data ++ transports, the order should match the order in the dscheck_VMware ++ function from the file ds-identify. ++ """ ++ ++ # Initialize the locally scoped metadata, userdata, and vendordata ++ # variables. They are assigned below depending on the detected data ++ # access method. ++ md, ud, vd = None, None, None ++ ++ # First check to see if there is data via env vars. ++ if os.environ.get(VMX_GUESTINFO, ""): ++ md = guestinfo_envvar("metadata") ++ ud = guestinfo_envvar("userdata") ++ vd = guestinfo_envvar("vendordata") ++ ++ if md or ud or vd: ++ self.data_access_method = DATA_ACCESS_METHOD_ENVVAR ++ ++ # At this point, all additional data transports are valid only on ++ # a VMware platform. ++ if not self.data_access_method: ++ system_type = dmi.read_dmi_data("system-product-name") ++ if system_type is None: ++ LOG.debug("No system-product-name found") ++ return False ++ if "vmware" not in system_type.lower(): ++ LOG.debug("Not a VMware platform") ++ return False ++ ++ # If no data was detected, check the guestinfo transport next. ++ if not self.data_access_method: ++ if self.vmware_rpctool: ++ md = guestinfo("metadata", self.vmware_rpctool) ++ ud = guestinfo("userdata", self.vmware_rpctool) ++ vd = guestinfo("vendordata", self.vmware_rpctool) ++ ++ if md or ud or vd: ++ self.data_access_method = DATA_ACCESS_METHOD_GUESTINFO ++ ++ if not self.data_access_method: ++ LOG.error("failed to find a valid data access method") ++ return False ++ ++ LOG.info("using data access method %s", self._get_subplatform()) ++ ++ # Get the metadata. ++ self.metadata = process_metadata(load_json_or_yaml(md)) ++ ++ # Get the user data. ++ self.userdata_raw = ud ++ ++ # Get the vendor data. ++ self.vendordata_raw = vd ++ ++ # Redact any sensitive information. ++ self.redact_keys() ++ ++ # get_data returns true if there is any available metadata, ++ # userdata, or vendordata. ++ if self.metadata or self.userdata_raw or self.vendordata_raw: ++ return True ++ else: ++ return False ++ ++ def setup(self, is_new_instance): ++ """setup(is_new_instance) ++ ++ This is called before user-data and vendor-data have been processed. ++ ++ Unless the datasource has set mode to 'local', then networking ++ per 'fallback' or per 'network_config' will have been written and ++ brought up the OS at this point. ++ """ ++ ++ host_info = wait_on_network(self.metadata) ++ LOG.info("got host-info: %s", host_info) ++ ++ # Reflect any possible local IPv4 or IPv6 addresses in the guest ++ # info. ++ advertise_local_ip_addrs(host_info) ++ ++ # Ensure the metadata gets updated with information about the ++ # host, including the network interfaces, default IP addresses, ++ # etc. ++ self.metadata = util.mergemanydict([self.metadata, host_info]) ++ ++ # Persist the instance data for versions of cloud-init that support ++ # doing so. This occurs here rather than in the get_data call in ++ # order to ensure that the network interfaces are up and can be ++ # persisted with the metadata. ++ self.persist_instance_data() ++ ++ def _get_subplatform(self): ++ get_key_name_fn = None ++ if self.data_access_method == DATA_ACCESS_METHOD_ENVVAR: ++ get_key_name_fn = get_guestinfo_envvar_key_name ++ elif self.data_access_method == DATA_ACCESS_METHOD_GUESTINFO: ++ get_key_name_fn = get_guestinfo_key_name ++ else: ++ return sources.METADATA_UNKNOWN ++ ++ return "%s (%s)" % ( ++ self.data_access_method, ++ get_key_name_fn("metadata"), ++ ) ++ ++ @property ++ def network_config(self): ++ if "network" in self.metadata: ++ LOG.debug("using metadata network config") ++ else: ++ LOG.debug("using fallback network config") ++ self.metadata["network"] = { ++ "config": self.distro.generate_fallback_config(), ++ } ++ return self.metadata["network"]["config"] ++ ++ def get_instance_id(self): ++ # Pull the instance ID out of the metadata if present. Otherwise ++ # read the file /sys/class/dmi/id/product_uuid for the instance ID. ++ if self.metadata and "instance-id" in self.metadata: ++ return self.metadata["instance-id"] ++ with open(PRODUCT_UUID_FILE_PATH, "r") as id_file: ++ self.metadata["instance-id"] = str(id_file.read()).rstrip().lower() ++ return self.metadata["instance-id"] ++ ++ def get_public_ssh_keys(self): ++ for key_name in ( ++ "public-keys-data", ++ "public_keys_data", ++ "public-keys", ++ "public_keys", ++ ): ++ if key_name in self.metadata: ++ return sources.normalize_pubkey_data(self.metadata[key_name]) ++ return [] ++ ++ def redact_keys(self): ++ # Determine if there are any keys to redact. ++ keys_to_redact = None ++ if REDACT in self.metadata: ++ keys_to_redact = self.metadata[REDACT] ++ elif CLEANUP_GUESTINFO in self.metadata: ++ # This is for backwards compatibility. ++ keys_to_redact = self.metadata[CLEANUP_GUESTINFO] ++ ++ if self.data_access_method == DATA_ACCESS_METHOD_GUESTINFO: ++ guestinfo_redact_keys(keys_to_redact, self.vmware_rpctool) ++ ++ ++def decode(key, enc_type, data): ++ """ ++ decode returns the decoded string value of data ++ key is a string used to identify the data being decoded in log messages ++ """ ++ LOG.debug("Getting encoded data for key=%s, enc=%s", key, enc_type) ++ ++ raw_data = None ++ if enc_type in ["gzip+base64", "gz+b64"]: ++ LOG.debug("Decoding %s format %s", enc_type, key) ++ raw_data = util.decomp_gzip(util.b64d(data)) ++ elif enc_type in ["base64", "b64"]: ++ LOG.debug("Decoding %s format %s", enc_type, key) ++ raw_data = util.b64d(data) ++ else: ++ LOG.debug("Plain-text data %s", key) ++ raw_data = data ++ ++ return util.decode_binary(raw_data) ++ ++ ++def get_none_if_empty_val(val): ++ """ ++ get_none_if_empty_val returns None if the provided value, once stripped ++ of its trailing whitespace, is empty or equal to GUESTINFO_EMPTY_YAML_VAL. ++ ++ The return value is always a string, regardless of whether the input is ++ a bytes class or a string. ++ """ ++ ++ # If the provided value is a bytes class, convert it to a string to ++ # simplify the rest of this function's logic. ++ val = util.decode_binary(val) ++ val = val.rstrip() ++ if len(val) == 0 or val == GUESTINFO_EMPTY_YAML_VAL: ++ return None ++ return val ++ ++ ++def advertise_local_ip_addrs(host_info): ++ """ ++ advertise_local_ip_addrs gets the local IP address information from ++ the provided host_info map and sets the addresses in the guestinfo ++ namespace ++ """ ++ if not host_info: ++ return ++ ++ # Reflect any possible local IPv4 or IPv6 addresses in the guest ++ # info. ++ local_ipv4 = host_info.get(LOCAL_IPV4) ++ if local_ipv4: ++ guestinfo_set_value(LOCAL_IPV4, local_ipv4) ++ LOG.info("advertised local ipv4 address %s in guestinfo", local_ipv4) ++ ++ local_ipv6 = host_info.get(LOCAL_IPV6) ++ if local_ipv6: ++ guestinfo_set_value(LOCAL_IPV6, local_ipv6) ++ LOG.info("advertised local ipv6 address %s in guestinfo", local_ipv6) ++ ++ ++def handle_returned_guestinfo_val(key, val): ++ """ ++ handle_returned_guestinfo_val returns the provided value if it is ++ not empty or set to GUESTINFO_EMPTY_YAML_VAL, otherwise None is ++ returned ++ """ ++ val = get_none_if_empty_val(val) ++ if val: ++ return val ++ LOG.debug("No value found for key %s", key) ++ return None ++ ++ ++def get_guestinfo_key_name(key): ++ return "guestinfo." + key ++ ++ ++def get_guestinfo_envvar_key_name(key): ++ return ("vmx." + get_guestinfo_key_name(key)).upper().replace(".", "_", -1) ++ ++ ++def guestinfo_envvar(key): ++ val = guestinfo_envvar_get_value(key) ++ if not val: ++ return None ++ enc_type = guestinfo_envvar_get_value(key + ".encoding") ++ return decode(get_guestinfo_envvar_key_name(key), enc_type, val) ++ ++ ++def guestinfo_envvar_get_value(key): ++ env_key = get_guestinfo_envvar_key_name(key) ++ return handle_returned_guestinfo_val(key, os.environ.get(env_key, "")) ++ ++ ++def guestinfo(key, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ guestinfo returns the guestinfo value for the provided key, decoding ++ the value when required ++ """ ++ val = guestinfo_get_value(key, vmware_rpctool) ++ if not val: ++ return None ++ enc_type = guestinfo_get_value(key + ".encoding", vmware_rpctool) ++ return decode(get_guestinfo_key_name(key), enc_type, val) ++ ++ ++def guestinfo_get_value(key, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ Returns a guestinfo value for the specified key. ++ """ ++ LOG.debug("Getting guestinfo value for key %s", key) ++ ++ try: ++ (stdout, stderr) = subp( ++ [ ++ vmware_rpctool, ++ "info-get " + get_guestinfo_key_name(key), ++ ] ++ ) ++ if stderr == NOVAL: ++ LOG.debug("No value found for key %s", key) ++ elif not stdout: ++ LOG.error("Failed to get guestinfo value for key %s", key) ++ return handle_returned_guestinfo_val(key, stdout) ++ except ProcessExecutionError as error: ++ if error.stderr == NOVAL: ++ LOG.debug("No value found for key %s", key) ++ else: ++ util.logexc( ++ LOG, ++ "Failed to get guestinfo value for key %s: %s", ++ key, ++ error, ++ ) ++ except Exception: ++ util.logexc( ++ LOG, ++ "Unexpected error while trying to get " ++ + "guestinfo value for key %s", ++ key, ++ ) ++ ++ return None ++ ++ ++def guestinfo_set_value(key, value, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ Sets a guestinfo value for the specified key. Set value to an empty string ++ to clear an existing guestinfo key. ++ """ ++ ++ # If value is an empty string then set it to a single space as it is not ++ # possible to set a guestinfo key to an empty string. Setting a guestinfo ++ # key to a single space is as close as it gets to clearing an existing ++ # guestinfo key. ++ if value == "": ++ value = " " ++ ++ LOG.debug("Setting guestinfo key=%s to value=%s", key, value) ++ ++ try: ++ subp( ++ [ ++ vmware_rpctool, ++ ("info-set %s %s" % (get_guestinfo_key_name(key), value)), ++ ] ++ ) ++ return True ++ except ProcessExecutionError as error: ++ util.logexc( ++ LOG, ++ "Failed to set guestinfo key=%s to value=%s: %s", ++ key, ++ value, ++ error, ++ ) ++ except Exception: ++ util.logexc( ++ LOG, ++ "Unexpected error while trying to set " ++ + "guestinfo key=%s to value=%s", ++ key, ++ value, ++ ) ++ ++ return None ++ ++ ++def guestinfo_redact_keys(keys, vmware_rpctool=VMWARE_RPCTOOL): ++ """ ++ guestinfo_redact_keys redacts guestinfo of all of the keys in the given ++ list. each key will have its value set to "---". Since the value is valid ++ YAML, cloud-init can still read it if it tries. ++ """ ++ if not keys: ++ return ++ if not type(keys) in (list, tuple): ++ keys = [keys] ++ for key in keys: ++ key_name = get_guestinfo_key_name(key) ++ LOG.info("clearing %s", key_name) ++ if not guestinfo_set_value( ++ key, GUESTINFO_EMPTY_YAML_VAL, vmware_rpctool ++ ): ++ LOG.error("failed to clear %s", key_name) ++ LOG.info("clearing %s.encoding", key_name) ++ if not guestinfo_set_value(key + ".encoding", "", vmware_rpctool): ++ LOG.error("failed to clear %s.encoding", key_name) ++ ++ ++def load_json_or_yaml(data): ++ """ ++ load first attempts to unmarshal the provided data as JSON, and if ++ that fails then attempts to unmarshal the data as YAML. If data is ++ None then a new dictionary is returned. ++ """ ++ if not data: ++ return {} ++ try: ++ return util.load_json(data) ++ except (json.JSONDecodeError, TypeError): ++ return util.load_yaml(data) ++ ++ ++def process_metadata(data): ++ """ ++ process_metadata processes metadata and loads the optional network ++ configuration. ++ """ ++ network = None ++ if "network" in data: ++ network = data["network"] ++ del data["network"] ++ ++ network_enc = None ++ if "network.encoding" in data: ++ network_enc = data["network.encoding"] ++ del data["network.encoding"] ++ ++ if network: ++ if isinstance(network, collections.abc.Mapping): ++ LOG.debug("network data copied to 'config' key") ++ network = {"config": copy.deepcopy(network)} ++ else: ++ LOG.debug("network data to be decoded %s", network) ++ dec_net = decode("metadata.network", network_enc, network) ++ network = { ++ "config": load_json_or_yaml(dec_net), ++ } ++ ++ LOG.debug("network data %s", network) ++ data["network"] = network ++ ++ return data ++ ++ ++# Used to match classes to dependencies ++datasources = [ ++ (DataSourceVMware, (sources.DEP_FILESYSTEM,)), # Run at init-local ++ (DataSourceVMware, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)), ++] ++ ++ ++def get_datasource_list(depends): ++ """ ++ Return a list of data sources that match this set of dependencies ++ """ ++ return sources.list_from_depends(depends, datasources) ++ ++ ++def get_default_ip_addrs(): ++ """ ++ Returns the default IPv4 and IPv6 addresses based on the device(s) used for ++ the default route. Please note that None may be returned for either address ++ family if that family has no default route or if there are multiple ++ addresses associated with the device used by the default route for a given ++ address. ++ """ ++ # TODO(promote and use netifaces in cloudinit.net* modules) ++ gateways = netifaces.gateways() ++ if "default" not in gateways: ++ return None, None ++ ++ default_gw = gateways["default"] ++ if ( ++ netifaces.AF_INET not in default_gw ++ and netifaces.AF_INET6 not in default_gw ++ ): ++ return None, None ++ ++ ipv4 = None ++ ipv6 = None ++ ++ gw4 = default_gw.get(netifaces.AF_INET) ++ if gw4: ++ _, dev4 = gw4 ++ addr4_fams = netifaces.ifaddresses(dev4) ++ if addr4_fams: ++ af_inet4 = addr4_fams.get(netifaces.AF_INET) ++ if af_inet4: ++ if len(af_inet4) > 1: ++ LOG.warning( ++ "device %s has more than one ipv4 address: %s", ++ dev4, ++ af_inet4, ++ ) ++ elif "addr" in af_inet4[0]: ++ ipv4 = af_inet4[0]["addr"] ++ ++ # Try to get the default IPv6 address by first seeing if there is a default ++ # IPv6 route. ++ gw6 = default_gw.get(netifaces.AF_INET6) ++ if gw6: ++ _, dev6 = gw6 ++ addr6_fams = netifaces.ifaddresses(dev6) ++ if addr6_fams: ++ af_inet6 = addr6_fams.get(netifaces.AF_INET6) ++ if af_inet6: ++ if len(af_inet6) > 1: ++ LOG.warning( ++ "device %s has more than one ipv6 address: %s", ++ dev6, ++ af_inet6, ++ ) ++ elif "addr" in af_inet6[0]: ++ ipv6 = af_inet6[0]["addr"] ++ ++ # If there is a default IPv4 address but not IPv6, then see if there is a ++ # single IPv6 address associated with the same device associated with the ++ # default IPv4 address. ++ if ipv4 and not ipv6: ++ af_inet6 = addr4_fams.get(netifaces.AF_INET6) ++ if af_inet6: ++ if len(af_inet6) > 1: ++ LOG.warning( ++ "device %s has more than one ipv6 address: %s", ++ dev4, ++ af_inet6, ++ ) ++ elif "addr" in af_inet6[0]: ++ ipv6 = af_inet6[0]["addr"] ++ ++ # If there is a default IPv6 address but not IPv4, then see if there is a ++ # single IPv4 address associated with the same device associated with the ++ # default IPv6 address. ++ if not ipv4 and ipv6: ++ af_inet4 = addr6_fams.get(netifaces.AF_INET) ++ if af_inet4: ++ if len(af_inet4) > 1: ++ LOG.warning( ++ "device %s has more than one ipv4 address: %s", ++ dev6, ++ af_inet4, ++ ) ++ elif "addr" in af_inet4[0]: ++ ipv4 = af_inet4[0]["addr"] ++ ++ return ipv4, ipv6 ++ ++ ++# patched socket.getfqdn() - see https://bugs.python.org/issue5004 ++ ++ ++def getfqdn(name=""): ++ """Get fully qualified domain name from name. ++ An empty argument is interpreted as meaning the local host. ++ """ ++ # TODO(may want to promote this function to util.getfqdn) ++ # TODO(may want to extend util.get_hostname to accept fqdn=True param) ++ name = name.strip() ++ if not name or name == "0.0.0.0": ++ name = util.get_hostname() ++ try: ++ addrs = socket.getaddrinfo( ++ name, None, 0, socket.SOCK_DGRAM, 0, socket.AI_CANONNAME ++ ) ++ except socket.error: ++ pass ++ else: ++ for addr in addrs: ++ if addr[3]: ++ name = addr[3] ++ break ++ return name ++ ++ ++def is_valid_ip_addr(val): ++ """ ++ Returns false if the address is loopback, link local or unspecified; ++ otherwise true is returned. ++ """ ++ # TODO(extend cloudinit.net.is_ip_addr exclude link_local/loopback etc) ++ # TODO(migrate to use cloudinit.net.is_ip_addr)# ++ ++ addr = None ++ try: ++ addr = ipaddress.ip_address(val) ++ except ipaddress.AddressValueError: ++ addr = ipaddress.ip_address(str(val)) ++ except Exception: ++ return None ++ ++ if addr.is_link_local or addr.is_loopback or addr.is_unspecified: ++ return False ++ return True ++ ++ ++def get_host_info(): ++ """ ++ Returns host information such as the host name and network interfaces. ++ """ ++ # TODO(look to promote netifices use up in cloud-init netinfo funcs) ++ host_info = { ++ "network": { ++ "interfaces": { ++ "by-mac": collections.OrderedDict(), ++ "by-ipv4": collections.OrderedDict(), ++ "by-ipv6": collections.OrderedDict(), ++ }, ++ }, ++ } ++ hostname = getfqdn(util.get_hostname()) ++ if hostname: ++ host_info["hostname"] = hostname ++ host_info["local-hostname"] = hostname ++ host_info["local_hostname"] = hostname ++ ++ default_ipv4, default_ipv6 = get_default_ip_addrs() ++ if default_ipv4: ++ host_info[LOCAL_IPV4] = default_ipv4 ++ if default_ipv6: ++ host_info[LOCAL_IPV6] = default_ipv6 ++ ++ by_mac = host_info["network"]["interfaces"]["by-mac"] ++ by_ipv4 = host_info["network"]["interfaces"]["by-ipv4"] ++ by_ipv6 = host_info["network"]["interfaces"]["by-ipv6"] ++ ++ ifaces = netifaces.interfaces() ++ for dev_name in ifaces: ++ addr_fams = netifaces.ifaddresses(dev_name) ++ af_link = addr_fams.get(netifaces.AF_LINK) ++ af_inet4 = addr_fams.get(netifaces.AF_INET) ++ af_inet6 = addr_fams.get(netifaces.AF_INET6) ++ ++ mac = None ++ if af_link and "addr" in af_link[0]: ++ mac = af_link[0]["addr"] ++ ++ # Do not bother recording localhost ++ if mac == "00:00:00:00:00:00": ++ continue ++ ++ if mac and (af_inet4 or af_inet6): ++ key = mac ++ val = {} ++ if af_inet4: ++ af_inet4_vals = [] ++ for ip_info in af_inet4: ++ if not is_valid_ip_addr(ip_info["addr"]): ++ continue ++ af_inet4_vals.append(ip_info) ++ val["ipv4"] = af_inet4_vals ++ if af_inet6: ++ af_inet6_vals = [] ++ for ip_info in af_inet6: ++ if not is_valid_ip_addr(ip_info["addr"]): ++ continue ++ af_inet6_vals.append(ip_info) ++ val["ipv6"] = af_inet6_vals ++ by_mac[key] = val ++ ++ if af_inet4: ++ for ip_info in af_inet4: ++ key = ip_info["addr"] ++ if not is_valid_ip_addr(key): ++ continue ++ val = copy.deepcopy(ip_info) ++ del val["addr"] ++ if mac: ++ val["mac"] = mac ++ by_ipv4[key] = val ++ ++ if af_inet6: ++ for ip_info in af_inet6: ++ key = ip_info["addr"] ++ if not is_valid_ip_addr(key): ++ continue ++ val = copy.deepcopy(ip_info) ++ del val["addr"] ++ if mac: ++ val["mac"] = mac ++ by_ipv6[key] = val ++ ++ return host_info ++ ++ ++def wait_on_network(metadata): ++ # Determine whether we need to wait on the network coming online. ++ wait_on_ipv4 = False ++ wait_on_ipv6 = False ++ if WAIT_ON_NETWORK in metadata: ++ wait_on_network = metadata[WAIT_ON_NETWORK] ++ if WAIT_ON_NETWORK_IPV4 in wait_on_network: ++ wait_on_ipv4_val = wait_on_network[WAIT_ON_NETWORK_IPV4] ++ if isinstance(wait_on_ipv4_val, bool): ++ wait_on_ipv4 = wait_on_ipv4_val ++ else: ++ wait_on_ipv4 = util.translate_bool(wait_on_ipv4_val) ++ if WAIT_ON_NETWORK_IPV6 in wait_on_network: ++ wait_on_ipv6_val = wait_on_network[WAIT_ON_NETWORK_IPV6] ++ if isinstance(wait_on_ipv6_val, bool): ++ wait_on_ipv6 = wait_on_ipv6_val ++ else: ++ wait_on_ipv6 = util.translate_bool(wait_on_ipv6_val) ++ ++ # Get information about the host. ++ host_info = None ++ while host_info is None: ++ # This loop + sleep results in two logs every second while waiting ++ # for either ipv4 or ipv6 up. Do we really need to log each iteration ++ # or can we log once and log on successful exit? ++ host_info = get_host_info() ++ ++ network = host_info.get("network") or {} ++ interfaces = network.get("interfaces") or {} ++ by_ipv4 = interfaces.get("by-ipv4") or {} ++ by_ipv6 = interfaces.get("by-ipv6") or {} ++ ++ if wait_on_ipv4: ++ ipv4_ready = len(by_ipv4) > 0 if by_ipv4 else False ++ if not ipv4_ready: ++ host_info = None ++ ++ if wait_on_ipv6: ++ ipv6_ready = len(by_ipv6) > 0 if by_ipv6 else False ++ if not ipv6_ready: ++ host_info = None ++ ++ if host_info is None: ++ LOG.debug( ++ "waiting on network: wait4=%s, ready4=%s, wait6=%s, ready6=%s", ++ wait_on_ipv4, ++ ipv4_ready, ++ wait_on_ipv6, ++ ipv6_ready, ++ ) ++ time.sleep(1) ++ ++ LOG.debug("waiting on network complete") ++ return host_info ++ ++ ++def main(): ++ """ ++ Executed when this file is used as a program. ++ """ ++ try: ++ logging.setupBasicLogging() ++ except Exception: ++ pass ++ metadata = { ++ "wait-on-network": {"ipv4": True, "ipv6": "false"}, ++ "network": {"config": {"dhcp": True}}, ++ } ++ host_info = wait_on_network(metadata) ++ metadata = util.mergemanydict([metadata, host_info]) ++ print(util.json_dumps(metadata)) ++ ++ ++if __name__ == "__main__": ++ main() ++ ++# vi: ts=4 expandtab +diff --git a/doc/rtd/topics/availability.rst b/doc/rtd/topics/availability.rst +index f58b2b38..6606367c 100644 +--- a/doc/rtd/topics/availability.rst ++++ b/doc/rtd/topics/availability.rst +@@ -64,5 +64,6 @@ Additionally, cloud-init is supported on these private clouds: + - LXD + - KVM + - Metal-as-a-Service (MAAS) ++- VMware + + .. vi: textwidth=79 +diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst +index 228173d2..8afed470 100644 +--- a/doc/rtd/topics/datasources.rst ++++ b/doc/rtd/topics/datasources.rst +@@ -49,7 +49,7 @@ The following is a list of documents for each supported datasource: + datasources/smartos.rst + datasources/upcloud.rst + datasources/zstack.rst +- ++ datasources/vmware.rst + + Creation + ======== +diff --git a/doc/rtd/topics/datasources/vmware.rst b/doc/rtd/topics/datasources/vmware.rst +new file mode 100644 +index 00000000..996eb61f +--- /dev/null ++++ b/doc/rtd/topics/datasources/vmware.rst +@@ -0,0 +1,359 @@ ++.. _datasource_vmware: ++ ++VMware ++====== ++ ++This datasource is for use with systems running on a VMware platform such as ++vSphere and currently supports the following data transports: ++ ++ ++* `GuestInfo <https://github.com/vmware/govmomi/blob/master/govc/USAGE.md#vmchange>`_ keys ++ ++Configuration ++------------- ++ ++The configuration method is dependent upon the transport: ++ ++GuestInfo Keys ++^^^^^^^^^^^^^^ ++ ++One method of providing meta, user, and vendor data is by setting the following ++key/value pairs on a VM's ``extraConfig`` `property <https://vdc-repo.vmware.com/vmwb-repository/dcr-public/723e7f8b-4f21-448b-a830-5f22fd931b01/5a8257bd-7f41-4423-9a73-03307535bd42/doc/vim.vm.ConfigInfo.html>`_ : ++ ++.. list-table:: ++ :header-rows: 1 ++ ++ * - Property ++ - Description ++ * - ``guestinfo.metadata`` ++ - A YAML or JSON document containing the cloud-init metadata. ++ * - ``guestinfo.metadata.encoding`` ++ - The encoding type for ``guestinfo.metadata``. ++ * - ``guestinfo.userdata`` ++ - A YAML document containing the cloud-init user data. ++ * - ``guestinfo.userdata.encoding`` ++ - The encoding type for ``guestinfo.userdata``. ++ * - ``guestinfo.vendordata`` ++ - A YAML document containing the cloud-init vendor data. ++ * - ``guestinfo.vendordata.encoding`` ++ - The encoding type for ``guestinfo.vendordata``. ++ ++ ++All ``guestinfo.*.encoding`` values may be set to ``base64`` or ++``gzip+base64``. ++ ++Features ++-------- ++ ++This section reviews several features available in this datasource, regardless ++of how the meta, user, and vendor data was discovered. ++ ++Instance data and lazy networks ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++One of the hallmarks of cloud-init is `its use of instance-data and JINJA ++queries <../instancedata.html#using-instance-data>`_ ++-- the ability to write queries in user and vendor data that reference runtime ++information present in ``/run/cloud-init/instance-data.json``. This works well ++when the metadata provides all of the information up front, such as the network ++configuration. For systems that rely on DHCP, however, this information may not ++be available when the metadata is persisted to disk. ++ ++This datasource ensures that even if the instance is using DHCP to configure ++networking, the same details about the configured network are available in ++``/run/cloud-init/instance-data.json`` as if static networking was used. This ++information collected at runtime is easy to demonstrate by executing the ++datasource on the command line. From the root of this repository, run the ++following command: ++ ++.. code-block:: bash ++ ++ PYTHONPATH="$(pwd)" python3 cloudinit/sources/DataSourceVMware.py ++ ++The above command will result in output similar to the below JSON: ++ ++.. code-block:: json ++ ++ { ++ "hostname": "akutz.localhost", ++ "local-hostname": "akutz.localhost", ++ "local-ipv4": "192.168.0.188", ++ "local_hostname": "akutz.localhost", ++ "network": { ++ "config": { ++ "dhcp": true ++ }, ++ "interfaces": { ++ "by-ipv4": { ++ "172.0.0.2": { ++ "netmask": "255.255.255.255", ++ "peer": "172.0.0.2" ++ }, ++ "192.168.0.188": { ++ "broadcast": "192.168.0.255", ++ "mac": "64:4b:f0:18:9a:21", ++ "netmask": "255.255.255.0" ++ } ++ }, ++ "by-ipv6": { ++ "fd8e:d25e:c5b6:1:1f5:b2fd:8973:22f2": { ++ "flags": 208, ++ "mac": "64:4b:f0:18:9a:21", ++ "netmask": "ffff:ffff:ffff:ffff::/64" ++ } ++ }, ++ "by-mac": { ++ "64:4b:f0:18:9a:21": { ++ "ipv4": [ ++ { ++ "addr": "192.168.0.188", ++ "broadcast": "192.168.0.255", ++ "netmask": "255.255.255.0" ++ } ++ ], ++ "ipv6": [ ++ { ++ "addr": "fd8e:d25e:c5b6:1:1f5:b2fd:8973:22f2", ++ "flags": 208, ++ "netmask": "ffff:ffff:ffff:ffff::/64" ++ } ++ ] ++ }, ++ "ac:de:48:00:11:22": { ++ "ipv6": [] ++ } ++ } ++ } ++ }, ++ "wait-on-network": { ++ "ipv4": true, ++ "ipv6": "false" ++ } ++ } ++ ++ ++Redacting sensitive information ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++Sometimes the cloud-init userdata might contain sensitive information, and it ++may be desirable to have the ``guestinfo.userdata`` key (or other guestinfo ++keys) redacted as soon as its data is read by the datasource. This is possible ++by adding the following to the metadata: ++ ++.. code-block:: yaml ++ ++ redact: # formerly named cleanup-guestinfo, which will also work ++ - userdata ++ - vendordata ++ ++When the above snippet is added to the metadata, the datasource will iterate ++over the elements in the ``redact`` array and clear each of the keys. For ++example, when the guestinfo transport is used, the above snippet will cause ++the following commands to be executed: ++ ++.. code-block:: shell ++ ++ vmware-rpctool "info-set guestinfo.userdata ---" ++ vmware-rpctool "info-set guestinfo.userdata.encoding " ++ vmware-rpctool "info-set guestinfo.vendordata ---" ++ vmware-rpctool "info-set guestinfo.vendordata.encoding " ++ ++Please note that keys are set to the valid YAML string ``---`` as it is not ++possible remove an existing key from the guestinfo key-space. A key's analogous ++encoding property will be set to a single white-space character, causing the ++datasource to treat the actual key value as plain-text, thereby loading it as ++an empty YAML doc (hence the aforementioned ``---``\ ). ++ ++Reading the local IP addresses ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++This datasource automatically discovers the local IPv4 and IPv6 addresses for ++a guest operating system based on the default routes. However, when inspecting ++a VM externally, it's not possible to know what the *default* IP address is for ++the guest OS. That's why this datasource sets the discovered, local IPv4 and ++IPv6 addresses back in the guestinfo namespace as the following keys: ++ ++ ++* ``guestinfo.local-ipv4`` ++* ``guestinfo.local-ipv6`` ++ ++It is possible that a host may not have any default, local IP addresses. It's ++also possible the reported, local addresses are link-local addresses. But these ++two keys may be used to discover what this datasource determined were the local ++IPv4 and IPv6 addresses for a host. ++ ++Waiting on the network ++^^^^^^^^^^^^^^^^^^^^^^ ++ ++Sometimes cloud-init may bring up the network, but it will not finish coming ++online before the datasource's ``setup`` function is called, resulting in an ++``/var/run/cloud-init/instance-data.json`` file that does not have the correct ++network information. It is possible to instruct the datasource to wait until an ++IPv4 or IPv6 address is available before writing the instance data with the ++following metadata properties: ++ ++.. code-block:: yaml ++ ++ wait-on-network: ++ ipv4: true ++ ipv6: true ++ ++If either of the above values are true, then the datasource will sleep for a ++second, check the network status, and repeat until one or both addresses from ++the specified families are available. ++ ++Walkthrough ++----------- ++ ++The following series of steps is a demonstration on how to configure a VM with ++this datasource: ++ ++ ++#. Create the metadata file for the VM. Save the following YAML to a file named ++ ``metadata.yaml``\ : ++ ++ .. code-block:: yaml ++ ++ instance-id: cloud-vm ++ local-hostname: cloud-vm ++ network: ++ version: 2 ++ ethernets: ++ nics: ++ match: ++ name: ens* ++ dhcp4: yes ++ ++#. Create the userdata file ``userdata.yaml``\ : ++ ++ .. code-block:: yaml ++ ++ #cloud-config ++ ++ users: ++ - default ++ - name: akutz ++ primary_group: akutz ++ sudo: ALL=(ALL) NOPASSWD:ALL ++ groups: sudo, wheel ++ ssh_import_id: None ++ lock_passwd: true ++ ssh_authorized_keys: ++ - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDE0c5FczvcGSh/tG4iw+Fhfi/O5/EvUM/96js65tly4++YTXK1d9jcznPS5ruDlbIZ30oveCBd3kT8LLVFwzh6hepYTf0YmCTpF4eDunyqmpCXDvVscQYRXyasEm5olGmVe05RrCJSeSShAeptv4ueIn40kZKOghinGWLDSZG4+FFfgrmcMCpx5YSCtX2gvnEYZJr0czt4rxOZuuP7PkJKgC/mt2PcPjooeX00vAj81jjU2f3XKrjjz2u2+KIt9eba+vOQ6HiC8c2IzRkUAJ5i1atLy8RIbejo23+0P4N2jjk17QySFOVHwPBDTYb0/0M/4ideeU74EN/CgVsvO6JrLsPBR4dojkV5qNbMNxIVv5cUwIy2ThlLgqpNCeFIDLCWNZEFKlEuNeSQ2mPtIO7ETxEL2Cz5y/7AIuildzYMc6wi2bofRC8HmQ7rMXRWdwLKWsR0L7SKjHblIwarxOGqLnUI+k2E71YoP7SZSlxaKi17pqkr0OMCF+kKqvcvHAQuwGqyumTEWOlH6TCx1dSPrW+pVCZSHSJtSTfDW2uzL6y8k10MT06+pVunSrWo5LHAXcS91htHV1M1UrH/tZKSpjYtjMb5+RonfhaFRNzvj7cCE1f3Kp8UVqAdcGBTtReoE8eRUT63qIxjw03a7VwAyB2w+9cu1R9/vAo8SBeRqw== sakutz@gmail.com ++ ++#. Please note this step requires that the VM be powered off. All of the ++ commands below use the VMware CLI tool, `govc <https://github.com/vmware/govmomi/blob/master/govc>`_. ++ ++ Go ahead and assign the path to the VM to the environment variable ``VM``\ : ++ ++ .. code-block:: shell ++ ++ export VM="/inventory/path/to/the/vm" ++ ++#. Power off the VM: ++ ++ .. raw:: html ++ ++ <hr /> ++ ++ ⚠️ <strong>First Boot Mode</strong> ++ ++ To ensure the next power-on operation results in a first-boot scenario for ++ cloud-init, it may be necessary to run the following command just before ++ powering off the VM: ++ ++ .. code-block:: bash ++ ++ cloud-init clean ++ ++ Otherwise cloud-init may not run in first-boot mode. For more information ++ on how the boot mode is determined, please see the ++ `First Boot Documentation <../boot.html#first-boot-determination>`_. ++ ++ .. raw:: html ++ ++ <hr /> ++ ++ .. code-block:: shell ++ ++ govc vm.power -off "${VM}" ++ ++#. ++ Export the environment variables that contain the cloud-init metadata and ++ userdata: ++ ++ .. code-block:: shell ++ ++ export METADATA=$(gzip -c9 <metadata.yaml | { base64 -w0 2>/dev/null || base64; }) \ ++ USERDATA=$(gzip -c9 <userdata.yaml | { base64 -w0 2>/dev/null || base64; }) ++ ++#. ++ Assign the metadata and userdata to the VM: ++ ++ .. code-block:: shell ++ ++ govc vm.change -vm "${VM}" \ ++ -e guestinfo.metadata="${METADATA}" \ ++ -e guestinfo.metadata.encoding="gzip+base64" \ ++ -e guestinfo.userdata="${USERDATA}" \ ++ -e guestinfo.userdata.encoding="gzip+base64" ++ ++ Please note the above commands include specifying the encoding for the ++ properties. This is important as it informs the datasource how to decode ++ the data for cloud-init. Valid values for ``metadata.encoding`` and ++ ``userdata.encoding`` include: ++ ++ ++ * ``base64`` ++ * ``gzip+base64`` ++ ++#. ++ Power on the VM: ++ ++ .. code-block:: shell ++ ++ govc vm.power -vm "${VM}" -on ++ ++If all went according to plan, the CentOS box is: ++ ++* Locked down, allowing SSH access only for the user in the userdata ++* Configured for a dynamic IP address via DHCP ++* Has a hostname of ``cloud-vm`` ++ ++Examples ++-------- ++ ++This section reviews common configurations: ++ ++Setting the hostname ++^^^^^^^^^^^^^^^^^^^^ ++ ++The hostname is set by way of the metadata key ``local-hostname``. ++ ++Setting the instance ID ++^^^^^^^^^^^^^^^^^^^^^^^ ++ ++The instance ID may be set by way of the metadata key ``instance-id``. However, ++if this value is absent then then the instance ID is read from the file ++``/sys/class/dmi/id/product_uuid``. ++ ++Providing public SSH keys ++^^^^^^^^^^^^^^^^^^^^^^^^^ ++ ++The public SSH keys may be set by way of the metadata key ``public-keys-data``. ++Each newline-terminated string will be interpreted as a separate SSH public ++key, which will be placed in distro's default user's ++``~/.ssh/authorized_keys``. If the value is empty or absent, then nothing will ++be written to ``~/.ssh/authorized_keys``. ++ ++Configuring the network ++^^^^^^^^^^^^^^^^^^^^^^^ ++ ++The network is configured by setting the metadata key ``network`` with a value ++consistent with Network Config Versions ++`1 <../network-config-format-v1.html>`_ or ++`2 <../network-config-format-v2.html>`_\ , depending on the Linux ++distro's version of cloud-init. ++ ++The metadata key ``network.encoding`` may be used to indicate the format of ++the metadata key "network". Valid encodings are ``base64`` and ``gzip+base64``. +diff --git a/requirements.txt b/requirements.txt +index 5817da3b..41d01d62 100644 +--- a/requirements.txt ++++ b/requirements.txt +@@ -32,3 +32,12 @@ jsonpatch + + # For validating cloud-config sections per schema definitions + jsonschema ++ ++# Used by DataSourceVMware to inspect the host's network configuration during ++# the "setup()" function. ++# ++# This allows a host that uses DHCP to bring up the network during BootLocal ++# and still participate in instance-data by gathering the network in detail at ++# runtime and merge that information into the metadata and repersist that to ++# disk. ++netifaces>=0.10.9 +diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py +index 5912f7ee..475a2cf8 100644 +--- a/tests/unittests/test_datasource/test_common.py ++++ b/tests/unittests/test_datasource/test_common.py +@@ -28,6 +28,7 @@ from cloudinit.sources import ( + DataSourceScaleway as Scaleway, + DataSourceSmartOS as SmartOS, + DataSourceUpCloud as UpCloud, ++ DataSourceVMware as VMware, + ) + from cloudinit.sources import DataSourceNone as DSNone + +@@ -50,6 +51,7 @@ DEFAULT_LOCAL = [ + RbxCloud.DataSourceRbxCloud, + Scaleway.DataSourceScaleway, + UpCloud.DataSourceUpCloudLocal, ++ VMware.DataSourceVMware, + ] + + DEFAULT_NETWORK = [ +@@ -66,6 +68,7 @@ DEFAULT_NETWORK = [ + OpenStack.DataSourceOpenStack, + OVF.DataSourceOVFNet, + UpCloud.DataSourceUpCloud, ++ VMware.DataSourceVMware, + ] + + +diff --git a/tests/unittests/test_datasource/test_vmware.py b/tests/unittests/test_datasource/test_vmware.py +new file mode 100644 +index 00000000..597db7c8 +--- /dev/null ++++ b/tests/unittests/test_datasource/test_vmware.py +@@ -0,0 +1,377 @@ ++# Copyright (c) 2021 VMware, Inc. All Rights Reserved. ++# ++# Authors: Andrew Kutz <akutz@vmware.com> ++# ++# This file is part of cloud-init. See LICENSE file for license information. ++ ++import base64 ++import gzip ++from cloudinit import dmi, helpers, safeyaml ++from cloudinit import settings ++from cloudinit.sources import DataSourceVMware ++from cloudinit.tests.helpers import ( ++ mock, ++ CiTestCase, ++ FilesystemMockingTestCase, ++ populate_dir, ++) ++ ++import os ++ ++PRODUCT_NAME_FILE_PATH = "/sys/class/dmi/id/product_name" ++PRODUCT_NAME = "VMware7,1" ++PRODUCT_UUID = "82343CED-E4C7-423B-8F6B-0D34D19067AB" ++REROOT_FILES = { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ PRODUCT_NAME_FILE_PATH: PRODUCT_NAME, ++} ++ ++VMW_MULTIPLE_KEYS = [ ++ "ssh-rsa AAAAB3NzaC1yc2EAAAA... test1@vmw.com", ++ "ssh-rsa AAAAB3NzaC1yc2EAAAA... test2@vmw.com", ++] ++VMW_SINGLE_KEY = "ssh-rsa AAAAB3NzaC1yc2EAAAA... test@vmw.com" ++ ++VMW_METADATA_YAML = """instance-id: cloud-vm ++local-hostname: cloud-vm ++network: ++ version: 2 ++ ethernets: ++ nics: ++ match: ++ name: ens* ++ dhcp4: yes ++""" ++ ++VMW_USERDATA_YAML = """## template: jinja ++#cloud-config ++users: ++- default ++""" ++ ++VMW_VENDORDATA_YAML = """## template: jinja ++#cloud-config ++runcmd: ++- echo "Hello, world." ++""" ++ ++ ++class TestDataSourceVMware(CiTestCase): ++ """ ++ Test common functionality that is not transport specific. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMware, self).setUp() ++ self.tmp = self.tmp_dir() ++ ++ def test_no_data_access_method(self): ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = None ++ ret = ds.get_data() ++ self.assertFalse(ret) ++ ++ def test_get_host_info(self): ++ host_info = DataSourceVMware.get_host_info() ++ self.assertTrue(host_info) ++ self.assertTrue(host_info["hostname"]) ++ self.assertTrue(host_info["local-hostname"]) ++ self.assertTrue(host_info["local_hostname"]) ++ self.assertTrue(host_info[DataSourceVMware.LOCAL_IPV4]) ++ ++ ++class TestDataSourceVMwareEnvVars(FilesystemMockingTestCase): ++ """ ++ Test the envvar transport. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMwareEnvVars, self).setUp() ++ self.tmp = self.tmp_dir() ++ os.environ[DataSourceVMware.VMX_GUESTINFO] = "1" ++ self.create_system_files() ++ ++ def tearDown(self): ++ del os.environ[DataSourceVMware.VMX_GUESTINFO] ++ return super(TestDataSourceVMwareEnvVars, self).tearDown() ++ ++ def create_system_files(self): ++ rootd = self.tmp_dir() ++ populate_dir( ++ rootd, ++ { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ }, ++ ) ++ self.assertTrue(self.reRoot(rootd)) ++ ++ def assert_get_data_ok(self, m_fn, m_fn_call_count=6): ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = None ++ ret = ds.get_data() ++ self.assertTrue(ret) ++ self.assertEqual(m_fn_call_count, m_fn.call_count) ++ self.assertEqual( ++ ds.data_access_method, DataSourceVMware.DATA_ACCESS_METHOD_ENVVAR ++ ) ++ return ds ++ ++ def assert_metadata(self, metadata, m_fn, m_fn_call_count=6): ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count) ++ assert_metadata(self, ds, metadata) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_subplatform(self, m_fn): ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ self.assertEqual( ++ ds.subplatform, ++ "%s (%s)" ++ % ( ++ DataSourceVMware.DATA_ACCESS_METHOD_ENVVAR, ++ DataSourceVMware.get_guestinfo_envvar_key_name("metadata"), ++ ), ++ ) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_only(self, m_fn): ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_userdata_only(self, m_fn): ++ m_fn.side_effect = ["", VMW_USERDATA_YAML, "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_vendordata_only(self, m_fn): ++ m_fn.side_effect = ["", "", VMW_VENDORDATA_YAML, ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_base64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_b64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_gzip_base64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gzip+base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_get_data_metadata_gz_b64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gz+b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_metadata_single_ssh_key(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_SINGLE_KEY ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ @mock.patch( ++ "cloudinit.sources.DataSourceVMware.guestinfo_envvar_get_value" ++ ) ++ def test_metadata_multiple_ssh_keys(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_MULTIPLE_KEYS ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ ++class TestDataSourceVMwareGuestInfo(FilesystemMockingTestCase): ++ """ ++ Test the guestinfo transport on a VMware platform. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMwareGuestInfo, self).setUp() ++ self.tmp = self.tmp_dir() ++ self.create_system_files() ++ ++ def create_system_files(self): ++ rootd = self.tmp_dir() ++ populate_dir( ++ rootd, ++ { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ PRODUCT_NAME_FILE_PATH: PRODUCT_NAME, ++ }, ++ ) ++ self.assertTrue(self.reRoot(rootd)) ++ ++ def assert_get_data_ok(self, m_fn, m_fn_call_count=6): ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = "vmware-rpctool" ++ ret = ds.get_data() ++ self.assertTrue(ret) ++ self.assertEqual(m_fn_call_count, m_fn.call_count) ++ self.assertEqual( ++ ds.data_access_method, ++ DataSourceVMware.DATA_ACCESS_METHOD_GUESTINFO, ++ ) ++ return ds ++ ++ def assert_metadata(self, metadata, m_fn, m_fn_call_count=6): ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count) ++ assert_metadata(self, ds, metadata) ++ ++ def test_ds_valid_on_vmware_platform(self): ++ system_type = dmi.read_dmi_data("system-product-name") ++ self.assertEqual(system_type, PRODUCT_NAME) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_subplatform(self, m_fn): ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ ds = self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ self.assertEqual( ++ ds.subplatform, ++ "%s (%s)" ++ % ( ++ DataSourceVMware.DATA_ACCESS_METHOD_GUESTINFO, ++ DataSourceVMware.get_guestinfo_key_name("metadata"), ++ ), ++ ) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_userdata_only(self, m_fn): ++ m_fn.side_effect = ["", VMW_USERDATA_YAML, "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_vendordata_only(self, m_fn): ++ m_fn.side_effect = ["", "", VMW_VENDORDATA_YAML, ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_metadata_single_ssh_key(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_SINGLE_KEY ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_metadata_multiple_ssh_keys(self, m_fn): ++ metadata = DataSourceVMware.load_json_or_yaml(VMW_METADATA_YAML) ++ metadata["public_keys"] = VMW_MULTIPLE_KEYS ++ metadata_yaml = safeyaml.dumps(metadata) ++ m_fn.side_effect = [metadata_yaml, "", "", ""] ++ self.assert_metadata(metadata, m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_base64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_b64(self, m_fn): ++ data = base64.b64encode(VMW_METADATA_YAML.encode("utf-8")) ++ m_fn.side_effect = [data, "b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_gzip_base64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gzip+base64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_get_data_metadata_gz_b64(self, m_fn): ++ data = VMW_METADATA_YAML.encode("utf-8") ++ data = gzip.compress(data) ++ data = base64.b64encode(data) ++ m_fn.side_effect = [data, "gz+b64", "", ""] ++ self.assert_get_data_ok(m_fn, m_fn_call_count=4) ++ ++ ++class TestDataSourceVMwareGuestInfo_InvalidPlatform(FilesystemMockingTestCase): ++ """ ++ Test the guestinfo transport on a non-VMware platform. ++ """ ++ ++ def setUp(self): ++ super(TestDataSourceVMwareGuestInfo_InvalidPlatform, self).setUp() ++ self.tmp = self.tmp_dir() ++ self.create_system_files() ++ ++ def create_system_files(self): ++ rootd = self.tmp_dir() ++ populate_dir( ++ rootd, ++ { ++ DataSourceVMware.PRODUCT_UUID_FILE_PATH: PRODUCT_UUID, ++ }, ++ ) ++ self.assertTrue(self.reRoot(rootd)) ++ ++ @mock.patch("cloudinit.sources.DataSourceVMware.guestinfo_get_value") ++ def test_ds_invalid_on_non_vmware_platform(self, m_fn): ++ system_type = dmi.read_dmi_data("system-product-name") ++ self.assertEqual(system_type, None) ++ ++ m_fn.side_effect = [VMW_METADATA_YAML, "", "", "", "", ""] ++ ds = get_ds(self.tmp) ++ ds.vmware_rpctool = "vmware-rpctool" ++ ret = ds.get_data() ++ self.assertFalse(ret) ++ ++ ++def assert_metadata(test_obj, ds, metadata): ++ test_obj.assertEqual(metadata.get("instance-id"), ds.get_instance_id()) ++ test_obj.assertEqual(metadata.get("local-hostname"), ds.get_hostname()) ++ ++ expected_public_keys = metadata.get("public_keys") ++ if not isinstance(expected_public_keys, list): ++ expected_public_keys = [expected_public_keys] ++ ++ test_obj.assertEqual(expected_public_keys, ds.get_public_ssh_keys()) ++ test_obj.assertIsInstance(ds.get_public_ssh_keys(), list) ++ ++ ++def get_ds(temp_dir): ++ ds = DataSourceVMware.DataSourceVMware( ++ settings.CFG_BUILTIN, None, helpers.Paths({"run_dir": temp_dir}) ++ ) ++ ds.vmware_rpctool = "vmware-rpctool" ++ return ds ++ ++ ++# vi: ts=4 expandtab +diff --git a/tests/unittests/test_ds_identify.py b/tests/unittests/test_ds_identify.py +index 1d8aaf18..8617d7bd 100644 +--- a/tests/unittests/test_ds_identify.py ++++ b/tests/unittests/test_ds_identify.py +@@ -649,6 +649,50 @@ class TestDsIdentify(DsIdentifyBase): + """EC2: bobrightbox.com in product_serial is not brightbox'""" + self._test_ds_not_found('Ec2-E24Cloud-negative') + ++ def test_vmware_no_valid_transports(self): ++ """VMware: no valid transports""" ++ self._test_ds_not_found('VMware-NoValidTransports') ++ ++ def test_vmware_envvar_no_data(self): ++ """VMware: envvar transport no data""" ++ self._test_ds_not_found('VMware-EnvVar-NoData') ++ ++ def test_vmware_envvar_no_virt_id(self): ++ """VMware: envvar transport success if no virt id""" ++ self._test_ds_found('VMware-EnvVar-NoVirtID') ++ ++ def test_vmware_envvar_activated_by_metadata(self): ++ """VMware: envvar transport activated by metadata""" ++ self._test_ds_found('VMware-EnvVar-Metadata') ++ ++ def test_vmware_envvar_activated_by_userdata(self): ++ """VMware: envvar transport activated by userdata""" ++ self._test_ds_found('VMware-EnvVar-Userdata') ++ ++ def test_vmware_envvar_activated_by_vendordata(self): ++ """VMware: envvar transport activated by vendordata""" ++ self._test_ds_found('VMware-EnvVar-Vendordata') ++ ++ def test_vmware_guestinfo_no_data(self): ++ """VMware: guestinfo transport no data""" ++ self._test_ds_not_found('VMware-GuestInfo-NoData') ++ ++ def test_vmware_guestinfo_no_virt_id(self): ++ """VMware: guestinfo transport fails if no virt id""" ++ self._test_ds_not_found('VMware-GuestInfo-NoVirtID') ++ ++ def test_vmware_guestinfo_activated_by_metadata(self): ++ """VMware: guestinfo transport activated by metadata""" ++ self._test_ds_found('VMware-GuestInfo-Metadata') ++ ++ def test_vmware_guestinfo_activated_by_userdata(self): ++ """VMware: guestinfo transport activated by userdata""" ++ self._test_ds_found('VMware-GuestInfo-Userdata') ++ ++ def test_vmware_guestinfo_activated_by_vendordata(self): ++ """VMware: guestinfo transport activated by vendordata""" ++ self._test_ds_found('VMware-GuestInfo-Vendordata') ++ + + class TestBSDNoSys(DsIdentifyBase): + """Test *BSD code paths +@@ -1136,7 +1180,240 @@ VALID_CFG = { + 'Ec2-E24Cloud-negative': { + 'ds': 'Ec2', + 'files': {P_SYS_VENDOR: 'e24cloudyday\n'}, +- } ++ }, ++ 'VMware-NoValidTransports': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-NoData': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-NoVirtID': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ ], ++ }, ++ 'VMware-EnvVar-Metadata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-Userdata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-EnvVar-Vendordata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo', ++ 'ret': 0, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_has_envvar_vmx_guestinfo_vendordata', ++ 'ret': 0, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-NoData': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-NoVirtID': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ ], ++ }, ++ 'VMware-GuestInfo-Metadata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-Userdata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 1, ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, ++ 'VMware-GuestInfo-Vendordata': { ++ 'ds': 'VMware', ++ 'mocks': [ ++ { ++ 'name': 'vmware_has_rpctool', ++ 'ret': 0, ++ 'out': '/usr/bin/vmware-rpctool', ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_metadata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_userdata', ++ 'ret': 1, ++ }, ++ { ++ 'name': 'vmware_rpctool_guestinfo_vendordata', ++ 'ret': 0, ++ 'out': '---', ++ }, ++ MOCK_VIRT_IS_VMWARE, ++ ], ++ }, + } + + # vi: ts=4 expandtab +diff --git a/tools/.github-cla-signers b/tools/.github-cla-signers +index 689d7902..cbfa883c 100644 +--- a/tools/.github-cla-signers ++++ b/tools/.github-cla-signers +@@ -1,5 +1,6 @@ + ader1990 + ajmyyra ++akutz + AlexBaranowski + Aman306 + andrewbogott +diff --git a/tools/ds-identify b/tools/ds-identify +index 2f2486f7..c01eae3d 100755 +--- a/tools/ds-identify ++++ b/tools/ds-identify +@@ -125,7 +125,7 @@ DI_DSNAME="" + # be searched if there is no setting found in config. + DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \ + CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \ +-OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud" ++OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud VMware" + DI_DSLIST="" + DI_MODE="" + DI_ON_FOUND="" +@@ -1350,6 +1350,80 @@ dscheck_IBMCloud() { + return ${DS_NOT_FOUND} + } + ++vmware_has_envvar_vmx_guestinfo() { ++ [ -n "${VMX_GUESTINFO:-}" ] ++} ++ ++vmware_has_envvar_vmx_guestinfo_metadata() { ++ [ -n "${VMX_GUESTINFO_METADATA:-}" ] ++} ++ ++vmware_has_envvar_vmx_guestinfo_userdata() { ++ [ -n "${VMX_GUESTINFO_USERDATA:-}" ] ++} ++ ++vmware_has_envvar_vmx_guestinfo_vendordata() { ++ [ -n "${VMX_GUESTINFO_VENDORDATA:-}" ] ++} ++ ++vmware_has_rpctool() { ++ command -v vmware-rpctool >/dev/null 2>&1 ++} ++ ++vmware_rpctool_guestinfo_metadata() { ++ vmware-rpctool "info-get guestinfo.metadata" ++} ++ ++vmware_rpctool_guestinfo_userdata() { ++ vmware-rpctool "info-get guestinfo.userdata" ++} ++ ++vmware_rpctool_guestinfo_vendordata() { ++ vmware-rpctool "info-get guestinfo.vendordata" ++} ++ ++dscheck_VMware() { ++ # Checks to see if there is valid data for the VMware datasource. ++ # The data transports are checked in the following order: ++ # ++ # * envvars ++ # * guestinfo ++ # ++ # Please note when updating this function with support for new data ++ # transports, the order should match the order in the _get_data ++ # function from the file DataSourceVMware.py. ++ ++ # Check to see if running in a container and the VMware ++ # datasource is configured via environment variables. ++ if vmware_has_envvar_vmx_guestinfo; then ++ if vmware_has_envvar_vmx_guestinfo_metadata || \ ++ vmware_has_envvar_vmx_guestinfo_userdata || \ ++ vmware_has_envvar_vmx_guestinfo_vendordata; then ++ return "${DS_FOUND}" ++ fi ++ fi ++ ++ # Do not proceed unless the detected platform is VMware. ++ if [ ! "${DI_VIRT}" = "vmware" ]; then ++ return "${DS_NOT_FOUND}" ++ fi ++ ++ # Do not proceed if the vmware-rpctool command is not present. ++ if ! vmware_has_rpctool; then ++ return "${DS_NOT_FOUND}" ++ fi ++ ++ # Activate the VMware datasource only if any of the fields used ++ # by the datasource are present in the guestinfo table. ++ if { vmware_rpctool_guestinfo_metadata || \ ++ vmware_rpctool_guestinfo_userdata || \ ++ vmware_rpctool_guestinfo_vendordata; } >/dev/null 2>&1; then ++ return "${DS_FOUND}" ++ fi ++ ++ return "${DS_NOT_FOUND}" ++} ++ + collect_info() { + read_uname_info + read_virt +-- +2.27.0 + diff --git a/SOURCES/ci-Revert-unnecesary-lcase-in-ds-identify-978.patch b/SOURCES/ci-Revert-unnecesary-lcase-in-ds-identify-978.patch new file mode 100644 index 0000000..2e5349c --- /dev/null +++ b/SOURCES/ci-Revert-unnecesary-lcase-in-ds-identify-978.patch @@ -0,0 +1,47 @@ +From 532a36edf0dea2b98835bd08e285bec9c50eb0f9 Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito <eesposit@redhat.com> +Date: Fri, 14 Jan 2022 16:42:41 +0100 +Subject: [PATCH 5/6] Revert unnecesary lcase in ds-identify (#978) + +RH-Author: Emanuele Giuseppe Esposito <eesposit@redhat.com> +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [5/6] 95634e4b42e3abfb91182b090c312eef29c63e54 +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy <mmorsy@redhat.com> +RH-Acked-by: Eduardo Otubo <otubo@redhat.com> + +commit f516a7d37c1654addc02485e681b4358d7e7c0db +Author: Andrew Kutz <101085+akutz@users.noreply.github.com> +Date: Fri Aug 13 14:30:55 2021 -0500 + + Revert unnecesary lcase in ds-identify (#978) + + This patch reverts an unnecessary lcase optimization in the + ds-identify script. SystemD documents the values produced by + the systemd-detect-virt command are lower case, and the mapping + table used by the FreeBSD check is also lower-case. + + The optimization added two new forked processes, needlessly + causing overhead. + +Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> +--- + tools/ds-identify | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/tools/ds-identify b/tools/ds-identify +index 0e12298f..7b782462 100755 +--- a/tools/ds-identify ++++ b/tools/ds-identify +@@ -449,7 +449,7 @@ detect_virt() { + read_virt() { + cached "$DI_VIRT" && return 0 + detect_virt +- DI_VIRT="$(echo "${_RET}" | tr '[:upper:]' '[:lower:]')" ++ DI_VIRT="${_RET}" + } + + is_container() { +-- +2.27.0 + diff --git a/SOURCES/ci-Update-dscheck_VMware-s-rpctool-check-970.patch b/SOURCES/ci-Update-dscheck_VMware-s-rpctool-check-970.patch new file mode 100644 index 0000000..492f1c6 --- /dev/null +++ b/SOURCES/ci-Update-dscheck_VMware-s-rpctool-check-970.patch @@ -0,0 +1,97 @@ +From cc79cb3958b943b755a9b11b3e87ce820058ccaa Mon Sep 17 00:00:00 2001 +From: Emanuele Giuseppe Esposito <eesposit@redhat.com> +Date: Fri, 14 Jan 2022 16:41:47 +0100 +Subject: [PATCH 4/6] Update dscheck_VMware's rpctool check (#970) + +RH-Author: Emanuele Giuseppe Esposito <eesposit@redhat.com> +RH-MergeRequest: 43: Datasource for VMware +RH-Commit: [4/6] 6f4d732c55c521869210d8aeedfa1150ea5a92f8 +RH-Bugzilla: 2040704 +RH-Acked-by: Mohamed Gamal Morsy <mmorsy@redhat.com> +RH-Acked-by: Eduardo Otubo <otubo@redhat.com> + +commit 7781dec3306e9467f216cfcb36b7e10a8b38547a +Author: Shreenidhi Shedi <53473811+sshedi@users.noreply.github.com> +Date: Fri Aug 13 00:40:39 2021 +0530 + + Update dscheck_VMware's rpctool check (#970) + + This patch updates the dscheck_VMware function's use of "vmware-rpctool". + + When checking to see if a "guestinfo" property is set. + Because a successful exit code can occur even if there is an empty + string returned, it is possible that the VMware datasource will be + loaded as a false-positive. This patch ensures that in addition to + validating the exit code, the emitted output is also examined to ensure + a non-empty value is returned by rpctool before returning "${DS_FOUND}" + from "dscheck_VMware()". + +Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> +--- + tools/ds-identify | 15 +++++++++------ + 1 file changed, 9 insertions(+), 6 deletions(-) + +diff --git a/tools/ds-identify b/tools/ds-identify +index c01eae3d..0e12298f 100755 +--- a/tools/ds-identify ++++ b/tools/ds-identify +@@ -141,6 +141,7 @@ error() { + debug 0 "$@" + stderr "$@" + } ++ + warn() { + set -- "WARN:" "$@" + debug 0 "$@" +@@ -344,7 +345,6 @@ geom_label_status_as() { + return $ret + } + +- + read_fs_info_freebsd() { + local oifs="$IFS" line="" delim="," + local ret=0 labels="" dev="" label="" ftype="" isodevs="" +@@ -404,7 +404,6 @@ cached() { + [ -n "$1" ] && _RET="$1" && return || return 1 + } + +- + detect_virt() { + local virt="${UNAVAILABLE}" r="" out="" + if [ -d /run/systemd ]; then +@@ -450,7 +449,7 @@ detect_virt() { + read_virt() { + cached "$DI_VIRT" && return 0 + detect_virt +- DI_VIRT=${_RET} ++ DI_VIRT="$(echo "${_RET}" | tr '[:upper:]' '[:lower:]')" + } + + is_container() { +@@ -1370,16 +1369,20 @@ vmware_has_rpctool() { + command -v vmware-rpctool >/dev/null 2>&1 + } + ++vmware_rpctool_guestinfo() { ++ vmware-rpctool "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]" ++} ++ + vmware_rpctool_guestinfo_metadata() { +- vmware-rpctool "info-get guestinfo.metadata" ++ vmware_rpctool_guestinfo "metadata" + } + + vmware_rpctool_guestinfo_userdata() { +- vmware-rpctool "info-get guestinfo.userdata" ++ vmware_rpctool_guestinfo "userdata" + } + + vmware_rpctool_guestinfo_vendordata() { +- vmware-rpctool "info-get guestinfo.vendordata" ++ vmware_rpctool_guestinfo "vendordata" + } + + dscheck_VMware() { +-- +2.27.0 + diff --git a/SPECS/cloud-init.spec b/SPECS/cloud-init.spec index 425df39..d6ae704 100644 --- a/SPECS/cloud-init.spec +++ b/SPECS/cloud-init.spec @@ -6,7 +6,7 @@ Name: cloud-init Version: 21.1 -Release: 7%{?dist}.3 +Release: 7%{?dist}.4 Summary: Cloud instance init scripts Group: System Environment/Base @@ -40,6 +40,18 @@ Patch15: ci-cc_ssh.py-fix-private-key-group-owner-and-permission.patch Patch16: ci-fix-error-on-upgrade-caused-by-new-vendordata2-attri.patch # For bz#2028756 - [RHEL-8] Above 19.2 of cloud-init fails to configure routes when configuring static and default routes to the same destination IP [rhel-8.5.0.z] Patch17: ci-cloudinit-net-handle-two-different-routes-for-the-sa.patch +# For bz#2040690 - [RHEL8] [Azure] cloud-init fails to configure the system [rhel-8.5.0.z] +#Patch18: ci-Add-gdisk-and-openssl-as-deps-to-fix-UEFI-Azure-init.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch19: ci-Datasource-for-VMware-953.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch20: ci-Change-netifaces-dependency-to-0.10.4-965.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch21: ci-Update-dscheck_VMware-s-rpctool-check-970.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +Patch22: ci-Revert-unnecesary-lcase-in-ds-identify-978.patch +# For bz#2040704 - [cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z] +#Patch23: ci-Add-netifaces-package-as-a-Requires-in-cloud-init.sp.patch BuildArch: noarch @@ -88,6 +100,10 @@ Requires: shadow-utils Requires: util-linux Requires: xfsprogs Requires: dhcp-client +# https://bugzilla.redhat.com/show_bug.cgi?id=2040690 +Requires: gdisk +Requires: openssl +Requires: python3-netifaces %{?systemd_requires} @@ -231,6 +247,18 @@ fi %config(noreplace) %{_sysconfdir}/rsyslog.d/21-cloudinit.conf %changelog +* Wed Jan 19 2022 Jon Maloy <jmaloy@redhat.com> - 21.1-7.el8_5.4 +- ci-Add-gdisk-and-openssl-as-deps-to-fix-UEFI-Azure-init.patch [bz#2040690] +- ci-Datasource-for-VMware-953.patch [bz#2040704] +- ci-Change-netifaces-dependency-to-0.10.4-965.patch [bz#2040704] +- ci-Update-dscheck_VMware-s-rpctool-check-970.patch [bz#2040704] +- ci-Revert-unnecesary-lcase-in-ds-identify-978.patch [bz#2040704] +- ci-Add-netifaces-package-as-a-Requires-in-cloud-init.sp.patch [bz#2040704] +- Resolves: bz#2040690 + ([RHEL8] [Azure] cloud-init fails to configure the system [rhel-8.5.0.z]) +- Resolves: bz#2040704 + ([cloud-init][RHEL8] Support for cloud-init datasource 'cloud-init-vmware-guestinfo' [rhel-8.5.0.z]) + * Wed Dec 08 2021 Jon Maloy <jmaloy@redhat.com> - 21.1-7.el8_5.3 - ci-cloudinit-net-handle-two-different-routes-for-the-sa.patch [bz#2028756] - Resolves: bz#2028756