# Remote reinstall with vnc

!!! warning

    This section needs attention and isn't meant to be simply a copy/paste operation. "Some Thinking Required [TM]" mantra applies here.

Assuming that you need to remotely reinstall a physical server (like a sponsored node in a remote DC) where you don't control dhcp (so no pxe install) and also without remote console access (so ipmi nor keyboard/video/mouse - kvm - feature), you can always combine multiple elements all together :

 * Downloading kernel and initrd from pxe images (vmlinuz and initrd.img)

 * kexec (from kexec-tools pkgs) to reboot into a new kernel/initrd without rebooting

 * anaconda parameters to init the network interface with correct fixed ip address/mask/gateway/dns (no dhcp, remember ?)

 * boot into install mode and start vnc with a password (so that you can reconnect to console to finish installation)

## Requirements check

Before reinstalling a node from a major version to a new major version, you need first to verify the HCL and if the network card and HBA is still supported as a kernel module. It happens that from centos release to new one, some kernel modules are gone (in the rhel kernel) and so you wouldn't have working network, nor disks.

### Network and Storage HBA info gathering

So from the machine that you need to reinstall (you probably have ssh/root access somehow) verify which kernel module is used for network card. Let's assume that it's `enp2s0f0` :

```

ethtool -i enp2s0f0|egrep 'driver|^version'

driver: mlx5_core

version: 4.18.0-338.el8.x86_64

```

So our kernel module is `mlx5_core`. Let's now check the Hard disk, assuming that it's `/dev/sda`.

We can use `udevadm` to show the 

```

udevadm info -a -n /dev/sda| egrep 'looking|DRIVER'

  looking at device '/devices/pci0000:00/0000:00:03.0/0000:03:00.0/host0/target0:2:0/0:2:0:0/block/sda':

    DRIVER==""

  looking at parent device '/devices/pci0000:00/0000:00:03.0/0000:03:00.0/host0/target0:2:0/0:2:0:0':

    DRIVERS=="sd"

  looking at parent device '/devices/pci0000:00/0000:00:03.0/0000:03:00.0/host0/target0:2:0':

    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:03.0/0000:03:00.0/host0':

    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:03.0/0000:03:00.0':

    DRIVERS=="megaraid_sas"

  looking at parent device '/devices/pci0000:00/0000:00:03.0':

    DRIVERS=="pcieport"

  looking at parent device '/devices/pci0000:00':

    DRIVERS==""

```

So device /dev/sda is attached to host0, which uses kernel module `megaraid_sas`

### Check on target major version

Now that we have our `mlx5_core` and `megaraid_sas` kernel modules, we have to verify on a target system that they exist in new kernel :

```

for i in mlx5_core megaraid_sas ; do modinfo $i|egrep name; done

filename:       /lib/modules/4.18.0-338.el8.x86_64/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko.xz

name:           mlx5_core

filename:       /lib/modules/4.18.0-338.el8.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz

name:           megaraid_sas

```

Fine, they exist so our hardware should be compatible for a (re)install with new major version.

## Remote reinstall

We can just define from which closest mirror we want to reinstall, define a unique and temporary vnc password (only used during the install, not used anymore after, generated with `pwgen -s 8 1`), a hostname for install and proceed with install :

```

mirror_url="http://mirror.centos.org/centos/8-stream/"

arch=$(uname -m)

hostname="reinstall1.dev.centos.org"

vnc_pass="Xs9x0mx9"

yum install -y wget kexec-tools 

cd /boot 

curl --location --fail ${mirror_url}/BaseOS/${arch}/os/images/pxeboot/vmlinuz > vmlinuz.install

curl --location --fail ${mirror_url}/BaseOS/${arch}/os/images/pxeboot/initrd.img > initrd.img.install

```

!!! info

    Worth knowing that if you want to access a user/pass protected mirror , you can use http://user:pass@mirror.fqdn or even better : https to ensure creds aren't sent in clear text

Let's gather some network informations : 

```

dns=$(cat /etc/resolv.conf |grep nameserver|head -n1|awk '{print $2}')

gateway=$(ip route|grep default|head -n 1|awk '{print $3}') 

eth_dev=$(ip route|grep default|head -n 1|awk '{print $5}')

ip_addr=$(ip addr show dev $eth_dev|grep inet|grep $eth_dev|head -n 1|awk '{print $2}'|cut -f 1 -d '/')

netmask=$(ipcalc --netmask $( ip addr show dev $eth_dev|grep inet|grep $eth_dev|head -n 1|awk '{print $2}')|cut -f 2 -d '=')

ip6_addr=$(ip -6 addr show dev $eth_dev|grep glob|awk '{print $2}')

ip6_gw=$(ip -6 route|grep default|awk '{print $3}')

echo "list of devices : "

echo "==================="

ip addr|grep qdisc|awk '{print $2}'|tr -d ':'

if [[ $eth_dev = *bond* ]] ; then

  echo "Bonding interface found ! "

  eth_dev=$(cat /proc/net/bonding/bond0 |grep 'Slave Interface'|head -n 1|awk '{print $3}')

  echo "Real device is $eth_dev"

elif [[ $eth_dev = *eth* ]] ; then

  echo "Device is still named eth[*] so using net.ifnames=0"

  eth_opts="net.ifnames=0"

  echo "Eth device = $eth_dev"

else

  echo "Eth device = $eth_dev"

  eth_opts=""

fi

echo ip=$ip_addr netmask=$netmask gateway=$gateway dns=$dns

echo IPv6 : $ip6_addr / gw : $ip6_gw

echo "nmcli con mod $eth_dev ipv6.method manual ipv6.address $ip6_addr ipv6.gateway $ip6_gw ; nmcli con up $eth_dev"

echo "eth device= $eth_dev"

echo "eth options = $eth_opts"

```

!!! danger

    Now verify closely the informations and if that looks correct (remember : thinking required), select *one* of the following possible ways to kick the reinstall. In case of issue, you can always ask the remote DC to just `reset` the node and it should come back on os installed on disk

```

# Normal

kexec -l vmlinuz.install --append="$eth_opts biosdevname=0 rd.neednet=1 ksdevice=$eth_dev inst.repo=${mirror_url}/BaseOS/${arch}/os/ inst.lang=en_GB inst.keymap=be-latin1 inst.vnc inst.vncpassword=$vnc_pass ip=$ip_addr::$gateway:$netmask:$hostname:$eth_dev:none nameserver=$dns" --initrd=initrd.img.install && kexec -e

# For Dell and biosdevname like eno1 etc

kexec -l vmlinuz.install --append="$eth_opts rd.neednet=1 ksdevice=$eth_dev inst.repo=${mirror_url}/BaseOS/${arch}/os/ inst.lang=en_GB inst.keymap=be-latin1 inst.vnc inst.vncpassword=$vnc_pass ip=$ip_addr::$gateway:$netmask:$hostname:$eth_dev:none nameserver=$dns" --initrd=initrd.img.install && kexec -e

# With console on ttyS0 (serial redirection, normally not needed)

kexec -l vmlinuz.install --append="$eth_opts biosdevname=0 rd.neednet=1 ksdevice=$eth_dev inst.repo=${mirror_url}/BaseOS/${arch}/os/ inst.lang=en_GB inst.keymap=be-latin1 inst.vnc inst.vncpassword=$vnc_pass ip=$ip_addr::$gateway:$netmask:$hostname:$eth_dev:none nameserver=$dns console=ttyS0,115200n8" --initrd=initrd.img.install && kexec -e

```

As you launched this over ssh, you'll lose your connection (as new kernel will be started).

From your workstation (or elsewhere) you can try to test pinging the machine and wait for the server to have fetched stage2 image from mirror and launched anaconda through vnc. If machine responds to `ping` you can just wait for vnc with a snippet like : 

```

host="ip.address.of.reinstalled.node"

while true 

do 

  sleep 2 

  >/dev/null 2>&1 >/dev/tcp/${host}/5901 

  if [ "$?" = "0" ] ; then

    notify-send "${host} VNC is ready to be connected"

    echo "${host} ready for vnc connection"|festival --tts

    break

  fi

done

# launching vnc

echo "launching vnc on ${host}"

vncviewer ${host}:1 &

```

# Default settings when reinstalling (manually) a node

Some default settings that we use by default:

 * package selection: minimal

 * temporary root_password (will be change when we init with ansible)

 * hard-disks layout

   * hardware raid controller : done at the HBA level

   * multiple disks (jbod) : software raid 1 (or 5, depending on case)

     * raid 1:

       * /boot : raid1 device, ext4

       * VG for the rest, also with raid1, extended to max capacity

         * / LV : ext4, 10G

         * /home LV : ext4, 2G by default, more for mirror

         * swap LV : 2G

  * single disk : same layout as above, without the md/raid part