HA Router and Failover
======================
This readme describes steps to add multiple HA OpenShift routers with
failover capability to achieve several nines of availability.


Build and Test
--------------
1.  Verify docker image build and run tests.

        $ make -f makefile.test  #  or make -f makefile.test image
        $ make -f makefile.test test


Pre-requisites/Prep Time
------------------------

1. Launch a OpenShift cluster via whatever mechanism you use. The steps
   below assume you are doing this on a dev machine using vagrant.

        $ export OPENSHIFT_DEV_CLUSTER=1
        $ cd $this-repo-git-co-dir  # cloned from git@github.com:ramr/origin
        $ vagrant up


2. Wait for the cluster to come up and then start the OpenShift router
   with two (_2_) replicas.

        $ vagrant ssh minion-1  # (or master or minion-2).
        #  Ensure KUBECONFIG is set or else set it.
        [ -n "$KUBECONFIG" ] ||  \
           export KUBECONFIG=/openshift.local.config/master/admin.kubeconfig
        #  openshift kube get dc,rc,pods,se,mi,routes
        oadm router arparp --replicas=2


3. Wait for the Router pods to get into running state (I'm just sitting
   here watching the wheels go round and round).

        $ vagrant ssh minion-1 # (or master or minion-2).
        pods="openshift/origin-haproxy-router|openshift/origin-deployer"
        while openshift kube get pods | egrep -e "$pods" |   \
                grep "Pending" > /dev/null; do
            echo -n "."
            #  "OkOk"
            sleep 1
        done
        echo ""


4. Check that the two OpenShift router replicas are up and serving.

        $ #  This will be a bit slow, but it should return a 503 HTTP code
        $ #  indicating that haproxy is serving on port 80.
        $ vagrant ssh minion-1
        [minion-1@localhost ~]$ sudo docker ps  | grep "openshift/origin-haproxy-router"
        [minion-1@localhost ~]$ curl -s -o /dev/null -w "%{http_code}\n"  http://localhost/

        $ #  Repeat on minion-2:
        $ vagrant ssh minion-2
        [minion-2@localhost ~]$ sudo docker ps  | grep "openshift/origin-haproxy-router"
        [minion-2@localhost ~]$ curl -s -o /dev/null -w "%{http_code}\n"  http://localhost/


5. Create an user, project and app.

        $ vagrant ssh minion-1
        #  Add user and project.
        oadm policy add-role-to-user view test-admin
        oadm new-project test --display-name="Failover Sample" \
           --description="Router Failover" --admin=test-admin
        #  Create a test app using the template.
        cd /vagrant/hack/exp/router-failover
        oc create -n test -f conf/hello-openshift-template.json

        echo "Wait for the app to startup and check app is reachable."
        for ip in 10.245.2.3 10.245.2.4; do
          curl -H "Host: hello.openshift.test" -o /dev/null -s -m 5  \
               -w "%{http_code}\n" http://$ip/
        done
        echo "Ensure HTTP status code is 200 for both http://10.245.2.{3,4}"
        #  && echo "YAY"


6. Ensure you can get to the hello openshift app from inside/outside the vm.

        $ #  minion-{1,2} use IPs 10.245.2.{3,4} in the dev environment.
        for ip in 10.245.2.3 10.245.2.4; do
          echo "$ip: $(curl -s --resolve hello.openshift.test:80:$ip  \
                            -m 5 http://hello.openshift.test)"
        done


HA Routing Failover Setup
=========================

1. Copy the router HA settings example config and edit it as needed.

        $ cd /vagrant/hack/exp/router-failover
        $ cp conf/settings.example  settings.minion-1
        $ cp conf/settings.example  settings.minion-2
        $ #
        $ #  And as per your environment, set/edit the values for
        $ #    ADMIN_EMAILS, EMAIL_FROM, SMTP_SERVER,
        $ #    PRIMARY_HA_VIPS, SLAVE_HA_VIPS and INTERFACE.

2. For demo purposes, we are going to flip the PRIMARY and SLAVE groups
   on minion-2 ... this allows both minions to serve in an Active-Active
   fashion.

        $ #  Flip PRIMARY+SLAVE groups on minion-2 ("Papoy?! Ah Papoy!!").
        $ sed -i "s/^PRIMARY_GROUPS=\(.*\)/PRIMARY_GROUPS_OLD=\1/g;
                  s/^SLAVE_GROUPS=\(.*\)/PRIMARY_GROUPS=\1/g;
                  s/^PRIMARY_GROUPS_OLD=\(.*\)/SLAVE_GROUPS=\1/g;" \
              settings.minion-2

        $ #  Check what the differences are on the minions.
        $ diff conf/settings.example  settings.minion-1
        $ diff conf/settings.example  settings.minion-2


3. Optionally clear the config - just so that we have a completely clean
   slate. Step 4 below does this - but this is here just for my demo env
   reuse purposes.

        $ #  Run these commands on the minions via vagrant ssh minion-{1,2}
        $ #    sudo service keepalived stop
        $ #    sudo rm -f /etc/keepalived/keepalived.conf

        $ #  OkOk
        for m in minion-1 minion-2; do
           vagrant ssh $m -c "sudo service keepalived stop;  \
                              sudo rm -f /etc/keepalived/keepalived.conf"
        done


4. Setup router HA with failover using the 2 config files we created.

        $ #  Run these commands on the minions via vagrant ssh minion-{1,2}
        $ #    cd /vagrant/hack/exp/router-failover
        $ #    sudo ./failover-setup.sh settings.minion-{1,2}

        $ #  OkOk - minion-1
        for m in minion-1 minion-2; do
           vagrant ssh $m -c "cd /vagrant/hack/exp/router-failover;  \
                              sudo ./failover-setup.sh settings.$m"
        done


5. On each minion, you can check what VIPs are being serviced by that
   minion via `ip a ls dev enp0s8`. Substitute the appropriate interface
   name for `enp0s8` in your environment.

        $ #  "minions laughing" ...
        for m in minion-1 minion-2; do
           vagrant ssh $m -c "ip a ls dev enp0s8"
        done


6. Check that you can get to the hello openshift app using the VIPs from
   inside/outside the vms.

        for ip in 10.245.2.90 10.245.2.111 10.245.2.222 10.245.2.223; do
          echo "$ip: $(curl -s --resolve hello.openshift.test:80:$ip  \
                            -m 5 http://hello.openshift.test)"
        done
        #  && echo "YAY"


HA Routing Failover Demo
========================
Whilst following the steps below, you can also monitor one of the VIPs on a
terminal on your host system. This just busy loops sending requests to a
specific VIP.

        tko="--connect-timeout 2"  #  Maybe use -m 2 instead.
        resolver="--resolve hello.openshift.test:80:10.245.2.111"
        while true; do
          echo "$(date): $(curl -s $tko $resolver hello.openshift.test)"
        done | tee /tmp/foo


HA Simple Failover Test (keepalived)
====================================
The simplest test on VIP failover is to stop keepalived on one of the
minions.

        $ vagrant ssh minion-1

        $ #  Check which VIPs are served by this minion.
        ip a ls dev enp0s8

        $ #  Make sure the VIP in the busy loop above 10.245.2.111 is
        $ #  "owned"/serviced by this minion. Or then use a VIP that's
        $ #  serviced by this minion in the above mentioned busy looper
        $ #  monitoring script (while true; curl ... done).
        sudo service keepalived stop

        $ vagrant ssh minion-2
        #  Check that the VIPs from minion-1 are taken over by this minion.
        ip a ls dev enp0s8

        $ vagrant ssh minion-1
        $ #  Set things back to a "good" state by starting back keepalived.
        sudo service keepalived start

        $ #  Check the VIPs served by this minion.
        ip a ls dev enp0s8


HA Hard Failover Test (bring down the minion)
=============================================
The hard failover VIP test basically involves stopping the whole shebang
(keepalived, openshift-router and haproxy) by bringing down one of
the minions.

1. Halt one of the minions ("Aww") ...

        $ #  If you are monitoring a specific VIP ala 10.245.2.111 in the
        $ #  example mentioned above, then bring down the minion that's
        $ #  "owns" that VIP. For now, bringing a random one down.
        $ vagrant halt minion-$((RANDOM%2 + 1))


2. Check that you can still get to the hello openshift app using the VIPs
   from inside/outside the vms.

        for ip in 10.245.2.90 10.245.2.111 10.245.2.222 10.245.2.223; do
          echo "$ip: $(curl -s --resolve hello.openshift.test:80:$ip  \
                            -m 5 http://hello.openshift.test)"
        done
        $ #  && echo "YAY"


3. Bring back the minion ("YAY") ...

        $ vagrant up minion-{1,2}


4. Wait for the minion to come back online.

5. Check how the VIPs are balanced between the 2 minions.

        for m in minion-1 minion-2; do
          vagrant ssh $m -c "ip a ls dev enp0s8"
        done

6. Check that you can still get to the hello openshift app using the VIPs
   from inside/outside the vms.

        for ip in 10.245.2.90 10.245.2.111 10.245.2.222 10.245.2.223; do
          echo "$ip: $(curl -s --resolve hello.openshift.test:80:$ip  \
                            -m 5 http://hello.openshift.test)"
        done
        $ #  && echo "YAY"



HA Soft Failover Test
=====================

1. Eventually this would test the keepalived process - but for now this
   just shows how long the Kubernetes Replication Controller takes to
   restart the services.

        $ #  Stop the router on one of the minions ("Aaw").
        $ vagrant ssh minion-$((RANDOM%2 + 1))
        sudo kill -9 $(ps -e -opid,args | grep openshift-router |  \
                          grep -v grep | awk '{print $1}')
        $ # OR:
        sudo docker rm -f $(sudo docker ps |  \
                               grep openshift/origin-haproxy-router |  \
                               awk '{print $1}')

2. Check that you can still get to the hello openshift app using the VIPs
   from inside/outside the vms.

        for ip in 10.245.2.90 10.245.2.111 10.245.2.222 10.245.2.223; do
          echo "$ip: $(curl -s --resolve hello.openshift.test:80:$ip  \
                            -m 5 http://hello.openshift.test)"
        done
        $ #  && echo "YAY"
        $ #  Wait for the router to come back up and run above check again.



TODOs/Edge CASES:
-----------------

##  *Beware of the dog - it bites! You have been warned*
There's a 2 second delay (process existence check) as of now, we can
tune this up/down appropriately.
And it is pertinent to mention here that this solution is not true
fault-tolerance (100% availbility) - its just failover capability to
provide high availability (99.[9]{n}% availability - cheap but by no
means perfect).
So be aware of this and use it appropriately within your environment.

One alternative to achieve several more 9s of availability is to
  * stop keepalived immediately if the router or the docker container
    running the router goes down.
  * And start keepalived start it when the router comes back up because
    the replication controller notices things ain't kosher.
But the bang for buck here is a low.


##  *Sound Effects*
Link quoted sound effects (ala "OkOk") to

        http://www.soundboard.com/sb/minions

