Building Enterprise-grade Kubernetes from scratch with Rancher Labs
Start Here
Following my journey in evaluating and building enterprise-grade Kubernetes platform from scratch on VMware ESXi hypervisor with different software stacks / frameworks, my next stop is to take a good look at what Rancher Labs has to offer.
The first thing that hits your face when you see Rancher UI is that it is SO CLEAN! Without knowing anything, you still feel curious and confident to navigate the UI. Simple but intuitive. It makes you feel at ease in understanding and getting familiar with, in a relatively shorter time.
Just like my previous articles on distributed Kubernetes and Photon-1.2, I am focussing of building distributed Kubernetes platform, but using Rancher Labs software stack this time. I am not venturing into the training aspect of Rancher Labs stack but I have provided links under 'Lab Exercises' for the same. Also, their YouTube channel is a great resource.
What you see below is the Rancher UI showing a total of 9 virtual machines in total out of which, 3 VMs are dedicated for ETCD distributed key/value store, 3 VMs dedicated to host all ORCHESTRATION (Kubernetes) components and 3 VMs are dedicated for COMPUTE to run customer applications/workloads.
Having said that, there are 8 more virtual machines that work in tandem as part of this Hands-On lab. In total, I have 17 virtual machines that make up this single Hands-On lab. The approach is to build a truly distributed systems and have active HA wherever possible. So when I say " enterprise-grade " in the title, I truly mean it.
Needless to say, I am going to hand hold and walk you though every single component step-by-step (refer links under Sections), and showcase how to integrate them all and then finally install and configure Rancher Labs software stack. And at the end, you will have a fully distributed Kubernetes platform on top of it at your disposal. So hang tight!
Please note that I will be using terms VMs, hosts and nodes interchangeably.
Here is the Architecture diagram:
Here is the list of all the virtual machines, their roles and services they provide:
1 x EDGE VM running EFW distribution, configured as network gateway
1 x DNS VM - CentOS7 VM running FreeIPA for DNS service
1 x JUMPBOX VM - CentOS7 VM where you log in that has visibility to all the VMs. This is also referred to as the Bastion host.
1 x OPENFILER VM - to provide shared storage (NFS) for the backup setup ETCD nodes
1 x MYSQL VM - CentOS7 VM providing the Database service at the backend
1 x LB VM (Load Balancer) - CentOS7 VM running HAProxy server that is placed in front of two SERVER VMs
2 x SERVER VMs - CentOS7 VMs dedicated to run only the Containers that make up the Rancher Labs software stack
3 x ETCD VMs - CentOS7 VMs dedicated to run only the Containers that make up the distributed Key/Value Datastore
3 x KUBER VMs - CentOS7 VMs dedicated to run only the Containers that that make up the Kubernetes platform
3 x COMPUTE VMs - CentOS7 VMs dedicated to run only user Containers / workloads
The configurations of each VM is as follows:
Here is my physical setup:
- I now have a powerful Intel Xeon based server with Supermicro board that which bought from MITXPC. Thanks to William Lam and Tai Ratcliff for their recommendations.
- I physically connect my server to my Macbook-Pro laptop with a crossover cable back-to-back. And I define a dedicated network (10.1.1.X/24) between the two
- I then use a second crossover cable to connect my laptop and my server's IPMI network interface for console access - for which I define a separate network (5.1.1.X/24)
- I finally build enterprise-grade Kubernetes platform on my server with Rancher stack, and there by accessing it from my laptop
[ You may also want to check out our Hands-On-Box that we are designing specifically for our targeted audience ]
My MITXPC details with Web Client:
List of 17 virtual machines that make up Rancher-Kubernetes Hands-on lab.
In addition to the above 17 VMs, there is yet another virtual machine - REGISTRY that I tried using it as a private Registry with VMWare's Project Harbor. Since I didn't get a chance to play with it much this time, I am excluding it in this write-up. I am hoping to revisit private registry at a later time but that is something definitely worth considering as it is enterprise-grade.
So let's get started:
I will start with the pre-requisites first, followed by installing Rancher Labs software components on multiple nodes, followed by configuring them and then finally getting the distributed Kubernetes environment up and running.
I'll go in detail on the Pre-requisites as that is where I believe most of us go in circles, get frustrated and eventually give up. My sincere hope is that the documentations that I have posted here come handy as references in one way or the other, for your other projects as well.
Section #1: Pre-requisites
Section #2: Installing Rancher Labs software stack
Section #3: Configuring Rancher Labs stack
Section #4: Configuring Kubernetes environment
Section #5: Verifying distributed Kubernetes platform
Section #1: Pre-requisites
a) Configuration of DNS server
b) Configuration of OPENFILER to provide shared storage (NFS) to ETCD VMs for backup
c) Configuration of ETCD-01, ETCD-02 and ETCD-03 in mounting shared storage for backup
d) Installation and configuration of MYSQL server
e) Installation and configuration of HAProxy on LB (Load Balancer) VM
Section #2: Installing Rancher Labs software stack
Section #3: Configuring Rancher Labs stack
a) Access LB (Load Balancer) stats page
b) Access Racher UI through LB (Load Balancer), served by SERVER-01 and SERVER-02
Section #4: Configuring distributed Kubernetes
a) Defining Kubernetes Template
b) Defining Kubernetes Environment
d) Adding Hosts and installing Client components on ETCD VMs (Data Plane)
e) Adding Hosts and installing Client components on KUBER VMs (Orchestration Plane)
f) Adding Hosts and installing Client components on COMPUTE VMs (Compute Plane)
Section #5: Verifying distributed Kubernetes platform
a) Verifying Kubernetes Dashboard
b) Verifying Infrastructure stacks
e) Verifying ETCD VMs (Key/Value distributed datastore) backup
Lab exercises:
Now that you have a fully built distributed Kubernetes platform at your disposal, the next logical step is to take some time in doing hands-on lab exercises for which, I would recommend the following:
a) Watch this webex recording by Rancher Labs
b) Download the same YAML files shown in webex recording and practice the same exercises on your own lab. Redo the lab exercises, start tinkering the YAML files, till you get clarity and confidence
c) Join their community. I would certainly recommend signing up for their online meetups which are very informal and informative
Conclusion:
For me, Rancher Labs feels like an IKEA thingie ( No offense 🙂 ) You don't need to be an expert but you will eventually figure things out and get it fully functional. If you are venturing into building distributed Kubernetes platform from scratch, Rancher Labs is definitely worth giving it a shot.
I sincerely hope you find this article informative and educational. Would love to hear your thoughts, suggestions and advice as I am planning to write more articles on this topic.
Wishing you all the very best on your plans with Kubernetes and Rancher Labs.
< ananda at fieldday dot io >
References:
Getting started with Kubernetes by Rancher Labs:
https://www.youtube.com/watch?v=Eo3-zKsZ4Gk
Rancher best practice guide:
https://chrisurwin.github.io/bpg/
Openfiler NFS share:
https://www.youtube.com/watch?v=h0ahq12pVHo
NFS mount on CentOS7:
https://www.howtoforge.com/nfs-server-and-client-on-centos-7
MySQL:
https://www.howtoforge.com/tutorial/centos-lamp-server-apache-mysql-php/
Building HAProxy on CentOS7:
https://www.upcloud.com/support/haproxy-load-balancer-centos/
