#Filebeats kubernetes series#

We can also set a flag to allow volume expansion on the fly. This can be done by specifying the volume type when creating the storage class. It is important to format the persistent volume before attaching it to the pod. The headless service in the case of data nodes provides stable network identities to the nodes and also helps with the data transfer among them. We will use the following manifest to deploy Stateful Set and Headless Service for Data Nodes: The headless service named elasticsearch-discovery is set by default as an env variable in the docker image and is used for discovery among the nodes. It can be seen above that the es-master pod named es-master-594b58b86c-bj7g7 was elected as the leader and the other 2 pods were added to the cluster. ‍ root$ kubectl -n elasticsearch logs -f po/es-master-594b58b86c-9jkj2 | grep ClusterApplierService When following the logs of the master-nodes, you will also see when new data and client nodes are added.

This is when the master-node pods choose which one is the leader of the group. If you follow the logs of any of the master-node pods, you will witness the master election among them. Image: quay.io/pires/docker-elasticsearch-kubernetes:6.2.4 PreferredDuringSchedulingIgnoredDuringExecution: ‍ Deployment and Headless Service for Master Nodesĭeploy the following manifest to create master nodes and the headless service: Let’s jump right at deploying these services to our GKE cluster. Setting correct Pod-AntiAffinity policies among similar pods in order to ensure HA if a worker node fails.In case of 3 masters we have set it as 2. Setting NUMBER_OF_MASTERS (to avoid split-brain problem) env variable for master deployment.HPA (Horizontal Pod Auto-scaler) deployed for Client Nodes to enable auto-scaling under high load.Kibana and ElasticHQ Pods are deployed as Replica Sets with Services accessible outside the Kubernetes cluster but still internal to your Subnetwork (not publicly exposed unless otherwise required).Elasticsearch Client Node Pods are deployed as a Replica Set with an internal service which will allow access to the Data Nodes for R/W requests.Elasticsearch Master Node Pods are deployed as a Replica Set with a headless service which will help in Auto-discovery.Elasticsearch Data Node Pods are deployed as a Stateful Set with a headless service to provide Stable Network Identities.The Elasticsearch setup will be extremely scalable and fault tolerant. We will be using Elasticsearch as the logging backend for this.

#Filebeats kubernetes series#

This is the first post of a 2 part series where we will set-up production grade Kubernetes logging for applications deployed in the cluster and the cluster itself.

Deployment and Headless Service for Master Nodes.