Tag: logging
9 Sep 2016 in Kubernetes, logging, Sumo Logic, Logentries

Off-Cluster Kubernetes Logging With Sumo Logic and Logentries

One of the best parts of my job as a solutions architect for Deis is working with an amazing array of talented engineers at companies solving truly interesting problems.

I was recently working with a company on the forefront of wearable fitness trackers. Their modest but world-class engineering team had reached the outer limits of what could be done with Ansible-based Docker deployments in AWS.

While everything worked, there were areas of API entanglement, and a lack of orchestration that created duplicated effort and an inefficient use of EC2 resources. The company is clearly on a rocketship growth trajectory, so scaling and efficient systems management are forefront on everyone's mind.

Fortunately, they also recognized that the time to pivot to more efficient and scalable architecture is while they're still in an early growth phase.

Kubernetes provides the perfect fit for their use-case because it allows a more atomic service distribution, scaling, and painless service discovery. Also, when the infrastructure below the cluster is configured with autoscaling, rapid growth should be no problem.

In this blog post, I'll take a look at one aspect of the work I did with them: how I got logs shipped off-cluster to Sumo Logic. I will also draw a link to some work I did for another company to send logs to Logentries.

Read More
30 Aug 2016 in Kubernetes, logging, Elasticsearch, Kibana

Kubernetes Logging With Elasticsearch and Kibana

Developers, system administrators, and other stakeholders need to access system logs on a daily (sometimes even hourly) basis.

Logs from a couple of servers are easy to generate and handle. But, imagine a Kubernetes cluster with several pods running. Handling and storing logs in itself becomes a huge task.

How does the system administrator collect, manage, and query the logs of the system pods? How does a user query the logs of their application, which is composed of many pods, all of which may be restarted or automatically created by Kubernetes?

Thankfully, Kubernetes supports cluster level logging.

Cluster level logging allows collecting logs for all the pods in the cluster, at a centralized location, with options to search and view the logs.

There are two ways you can use the cluster level logging: either via Google Cloud Logging or Elasticsearch and Kibana.

It is important to note here that the default logging components may vary based on the Kubernetes distribution you are using. For example, if you are running Kubernetes on a Google Container Engine (GKE) cluster, you'll have Google Cloud Logging. But if you're running your Kubernetes cluster on AWS EC2, you'll have Elasticsearch and Kibana available by default.

In this post we'll focus on Kubernetes on an AWS EC2 instance. We'll learn how to set up a Kubernetes cluster on an AWS EC2 instance and ingest logs into Elasticsearch and view them using Kibana.

But first, let's make some introductions.

Read More