Blog

I use Elasticsearch Logstash Kibana (ELK) stack for visualizing lot of things like Nginx logs, HaProxy logs etc. I also ingest application logs into the ELK. One thing I have been looking to do is visualization of errors happening into the system. Surely, Nginx and Haproxy logs can give you visualization of 500 status codes but there are many cases where the status code is 2xx but the internal error in the system is logged....

We use ELK stack heavily in our production systems for log aggregation and monitoring. Our daily log size generated is aroudn 100GB. Since, we do not intend to keep the log files in ELK for more than a month, it becomes important that we delete those indices to free up disk space. Fortunately, Logstash creates a new index every day by default. Thus, we can ask Curator to simply delete the indices x days old...

Recently, I have been working on Angular2 quite a lot. We had a requirement to come up with a reusable confirmation dialog component. Below is the quick implementation I came up with. I will soon wrap it into a npm module. Any feedbacks? confirmation-dialog.component.ts import { Component, OnInit } from '@angular/core'; import { MdDialogRef } from '@angular/material'; @Component({ selector: 'app-confirmation-dialog', templateUrl: './confirmation-dialog.component.html', styleUrls: ['./confirmation-dialog.component.scss'] }) export class ConfirmationDialogComponent implements OnInit { public title: string; public...

Tcpdump and Netcat are one of the most useful utilities for Linux network debugging. Some of the examples where I have found tcpdump to be extremely useful are given below. In the next post, I will cover netcat. Look for traffic based on IP address tcpdump host 1.2.3.4 Capture based on protocol tcpdump udp Capture based on interface tcpdump -i eth1 Capture filter based on certain port tcpdump port 80 Capture filter based on source...

Recently, I was looking a way to do a git clone in bulk. I came across this nifty little script to do the same #!/bin/bash # The script clones all repositories of an GitHub organization. # Author: Jens-Andre Koch # the github organization to fetch all repositories for GITHUB_ORGANIZATION=$1 # the git clone cmd used for cloning each repository # the parameter recursive is used to clone submodules, too. GIT_CLONE_CMD="git clone " # fetch repository...

Focus on performance and scalablity is one of my primary personal and professional goal when working with tech products. The server can be any Linux based server such as CentOS or Debian derivative such as Ubuntu. In this post, I will outline my learnings on scaling the Linux server. Scaling here implies many things such as Being able to open many files at once. Here files can be generally applied to concept such as open...

If you have worked with Nginx cache, one must know and understand proxy_cache_lock and proxy_cache_use_stale directives. To configure simply Nginx caching, we basically add these 2 lines in our configuration block: proxy_ignore_headers X-Accel-Expires Expires Cache-Control; proxy_cache_valid any 30s; This will ignore any of the cache related headers from upstream and keep the cache valid for 30 seconds. This means that a request will only go out to upstream every 30 seconds. But it won't be...

Recently, there is lot of focus on dividing a complex application into small microservices communicating over REST. In most of the cases, REST is implemented over HTTP protocol. All this is great, but HTTP protocol adds lot of communciation overhead. There is a Request header, response headers which adds to considerable bytes in payload. Apart from that, you need to take care of things like keepalive to eliminate TCP connection time overhead. The keepalive does...

Effective logging is crucial to any application deployed in production. For node.js, express based applications, we use Winston as our logging framework. In our scenario, we also want some crucial logs to be logged to Sentry. A simple setup of Sentry is described below. This is same as what is described in sentry docs. var sentry_enabled = false; if (process.env.NODE_ENV == 'production') { sentry_enabled = true; } if (sentry_enabled) { app.use(raven.middleware.express.requestHandler(config.sentry_dsn)); } // setup routes...

We use Couchbase as our NOSQL document store. Couchbase provides lots of software development kits (SDKs) in different programming languages, including one in Node.js, so JavaScript development integrates Couchbase into the product very easily. For example: var couchbase = require('couchbase'); var cluster = new couchbase.Cluster('couchbase://127.0.01'); var bucket = cluster.openBucket('default'); bucket.get(userId, function (err, res) { if (err) { console.log(err); //handle error } else { //access data here } }); However, there are some caveats which we...