Skip to main content

Storing Request Statistics Using Redis

If we build a backend service, we probably need to keep any statistics related to our service such as number of requests, average time for processing, number of errors, etc. We may also need to have statistic records in several time precisions for example hourly and daily. The simplest solution, we can just store any information into a table in our database, then calculate the summary when needed or by a request. But, it will cost our system storage and computing resources.

Unlike relational databases or NoSQL that only have CRUD operations in general, Redis operations are related to the type of data that is stored. For example, a list will have push or pop operations, a sorted set will have a ranking, incrementing, or union operation, and so on. For instance for storing request statistics in Redis, we will store statistics of the response time of a backend application. The metrics are minimum time, maximum time, number of responses, and total response time. The last two metrics can be used for calculating the average time to respond. We will also store the statistics for hourly and daily information.

We need several records in sorted sets to store the following information.

  • List of starting times of all periods
  • Metrics of each period
  • Temporary data for aggregating minimum and maximum values

A list of starting times is used to help us to list all metrics records that we have. The record of metrics of each period will be named with starting time of the period.

Our backend application will call a function every time it handles a request with a response time value in milliseconds as the first parameter of the function.

All metrics in a period will be aggregated in a single record directly. Every time the function is called, it will calculate the period based on the calling time and the precision value.

The processes are as follows.

  1. Calculate starting time of current period
  2. Store the response time value in temporary records as both minimum and maximum values of response time
  3. Aggregate data in temporary records with matrics record of the current period
  4. Increment number of responses and total response time in the matrics record of the current period

For this example, we utilize Node.js with the redis module. We also need uuid for generating a random key for temporary records. The implementation is as follows.

function captureStats(responseTime) {
  const now = Date.now();
  const hourly = 3600 * 1000;
  const daily = 24 * 3600 * 1000;
  
  const tempKeyMin = `stats:temp:${uuidv1()}`;
  const tempKeyMax = `stats:temp:${uuidv1()}`;

  // get starting time of the period
  const stimeHourly = Math.floor(now / hourly) * hourly;
  const stimeDaily = Math.floor(now / daily) * daily;
  
  // to store starting time of each period
  const timesHourlyKey = `stats:times:${hourly}`;
  const timesDailyKey = `stats:times:${daily}`;
  
  // to store metrics of each period
  const statsHourlyKey = `stats:${hourly}:${stimeHourly}`; 
  const statsDailyKey = `stats:${daily}:${stimeDaily}`; 

  redisClient.multi()
    .zadd(timesHourlyKey, now, stimeHourly)
    .zadd(timesDailyKey, now, stimeDaily)
    .zadd(tempKeyMin, responseTime, 'min')
    .zadd(tempKeyMax, responseTime, 'max')
    .zunionstore(statsHourlyKey, 2, statsHourlyKey, tempKeyMin, 'aggregate', 'min')
    .zunionstore(statsHourlyKey, 2, statsHourlyKey, tempKeyMax, 'aggregate', 'max')
    .zunionstore(statsDailyKey, 2, statsDailyKey, tempKeyMin, 'aggregate', 'min')
    .zunionstore(statsDailyKey, 2, statsDailyKey, tempKeyMax, 'aggregate', 'max')
    .zincrby(statsHourlyKey, 1, 'count')
    .zincrby(statsDailyKey, 1, 'count')
    .zincrby(statsHourlyKey, responseTime, 'sum')
    .zincrby(statsDailyKey, responseTime, 'sum')
    .expire(tempKeyMin, 10)
    .expire(tempKeyMax, 10)
    .exec((err, results) => {
      // check
    });

}

Each metric record will have the following parameters.

  • min. The value will be updated by aggregating the latest value in the temporary record with the current value in the metric record using zunionstore and the aggregate min method.
  • max. The value will be updated by aggregating the latest value in the temporary record with the current value in the metric record using zunionstore and the aggregate max method.
  • count. The value will be incremented by one in each function call.
  • sum. The value will be incremented by the response time value in each function call.

Comments

Popular posts from this blog

Rangkaian Sensor Infrared dengan Photo Dioda

Keunggulan photodioda dibandingkan LDR adalah photodioda lebih tidak rentan terhadap noise karena hanya menerima sinar infrared, sedangkan LDR menerima seluruh cahaya yang ada termasuk infrared. Rangkaian yang akan kita gunakan adalah seperti gambar di bawah ini. Pada saat intensitas Infrared yang diterima Photodiode besar maka tahanan Photodiode menjadi kecil, sedangkan jika intensitas Infrared yang diterima Photodiode kecil maka tahanan yang dimiliki photodiode besar. Jika  tahanan photodiode kecil  maka tegangan  V- akan kecil . Misal tahanan photodiode mengecil menjadi 10kOhm. Maka dengan teorema pembagi tegangan: V- = Rrx/(Rrx + R2) x Vcc V- = 10 / (10+10) x Vcc V- = (1/2) x 5 Volt V- = 2.5 Volt Sedangkan jika  tahanan photodiode besar  maka tegangan  V- akan besar  (mendekati nilai Vcc). Misal tahanan photodiode menjadi 150kOhm. Maka dengan teorema pembagi tegangan: V- = Rrx/(Rrx + R2) x Vcc V- = 150 / (150+10) x Vcc V- = (150/160) x 5

Kerusakan pada Motherboard

1. Sering terjadi hang memory tidak cocok --- ganti memory ada virus di harddisk --- scan harddisk over clock --- seting kembali clock prosesor ada bad sector di harddisk --- partisi harddisk dengan benar 2. Pembacaan data menjadi lambat memori tidak cukup --- tambah memori harddisk penuh atau ada virus --- kurangi isi harddisk, scan harddisk, atau ganti hardisk 3. CMOS failure baterai habis --- ganti baterai CMOS seting BIOS berubah --- seting kembali BIOS 4. Tidak bisa booting cache memory rusak --- disable eksternal cache memory di BIOS memori tidak cocok --- ganti memori boot sector pada harddisk rusak --- masukkan operating system baru ada bad sector pada trek awal harddisk --- partisi harddisk 5. Suara bip panjang berkali-kali memori rusak --- periksa kedudukan memori memori tidak cocok --- ganti memori memori tidak masuk slot dengan sempurna --- periksa kembali kedudukan memori 6. Suara bip bagus tetapi tidak ada tampilan / bip dua kali VGA card

Raspberry Pi Bluetooth Connection

Raspberry Pi 3 provides a built-in Bluetooth module. The latest Raspbian has been bundled with tools for enabling Bluetooth connection. The Bluetooth icon will be shown up on the top right corner of the desktop. It's a tool to discover available Bluetooth devices and connect Pi with Bluetooth devices. It is easy to connect any Bluetooth-enabled electronic device with Pi. But, sometimes Pi will fail to connect, especially for Bluetooth device that has no standardized services. From a terminal, we can use the  bluetoothctl tool to scan and connect with a Bluetooth device. You should make sure that the BlueZ protocol stack has been installed by running $ apt-get install bluez Run bluetoothctl to enter the tool command window Turn the power on by running power on (Optional) You can set AutoEnable=true in /etc/bluetooth/main.conf if you want to make the Bluetooth auto power-on after reboot. Run devices to see which devices have been paired Run scan on if your desired d

How To Use Git in Netbeans

Git is a popular version control application nowadays. Recently I have created a note about its differences with SVN and how to use it in Eclipse . There are many Git client tools. But I just want to show how to use Netbeans built-in Git tools. It makes the development process easier because it has been integrated with the IDE. Create Remote Git Repository We need a remote Git repository so everyone can store or receive any revision or updated files through the networks. We can set up our own Git server or use a public Git server like Github . In this note, I use Github. 1. Create an account in Github and create an empty Git repository Create an empty public repository in Github 2. Get the remote repository link Your Github Repository URL Create a New Project in Netbeans and Create Local Git Repository After we have a remote Git repository, we can create a project stored in the remote repository. We also need to create a local repository before we can push

I Love Books

I like reading books since I was a kid. Thanks to my parents that let me bought any books and embraced me to read. Bookstore becomes one of my favorite places. I like to learn any topics, not only about science or IT stuffs. For me, reading a book isn't always intended to have possession in the topic, but it's for my personal pleasure and building an insight of something. some of them There're many things around us that we don't understand really well. If we try to figure it out by ourselves, we will waste most of our precious time. So, it's better to learn from the expert who has studied it well and then considering it with our common sense and logic. I grow up, many affairs had occurred, and shit happens. I realize that I have forgotten a lot of things. I likely tend to be aware of fewer things. So let me open a cupboard full of books which I've collected since a long time ago.