Event logging with fluentd: How we did it

Baran Bartu Demirci

Head of Engineering @ Volt Lines

Oct 5, 2018

l

Hello! I’m Baran, working as the Head of Engineering at Volt Lines. Since we are disrupting our way of how we make things almost daily on Engineering side here in Volt Lines, we are learning lots of stuff while we are having issues on the road. We thought that sharing our best practices will be helpful for you. I’m holding the Engineering team’s flag by writing our first blog post about what we did for our Operational Disturbance module under our Mission Control app (and Burcu, aka our Product Manager, is summarizing what we had done for the third quarter of the year here). Let’s dive into details.

Event logging with fluentd: How we did it

/

We’re a tech company that runs mostly on data. All the decisions that we make are data-driven, and for that reason, it’s super important to collect this data first. Aside from gaining flexibility in making decisions, monitoring allows you to see the potential problems before they happen and improve what has to be improved to have better output.

I’ll explain through an example to help you understand better. Recently, we needed to see whether our drivers take necessary actions during the correct time or not, so we wanted to implement a sample dashboard to see the results. The sunny side of the picture: We only had 3 days to make it work. We immediately had a quick meeting with the team and focused on a few options:

  1. Since our development stack is based on Python, the first option would be using Celery,

  2. The second option would be using a pub/sub like RabbitMq, Redis or something else and make our own implementation,

  3. Third and the best option would be using Fluentd.

Considering we had 3 days to implement, our time was counting down. And of course we picked Fluentd up.

***This is not a deeply technical document so I’m not going to explain how to install all of those stack. This is only to tell you why you must use it.

Regardless of how many servers you have under the load balancer, one fluentd instance works on each server which is localhost. Can you imagine that each server sends data to localhost? And what’s more is via TCP. Sounds good, right? On the application, just install which fluentd client library you need to, and send the data as json where you need. Fluentd collects data and buffers them first. On the fluentd configuration you can add any plugin you want: for example, we used MongoDB to store data and we added only a few lines on configuration file and BOOM! Your data will be on your database in seconds. Using that approach, you can tune your configuration according to your needs. Our configuration was adding 2 seconds buffer so that each data on buffer will be posted to MongoDB per 2 seconds as batch. Also according to some of our new features (e.g. real-time gamification on drivers’ side), we copied the configuration and added for AWS Kinesis to stream the same dataAll of those took 5 minutes to configure! After that, we were able to focus on making good screens to monitor our users properly. No effort on logging, no code implementation!

Hope this post makes your life easier so you can focus on your business instead of reinventing the wheel again. You can click this link to see what other things you can do with fluentd. Enjoy!

Bizi sosyal medyada da takip etmeyi unutmayın.

I

I

Volt Lines Akıllı Ulaşım Teknolojileri ve Taşımacılık Anonim Şirketi © 2019

Event logging with fluentd: How we did it