How code modularization can solve log analysis issues in production
Logs are the best friend for developers or people on production support. When things go wrong they are the ones which play the role of witness and gives details of crime scene and how our system failed to perform duty or died on the spot.
As a developer I would like to tell them as much detail as possible so that my life becomes easy as investigation officer of the incident.
We all know logs also take up space in system where application is running and will take bit of processing power and bandwidth to generate logs and transfer those to tools like Splunk/Kibana for further analysis.
So changing log levels blindly in system for debugging purpose can actually slow down the system and you might end up missing important logs.
Same situation happened in my project when I enabled trace logging for all the services.
We were using log back for logging configuration and splunk as centralizing logs.
As I enabled trace logging it started generated so many logs that splunk forwarder was taking some time even 30 minutes to send them to Splunk and because of our logback roll back policy so many logs started getting deleted even before reaching to Splunk for analysis.
So never ever enable debugging/trace logging for entire code base.
Here modularizing the codes actually helped me big time, In any production issues as a developer I will have an idea about areas which are definitely not problematic.
If you have those areas in different modules/classes change the logging for those modules to ERROR. So they have minimum logging.
Now areas which are suspicious only for those enable logging at TRACE/DEBUG level.
Spring boot actuator end point actually can help in changing log levels at run time, so there is no need of code changes or deploy app again.
I used postman collection runner where I can actually mention different loggers and their levels to be configured.
Here is actuator endpoint which can help in changing log levels:
URL: http://{{app_url}}/actuator/loggers/{{logger_name}}
Request body: {“configuredLevel”: “{{log_level}}”}
app_url, logger_name and log_level can be passed through CSV file in runner so that you can quickly change log levels for multiple logs.
If you have not modularized code in right way and have one big class taking responsibilities of so many things then this actually might not help.
