I am a totally book lover.They have been my guide for so many years. I love reading leadership books, management books, biographies .
One thing which I know very well that reading books require good time which most of us struggle to find in life.
When I started my motherhood I actually had to give break to book reading to great extent and was itching to go back to it. …
Every spring boot project provides way to set properties values through application.properties or application.yml file which are bundled in code. But there are could be requirement to change those properties values while starting app. For example,
Spring boot provides ways to override default properties.
Having different profile
We can have properties defined for every environment. Defining properties files for every environment helps to do that.
So if we want to have properties defined for test env, we can define application-test.yml.
Starting app with test profile will make sure…
Logs are the best friend for developers or people on production support. When things go wrong they are the ones which play the role of witness and gives details of crime scene and how our system failed to perform duty or died on the spot.
As a developer I would like to tell them as much detail as possible so that my life becomes easy as investigation officer of the incident.
We all know logs also take up space in system where application is running and will take bit of processing power and bandwidth to generate logs and transfer those…
In any application logs are first tool to analyse production issues and are best friend for the person in production support.
In today’s world logs are helpful in data analytics and figuring out usage patterns as well hence logging has become business requirement as well and not just functional.
Having logs statements while developing functionality can help in deciding right levels of logs so should be baked in from start in codebase.
Its difficult to decide right balance in logging because as a support person I would like to know every small detail possible to debug the issues but too…
Thought of being working mom used to scare me so much all the time and I postponed being in that position for long. Finally the day arrived on 19th Aug 2019 when I became a mom but not working.
I took good 10 months break and used it to bond with my little one. We got to understand each other well.
I believe having financially stability before we started another journey in life helped us so much because I did not need to rush to end my maternity leave and extended it as much as I wanted.
Getting ready to…
In a distributed publish-subscribe messaging system, services are running in distributed fashion which can fail for different reasons. This failure can lead to incorrect processing and results and duplication as well. In the case of Kafka, following could be the reason which can lead to failures:
1. Broker can fail just before sending acknowledgement to producer.
2. Producer instance might go down before receiving acknowledgment.
3. Consumer fails before sending acknowledgment.
Depending on the action the producer takes to handle such a failure, you can get different semantics:
At least once semantics: Producer will make sure that message is delivered…
In this blog I am referring component level integration tests. In testing framework integration tests is something which is above unit testing and can reduce huge load on E2E testing.
Component level automated integration tests can be very beneficial to give early feedback.
My last 3 projects have been using integration tests extensively to find issues early with every code changes as it runs with every build pipeline.
Integration tests will be self sufficient to bring up any external dependencies like DB, Kafka broker etc and at the end of test case/test suite all the dependencies should be down.
In one of my project Avro schemas was very complex and they will run with 50 or more fields. Lots of schemas will share the fields and less than 10 fields will be unique for a individual schemas.
It was quite exhausting to maintain different schemas when one or more fields will change.
That’s where I figured out that you can divide the schemas into multiple and make them reusable across multiple schemas.
How to define multiple Avro schema
I have defined a very short schema to show the syntax where one complex type uses another one.
Here is Author…
Every kafka data record will have two main parts: Key and value
Key is optional in any data record and if not provided will be considered as null by Kafka.
Before we go further on crucial role key plays in Kafka lets understand few basic concepts on how Kafka stores data published by publisher and how it gets consumed by consumer group.
Data records are organized and…
Apache Avro is a language-neutral data serialization system. It was developed by Doug Cutting, the father of Hadoop. Avro has a schema-based system. A language-independent schema is associated with its read and write operations. Avro serializes the data which has a built-in schema. Avro serializes the data into a compact binary format, which can be deserialized by any application.
Avro is used to transfer data over network or for data persistent.
Avro will convert the data in binary format before transferring hence it reduces the payload size to great extent.
Avro is also used in Remote Procedure Calls (RPCs). During…