Photo by Darwin Vegher on Unsplash

TTL(Time to Live) in Redis: MYTH BUSTED for me

Redis is basically in-memory data structure store, used as a database and cache. So its not meant to store as high volume of data as traditional SQL/NOSQL databases.

If you don’t need to store data for very long period of time Redis could be a good choice and Redis has this amazing property TTL associated with every key which will deletes the key after certain time so space becomes free.

By default TTL is -1 for any Redis key which means key lives forever and this value can be changed while storing the key in DB.

My understanding was if I store keys in Redis at 10th Nov 7:25 AM with TTL of 24 hours those keys will be deleted at 11th Nov 7:25AM and that space becomes free.

How TTL is implemented in Redis is little different and this is what I figured out looking at Redis code.

If key reaches TTL limit it becomes unaccessible but there are chances that it might still be occupying space in memory.

Redis expires keys in 2 ways:

Active key expiration: Redis runs periodic jobs in background to scan sample of keys and delete them if it has already passed TTL.

Passive key expiration: A key has passed TTL and when some client tries to access it, key is deleted and client cannot access the key.

Key expiration only takes place in master and when master node deletes the key it send DEL command to slaves nodes to delete the keys.

Passive key expiration seems very straightforward but active key expiration is little complex because Redis having long running jobs to expire keys might impact client facing operations.

In active key expiration sample size(20 for redis 6.0) is defined and in every cycle these keys are randomly picked for scanning and if they found to be expired they are deleted. Redis also has stale key threshold defined(10% for redis 6.0) and if number of deleted keys in a cycle exceeds the cycle is repeated again.

Active key expiration also gets executed in 2 modes: Fast and slow.

Fast Cycle: Two scenarios where fast cycle is triggered,

  • Redis is persisting data to disk,
  • When Client is trying to persist data and there is less memory available

Fast cycle will run for only for certain time duration and will not be repeated after that( 1000 micro sec for redis 6.0) duration. This cycle will also not run if slow cycle is already going on.

When fast cycle starts it will pick sample keys and delete expired keys, if expired keys are more than threshold defined, cycle will be repeated only if timeout for cycle has not occurred.

Slow Cycle: This is a cron job which runs 10 times in every second. It will only run on master nodes. Sample keys and stale keys threshold is applicable for this cycle as well. In case of stand alone redis where you can define multiple DBs, it will not scan all DBs in single cycle and will only scan defined number of DBs(16 for Redis 6.0 ) in one cycle and next cycle will make sure that next set of DBs are scanned. In Redis cluster you cannot define multiple DBs but same thing is applicable for multiple master nodes. Unlike fast cycle this does not have timeout defined but it ends in 2 ways:

If number of expired keys in a scan is below threshold in single loop or

Maximum CPU usage in a cycle starts going beyond certain threshold(25% for Redis 6.0)

After knowing this details on Redis key expiration I know if you have been storing data for short period of time in Redis with appropriate TTL, Redis has built good mechanism to handle clean up expired keys.

I found below article where Twitter runs into issues when Redis keys was not expired and they have to built mechanism to expire them explicitly.

https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/improving-key-expiration-in-redis

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store