When do you actually say 'we'd add Redis here' in a design round?

I keep adding caching to every diagram because it sounds right. Then the interviewer asks what gets invalidated and I fall apart.

What's your rule of thumb for when caching is the real answer and when it's just a reflex? I'm especially unsure how to think about read-heavy vs write-heavy paths.