how do you actually decide if relocating for an offer is worth it beyond just comparing the salary numbers

Offer is meaningfully more than I make now but it means leaving a city where my entire support system is. Cost of living calculators only get you so far, nobody talks about the harder to quantify stuff. Anyone regret taking a relocation offer for reasons that had nothing to do with the job itself?