That really sounds like something that would sound superficially plausible but then turn out to not work quantitatively, once you started running numbers on how many empty houses you needed all over to have them available for people to randomly move into…
"So even at the cleric or wizard level, houses must not have... expensive, non-modularly-removable features that only a few people want, because you can't take it with you if you leave the house? Just features that everybody wants, or stuff only you want that you can take with you when you leave?"