I view frugal living as essentially hedonistic: I don't deny myself things. Rather, I spend less on the things I don't care about in order to be able to spend more on the things that matter to me. Anybody who does this, though, sooner or later (as soon as friends, relatives, coworkers, or neighbors notice some of that spending on things that matter more) is going to hear, "You're so lucky to be able to afford that!"
In my case, it's been leaving the regular job behind to write full time that's draw the comments about how lucky I am. Our occasional vacations to the islands have drawn comments on our great luck as well.
The first few times people suggested that my being able to write full time was "lucky," I was speechless. I wanted to explain that it's all about trade-offs. I wanted to talk about how, contrary to how people seem to view it, frugal living gives you more choices, because you're not tied to a high-earning job.
I didn't, though, because I learned long ago that most people don't want to hear it. Most people don't understand--and the ones who do often react as if I'm criticizing their lifestyle. (And maybe I am, although that's not my intention.)
Somewhere along the way I came up with a response that I'm comfortable with. I say, "It's not all luck." That way, if I run in to the rare person who does want to hear, all they have to do is ask. And everyone else is able to nod and pretend that I've partially agreed with them.
After all, they are partially right. I have been very lucky.