What you say is true. But it all depends on how one defines his field of view. At a community level, Yes, our actions can make a difference. At a National level, most of us make minor contributions. At a global level, the impact is even less and finally, beyond this level none of us or our actions matter.
Activities you mentioned keep will keep a man involved, but in the smaller truths and smaller nitty-gritties of life. But once you know that your life doesn't serve any purpose beyond the token help and support to a few similar humans or some animals, it does indeed feel like we've inflated ourselves and our roles and actions to find some measure of meaning to our existence.
I think the reason why these yogis or gurus renounce materialism and prefer seclusion is because only then are they truly in some control of their lives. It gives them the freedom to stay outside the great competition and jostling that the society is and live life on their terms. Staying in a society, it's often the case that we mould our lives in many ways so as to feel accepted.
I ask you this, will any of our actions, including helping the poor and needy, sheltering rare plants and animals, supporting a family etc really matter to this solar system, let alone the planet? These are actions that make us feel good about ourselves and find a meaning, however insignificant and contrite, to continue and prolong our lives. It's not unlike a placebo.
It's true that this concept was thoroughly exploited by the upper castes in perpetuating the subjugation of Dalits. But fact remains that there have been plenty of cases even in the Western and other civilizations that people have said that they could remember things from the past.
In short: the phenomenon was global, it's authenticity unproven, but it's exploitation for ulterior ends very real.