You can read verses in Gita legitimising Caste system.
As @Nilgiri has aptly said, there is nothing wrong in caste system but the wrong lies in exploitation of it.
A society is anyhow divided and its best to have segregation based on your competency rather than on your skin/color/race.
What we see today was never legitimised and thus we face no religious objection to call for its abolition.
Actually its the reservation system which has kept it alive and we know for what !! The same oppressed are lining up to get certified shudra rather than getting the tag removed.