I would disagree. Ladakh was never ruled as a colony but as an addition in the kingdom. Moghuls too ruled India but they ruled it not as a colony but made India their home. Australia, the US, New Zealand are all occupied territories not ruled by natives but occupiers. The difference is that the occupiers never treated these territories as a colony but their own country. Thats the difference between Ladakh and Goa. If portuguese had decided to settle in Goa and rule as citizens of Goa, my argument then would be different perhaps.
What defined a chinese territory before the McMahon Line? Not long ago, just about less than a hundred year ago, the boundaries of a kingdom kept changing. People were unconcerned about it and the rulers too did not take it to their heart too much about losing a territory. It was more of a military triumph or loss and not a matter of ego. Geographycal surveys too are not a very old thing.
Nepal is a sovereign independent country. The people on southern border with India are almost the same people on the indian side in terms of race, language, culture, religion, caste etc. they are referred as Madhesis in Nepal. Similarly on Nepal's eastern border with India, there are Gorkhas on Indian side who speak Nepali and have more mongoloid features. All this happenned over a long period of history. Now it would be wrong for India to stake claim on madhesi people and land of Nepal and it would be wrong for Nepal to claim the Gorkhas of India.
If the nations decide to draw a boundary based on some long gone historical fact then there will be no end to that. One of the most sacred place for hindus is Mount Kailash and Lake Mansarovar where they believe their Lord Shiva to be residing. This place falls in Tibet, China and has no religious significance for chinese. Should India stake claim on Mount Kailash and Mansarovar Lake based on the mythology???
Sometimes, you say that MacMahon Line was drawn by the British and India inherited that in 1947 and at the same time you say that India invaded and occupied Arunachal Pradesh in 1951. And you replied to that invasion after 11 years.
We dont know what ethnic chinese in Tibet are made to believe by the authoritarian chinese govt. but we have about 100,000 refugees from Tibet who accuse China of occupying their territory and they demand chinese to withdraw from their land.