Yes you are right. When you talk to a person from Gilgit-Baltistian they will tell you they are not Kashmiris and they have their own separate culture, language, and history like Punjab, Sindh, Balochistan, and Khyber Pakhtunkhwa each have their own separate culture, language, and history. The British and the hindu hari singh annexed Gilgit-Baltistan to their Kashmir Kingdom.
And the people of Gilgit-Baltistan are patriotic Pakistanis, most want a province status for their homeland like Punjab, Sindh, Balochistan, and Khyber Pakhtunkhwa. The only real Kashmiri Sarzameen that Pakistan has in its possession is the small strip of land called Azad Kashmir, the rest of Kashmir is occupied by india.