I'm sorry but you have no right to say they are not kashmiris. My family has not been in Kashmir for probably a century yet still I am proud to call myself Kashmiri. Regarding environment and influence, then according to your logic, the Palestinians who were forced out of their homes in 1948 are not allowed to return either.
And I know AJK and Kashmir valley are two separate areas. I think the people of these two areas have never really been that close anyway, I mean there are a lot of AJK people here in UK, but I feel no belonging to them. They are actually very paindu, even when living in Uk, see, their environment hasn't really change them.