That is not the marker for cultural influence at all. Persian wasn't the dominant language in most of Iran itself for hundreds of years, still isn't the sole language there (Much like Urdu in Pakistan). The Persians literally ruled, on and off, over most of what is today Pakistan since 550 BC.
The Achaemenid Empire (550–330 BC):
Pashto, Balochi and Darri are three major examples of Iranic languages endemic to Pakistan. Pashto, btw, is the second most spoken Iranic language after Persian and can be traced BCE. Don't think I need to state the Iranic genetic influences on Balochis, Pukhtoons, Darris, Hazaras, etc.
It's very natural, tbh. Pakistan lays smack on the crossroads of the ancient and the modern worlds. There aren't many major civilisations that haven't camped or passed through here. If you read Alexander's historians' accounts, you'll find that there was European influence here even before his conquest.
All that is not to say that we are Iranic or any other foreign group. We are natives of the land who have had a lot of external influence. Urdu is beautifully analogous to our people. It has words from Persian, Arabic, Turkish, Hindi, English, and a fair few others but is itself none of them. We are Pakistanis, we are a boiling pot, we are unlike any other. No wonder our Indian friends here want so desperately to be associated with us.