Not only that is a stupid argument, it is a dangerous one.
You have effectively opened the door to other justifications. How about weakness, for one? If you are weak, you should be lord over by foreigners. It was used in the past, now you tacitly said it can be used again in the present.
It begs the question of why there are illegal immigrants in the first place. Did these people entered, legally or illegally, the modern day democratic Britain because of what Imperial Britain did a couple hundred years ago? I doubt it and I doubt they even cared for that far back in history. People do not abandon their lands because of history but because of wealth, or at least the potential for wealth. Not the palatial and butler-ed kind of wealth, but the kind that would make even an incremental increase over their current living conditions.
There is a great difference between being an immigrant versus being a temporary worker. An immigrant is one who intends to take roots in the country. He does it because he feels this country have far better opportunities than the one he is currently living under, or the one he left behind. Roots that WILL involve deeper community relationships, usually deeply emotional ones such as marriage and politics. A temporary worker do not care for these emotional relationships but only one -- money.
This has nothing to do with historical wrongs. If you do not like the current Britain -- leave.
The UK is a country that plays a double-game when it comes to immigration.
It actually allows any and all to come into their country, both EU and non-EU but then pretends itself the innocent victim of this immigration.
This scheme is just a smoke-screen to fool the gullible public that the Tories actually care about curbing illegal immigrants, when all they care about is protecting the interests of
their rich friends who need the immigrants to provide a never-ending source of cheap labour.