That is correct. Per everything clearly indicated in the Quran, there is no perpgative to worry about the Kaaba besides it’s cleanliness.Based on that argument, if God forbid Holly Kabah is under attack, and Pakistan is busy fighting TTP, Turkey is fighting Kurds in Syria then we all should mind our own business and let Allah take care of it? Is that correct?
It has been destroyed by Allah before with floods and protected by him, not our responsibility.
If tomorrow a nuclear weapon lands right on the Kaaba, Islam will not stop. People will still need to face the Kaaba. The Kaaba is a focal point for us to align our worship at, not a stone monumnet for us to worship.