In colloquial English, "oh brother" is just an exclamation, like "oh God" or "oh boy". It is not to address another person. So, he wasn't calling you his brother. Just like saying "oh God" isn't calling someone God.
That is a natural habit of people imbued in collectivism but quite alien to those who embrace individualism. It feels stronger and more convincing when you hide yourself behind a big group of people. That is why sweeping statements are fairly common among mainland Chinese. And if you happen to be a Chinese and don't agree with them, they kick you out of the group of Chinese.