It points to the fact that certain sports require wealth to become expert at. And no one denies that Black people on an average are less wealthy than whites. There are enough blacks in athletics and basketball and boxing
Similarly no one denies that the average Brahmin is better off than the average dalit. The OP is trying to point to a systematic discrimination through the example of a Brahmin only restaurant, which actually in the context of eating is rather puerile, because it is not a "Only Brahmin will eat here" restaurant. OP is also deliberately trying to be oblivious to the fact that cuisine in India is often characterised through caste - Nati (Gowda / Reddy) cuisine, Maratha cuisine, Brahmin cuisine, Mudaliar cuisine and so on. These cuisines are more or less in the hands of their respective castes till today. They are not meant to depict exclusion, but expertise.