I can relate. I tried to tell my middle sister first, even before I was diagnosed, because I’m closest with her. And it explained what was up with me, my Mum, my Aunt, my grandmother to a tee.
My sister yelled at me that She couldn’t understand why I would want something to be wrong with me and I needed to stop.
Anyway, I’ve come to realize that because my brain works differently, and because at times I can solve other people’s struggles in the blink of an eye - my sister actually values the way I am and Therefore, cannot accept the label that something is “wrong” with me.
I believe it is out of love that your brother is rejecting the label. For you it comes across if a lack of understanding from your brother. However, if you can try to see it as coming from his love for you it will make you less defensive towards him. If you can see a brother, who loves his brother so much that he is unwilling to hear and accept that anyone says he’s “broken” and not your brother who is rejecting part of you, then I believe he will come around.