Honestly most of Brian’s detractors are arguably fans who wish he wasn’t boring and wanted to see more of him. He’s the most mundane of the group (Taylor’s time with him was basically mundane shippyness) and due to being so reserved we have to piece his past together. Just a consequence of being around a show stealing cast tbh.
That makes sense.
There’s something similar going on in the MLP fandom, where Applejack keeps coming last in Mane Six popularity polls, in part because she’s often the straight mare of the group, the reasonable one (though Twilight gets her share of that role too).
And in the Homestuck fandom, I don’t see people saying John Egbert, who starts out as an audience proxy and often ends up as a straight man towards the ridiculousness of other characters, is their favorite human kid very often. (I do like him, though. He’s a relatable dork.)