For me there’s stuff like Samurai Jack which isnt voiced by a Japanese person, and that voice to me never left me feeling it would have been better voiced by someone else. I just heard and saw the animated character of Jack.
Same goes for many younger male characters voiced by females that I never found myself wishing it was voiced by a male like voices of Bart Simpson and Timmy Turner.
To me it sounds like you are more speaking of native language voices? Like if a character speaks English you want it to be voiced by someone who speaks like a native speaker, or Japanese you want someone who speaks like a native speaker and so on. That I do agree with. You want someone who can deliver the performance of native speaker regardless of race. So you wouldn’t want like Mark Hamill doing a Japanese dub of the Joker.
Like an example I think of is Vinland Saga voiced by Japanese for the Japanese language. I don’t think the performance would be better replacing them with white Japanese speakers to deliver the performance in Japanese if their performance isn’t as good. And for English I’d apply the same where the delivery is more important like how it wasn’t important that House for example was a British actor delivering the performance of an American doctor.
I like animation to be a place where anyone can voice anyone if their performance is good and none of their other traits matter. But only voice performance. Like how orchestras may hire based on auditions with the performance being done behind a curtain.
I’m not deciding for them. I’m explaining why I don’t find physical or background traits to be something I find important for voice acting.