They can be. I mean, I don't think xe is inherently less legitimate as a word. But I do think it's ugly and unnecessary, as are neopronouns in general. I have similarly pedantic opinions on things like 'irregardless'.

If someone asked me to use 'xe' I would out of courtesy, but I would much prefer we all come together and agree that the singular they is the best choice for the job.