I agree with you, the only caveat here is that the artist mentioned say that their books were illegally obtained. Which is a valid argument. I don't see c how training an ai in publicly available information is any different than a human reading/seeing said information and learning from it. Now that same human pirating a book is illegal.
The additional complexity here are laws that were written and are enforced by people that don't fully grasp this technology. If this was traditional code, then yes it could be a copywrite issue, but the models should be trained on enough data to create derivative works.
I am in a similar spot. For the better!