The main problem I see is that Google just shouldn't include AI results. And they definitely shouldn't put their unreliable LLM front and center on the results page. When you google something, you want accurate information, which the LLM might have, but only if that data was readily available to begin with. So the stuff it can help with is stuff the search would put first already.
For anything requiring critical thought or research, the LLM will often hallucinate or misrepresent. The danger is that people do not always apply critical thinking. Defaulting to showing an LLM response is extremely dangerous, and it's basically pointless.
I agree with you, and I use it that same way. But I think it should be something the user explicitly seeks out. The problem is that everyone who uses Google now unintentionally use an LLM in the exact same way they've always found human-written content. It's fundamentally different content, so shoving it into the existing interface is begging for confusion.