I think the easiest thing "plot-wise" is to endow the magical object itself with a sort of sentience. Like not as much as the Sorting Hat in HP, but to the tune of that.

I strongly feel Vision definitely is sentient, as anyone who's seen TNG would know the arguments made for Data, and they very much apply here. If we assume the first, and there is something in the hammer that judges people, then yeah, it's hard to explain how some can move the hammer a bit, but not gain the power, despite the wording of the spell. Cap almost moves the hammer, even making tor raise an eyebrow slightly. So perhaps there's "degrees of worthiness". And Vision is pretty high up there, but not "have all the power" high. Or perhaps the power just didn't manifest at all because Vision didn't even try to "tap into" it. Who knows, comics can explain it however they feel like.

But I would not agree that Vision is "an elevator".

"Prove to the court I am sentient." - Captain Picard (from "ST:TNG The Measure of a Man")

These are fun to talk about but yeah, it does boil down to "take your pick." Still, doesn't mean we can't have fun theorising.