I think an important aspect of Nintendo losing their best sellers during the early 2000s was the expensive choice to keep using cartridges on the N64. Then they failed with the 64DD to fix the problem of insufficient storage space available on the cartridges for the games other developers wanted to make.

The prime example is Square moving the Final Fantasy series to PlayStation starting with FF7. They could not have achieved the scale and 3D animations of FF7 on the N64. They needed the cheap and abundant storage space of CDs to make FF7.

If I was a single-console gamer who was more satisfied with the games available on PlayStation than on N64, then for the next generation I likely would choose a PS2 over whenever the N64's successor would be (as Nintendo has typically released last within a console generation while Sony usually releases early). And then at that point, I'm not buying Nintendo games at all, so I'm not contributing to their sales numbers. Xbox also rose during those years too.

Nintendo dug their own grave with respect to being the market leader in large part because they did not value their third party partners as much as they should have, and by the time the N64 was out, they had chased away a lot of important game developers. Those developers being absent for the GameCube generation hurt them. They somewhat repaired things during the Wii era, but I don't think their relationships with 3rd party developers and publishers were close to the same until the Switch.