So I was recently gifted some Mellanox 40gig network cards that I installed in my NAS and my desktop and connected with AOC fiber. I gave them both static IP addresses on their own dedicated subnet that's not used anywhere else in my network. I was able to run iperf3 between both computers, and that worked exactly as expected.
At that point, I edited /etc/fstab to update the IP addresses for my mounted network shares. I ran # mount -a successfully and thought all was well.
The problem is, my computer defaults to my one gig lan connection for some reason, despite the entries in fstab using a completely different subnet.
The only way I've found to force it to work properly is to disable my LAN connection, then remount the network shares, then reenable the LAN port.
On one occasion I noticed that a file I was duplicating on my NAS was being downloaded via my LAN to my computer to duplicate, then being uploaded back to the NAS via the fiber connection.
Does anyone have any clue why this may be happening or how to fix it more permanently?
The NAS is Debian, my desktop is Manjaro.
I did run a daemon-reload. I'm directly referencing the IP address in my fstab. On my desktop I have DNS turned off for the Mellanox interface.
I'll have to look into the nfs exports file. I'm using OMV, and I just have my exports wide open. But, I guess I could solve this by limiting the connection to the mounts by client IP.
There might also be some magic/weirdness with IP routing in the kernel. Have a look at net.ipv4.ip_forward system variable.