Firefox

Type location into the "find in settings" search bar on the settings page. First result should be the location settings from the Privacy and Security section of the settings. Click the Settings button there and enable the checkbox at the bottom of that settings dialog to block all future requests asking to access your location. If you're in a Private window, or have any privacy addons that clear cookies or other site information when a tab is closed, that would cause the original issue, as location permissions are not stored in your Firefox profile but in a cookie. The checkbox I described above should prevent the location prompt regardless of your other privacy measures.