Here's the relevant safety guides from Stanford and MIT.
In short, if you do a lot of soldering, there are long-term occupational hazards from both lead oxides and rosin. Both guides agree that the main hazards are the fumes (workstation should have a fume extractor or suitable filter) and residue on your hands (wash hands with soap and water before eating).
I couldn't find any numbers on how much material is removed by washing, but every reference emphasized that soap and water are vitally important.
Lead-based solder is preferred for high-reliability electronics (space, nuclear, military, etc.) because it's easier to rework, easier to verify by visual inspection, and it's not vulnerable to tin whiskers.