Or to put it another way, the wheel zoom rate is proportional to the apparent distance from you (camera) to the object directly under the cursor (even if not selected). So the closer you are the "slower" the zoom. And that also accounts for the so-called "Hyper-Pan" or "Hyper-Zoom" when your cursor lands on empty space during repeated zooms or pans.
The best recourse after a very slow zoom or when in deep space is to "View Extents" which will ballpark you to your object. "Previous View" is only good for the last 3 or 4 positions before cycling back.
Another trick is to create a temporary scene before zooming in if you are going to be in close for multiple operations, and need to get back out to a common viewpoint multiple times.