It may be your mouse. I have had this, too. When you double click with the middle mouse button, it is actually a sort of mouse shortcut to quickly zoom into (center the view at) what you are double clicking. Now when my mouse got broken, it sometimes double clicked automatically which drove me crazy. If you double click into empty space (even "seemingly" near your component), you may be taken far away.
Check if another mouse does the same.