House Builder does NOT 'come' with any SketchUp installation.
You have to deliberately install it yourself.
There are only a small handful of su_ extensions shipped with SketchUp's installer...
If you get and install the SketchUcation PluginStore Toolset [now v3 ExtensionStore] from the PluginStore then that has Managers.
This allows you to disabled selected Extensions AND Plugins.
Disable [red-button] House Builder and restart SketchUp.
You can enable it temporarily [orange-button] when you want to use it.
Or enable it fully [green-button], as you desire...