Going to camera previous doesn't guarantee that other settings from your scene (like visible layers) will be restored. My solution is to select the matched photo scene and then switch to parallel projection. When I switch back to perspective, I'm back at the same angle but without the photo. I can then add a new scene and delete the old one. This method doesn't mess with any of your other settings.
You do have to make sure you start from the right place. If you're zoomed in on your photo matched scene, the photo will still show but the new scene you create will be a zoomed version. So click the scene tab before you get started just to make sure it's perfectly set-up.