Don't know how to help you directly, but you might consider using a different codec. Some codecs will be more lossy in their compression than others.
There is a freeware screen capture program called CamStudio that has a lossless codec available for download. Once installed, you may be able to choose the lossless codec for your video export, giving you a file that is gigantic in size and hard to work with, but which may then be transcoded successfully to another format with better results.
I am guessing, but it's worth a shot.