I found somewhat of a solution to this, at least what helps for me. Use only rectangular lights, STORE TO IRRADIANCE MAP checked for sure, render with irradiance map as primary engine and use as little omnis and other lights as possible because they cant be stored to irradiance map (no option for that). And output file as .exr vrimage.