My suggestion is to not use displacement on brick. You really want to reserve displacement for things that are
close to camera have a lot of 3D relief and detail (stonework, etc.)Bricks can be rendered much faster with a bump or normal map. Try plugging your diffuse map into this website and create a normal map, put that in your bump slot.