My suggestion is to not use displacement on brick. You really want to reserve displacement for things that are close to camera have a lot of 3D relief and detail (stonework, etc.) Bricks can be rendered much faster with a bump or normal map. Try plugging your diffuse map into this website and create a normal map, put that in your bump slot. http://cpetry.github.io/NormalMap-Online/