An app requests more memory allocation from the OS when it needs it. Perhaps vray has a preference setting to set or change a limit what it uses? So far as I know, there is no macOS way either to limit or to increase the amount of memory used by an app.