we have the certificate on backend node wit algo RSASSA-PSS . We need to know that algo is upported by the brocade vtm 10.4 or not
That certificate type is not currently supported in vTM, up to and including the current vTM 17.2 release.
Alternatively, vTM 10.4 supports ECDSA certificates, which could be a way forward. ECDSA certificates do not suffer the same weaknesses of the RSA PKCS#1 v1.5 signature scheme used by RSA SSA-PSS.