I'd like to know if it's possible to check group membership BEFORE authentification (ie without realm checking).
I want to block DoS attack on every accounts of our AD domain.
I don't think it would be possible to do group membership test before authentication but the "Lockout options" on the Configuration\Security page are designed to mitigate DoS attacks. It's a good idea to read the example in the help to understand how to configure these but basically if the box detects repeated failed attempts from a single source it will temporarily lock it out that source.
If you use two factor authentication, you can put your token as the primary authentication server and AD/LDAP as the secondary server.
Depending on your token lockout settings, this might allow a token to be locked and prevent remote logon, but effectively eliminates the possibility of DOS attack on an AD account.