I can't see the nested group option under auth.server --> AD.
attached is the capture of AD auth server on SA
you have chosen "Active Directory / Windows NT" when you created your Auth. Server, right?The nested group option is only available for Auth. Server type "LDAP Server".
Try to create a new Auth. Server. Here you can configure nested groups etc.
Although we also have an AD, we use "LDAP Server" as server type because it is more flexible.