you have chosen "Active Directory / Windows NT" when you created your Auth. Server, right?
The nested group option is only available for Auth. Server type "LDAP Server".
Try to create a new Auth. Server. Here you can configure nested groups etc.
Although we also have an AD, we use "LDAP Server" as server type because it is more flexible.