To prevent Mac or Linux users from logging in to a particular realm in the first place, just require "Cache Cleaner" to be loaded. (On the authentication policy for Cache Cleaner for the realm, select "Load and enforce cache cleaner.") Since cache cleaner only loads and runs on Windows platforms, this will prevent Mac and Linux users from logging into this realm. The reason you might want to do this is for realms that map users to resources that only work on Windows, such as applications requiring WSAM. In our case, we have some roles that auto-launch WSAM (so that would only work for Windows users), while we have others that auto-launch JSAM (for the Mac users).
If you want to have all users log in to the same realm but map them to different roles based on OS and antivirus host check, you'll need at least two different roles (one for Windows, one for Mac/Linux) and three host check policies: one to pick out Mac/Linux users, one to pick out Windows users, and a third to require Windows users have the proper antivirus. Of course you could combine the Windows and antivirus checks into one host checker policy, but I find it cleaner to separate the two. This also allows a third "restricted" role, for Windows users who pass the OS check but fail the antivirus check. You can then use expressions to mix and match the host checker policy results to map users to the appropriate roles.
To pick out Mac/Linux users and prevent Windows users: "Unix check"
Windows tab: "Process" rule type: Criteria--Process name * ; Action "Deny"
Mac and Unix tab same: "File" rule type; Criteria--File name: /tmp/* ; Action "Required"
Pick out Windows users and prevent Mac/Unix users: "OS check"
Windows tab: "Predefined-OS Check" rule type: Criteria--Specify windows OS *
Mac and Unix tab same: "File" rule type; Criteria--File name: /tmp/* ; Action "Deny"
The "antivirus check" would only pertain to the Windows tab; leave the Mac and Linux tabs blank.
Then, you can create role mappings using custom expressions:
Unix role: map based on custom expression " hostCheckerPolicy = 'Unix Check' "
Windows full role: map based on custom expression " hostCheckerPolicy = 'OS Check' AND hostCheckerPolicy = 'antivirus check' "
Windows restricted role: map based on custom expression " hostCheckerPolicy = 'OS Check' AND hostCheckerPolicy != 'antivirus check' "
Final realm/role mapping would be either a catch-all bare-bones role such as "user=*" - map to Basic role, or none at all if you want to prevent logins that don't pass some sort of host check.