I think your best bet will be getting AD under control. We use dedicated AD groups for our remote access to give various different roles and make the name mean something. Also, not allowing nested groups within that group (only individual users) will help.
Name something like RemoteAccess-RDP, RemoteAccess-View, RemoteAccess-Meetings, RemoteAccess-SSH, etc.