Handling of users that belong to many groups
Users can belong to many different (UNIX) groups. These groups are generally used to allow or deny permissions for executing commands or access to files and directories.
The number of groups a user can belong to depends on the operating system, but there are also components that support fewer groups. In Gluster, there are different restrictions on different levels in the stack. The explanations in this document should clarify which restrictions exist, and how these can be handled.
- if users belong to more than 90 groups, the brick processes need to resolve
the secondary/auxilary groups with the
- the linux kernels
/procfilesystem provides up to 32 groups of a running process, if this is not sufficient the mount option
resolve-gidscan be used
- Gluster/NFS needs
nfs.server-aux-gidswhen users accessing a Gluster volume over NFS belong to more than 16 groups
For all of the above options counts that the system doing the group resolving
must be configured (
sssd, ..) to be able to get all groups when
only a UID is known.
Limit in the GlusterFS protocol
When a Gluster client does some action on a Gluster volume, the operation is sent in an RPC packet. This RPC packet contains an header with the credentials of the user. The server-side receives the RPC packet and uses the credentials from the RPC header to perform ownership operations and allow/deny checks.
The RPC header used by the GlusterFS protocols can contain at most ~93 groups. In order to pass this limit, the server process (brick) receiving the RPC procedure can do the resolving of the groups locally, and ignore the (too few) groups from the RPC header.
This requires that the service process can resolve all of the users groups by the UID of the client process. Most environments that have many groups, already use a configuration where users and groups are maintained in a central location. for enterprises it is common to manage users and their groups in LDAP, Active Directory, NIS or similar.
To have the groups of a user resolved on the server-side (brick), the volume
server.manage-gids needs to be set. Once this option is enabled, the
brick processes will not use the groups that the Gluster clients send, but will
use the POSIX
getgrouplist() function to fetch them.
Because this is a protocol limitation, all clients, including FUSE mounts, Gluster/NFS server and libgfapi applications are affected by this.
Group limit with FUSE
The FUSE client gets the groups of the process that does the I/O by reading the
/proc/$pid/status. This file only contains up to 32 groups.
If client-side xlators rely on all groups of a process/user (like posix-acl),
these 32 groups could limit functionality.
For that reason a mount option has been added. With the
option, the FUSE client calls the POSIX
getgrouplist() function instead of
Group limit for NFS
The NFS protocol (actually the AUTH_SYS/AUTH_UNIX RPC header) allows up to 16
groups. These are the groups that the NFS-server receives from NFS-clients.
Similar to the way the brick processes can resolve the groups on the
server-side, the NFS-server can take the UID passed by the NFS-client and use
that to resolve the groups. the volume option for that is
Other NFS-servers offer options like this too. The Linux kernel nfsd server
rpc.mountd --manage-gids. NFS-Ganesha has the configuration option
Implications of these solutions
All of the mentioned options are disabled by default. one of the reasons is that resolving groups is an expensive operation. in many cases there is no need for supporting many groups and there could be a performance hit.
When groups are resolved, the list is cached. the validity of the cache is
configurable. the Gluster processes are not the only ones that cache these
sssd also keep a cache when they handle the
getgroupslist() requests. When there are many requests, and querying the
groups from a centralized management system takes long, caches might benefit
from a longer validity.
An other, less obvious difference might be noticed too. Many processes that are
written with security in mind reduce the groups that the process can
effectively use. This is normally done with the
setegids() function. When
storage processes do not honour the fewer groups that are effective, and the
processes use the UID to resolve all groups of a process, the groups that got
setegids() are added back again. this could lead to permissions
that the process should not have.