Typical reason: a manager user has changed the ACL of the database on a server, which itself is NOT manager of the database. As a result, the database will no longer replicate with other servers.
Solution: Change and save the ACL on a server who himself is manager of the database. Then replicate from this server. The “manager-server” ACL will now be pushed to the other database, and they will start replicating again.
Background: the “consistent ACL setting” requires all changes to the ACL being done on a copy on a server which himself is a manager to that database. If that is not guaranteed, replication will cease. In addition, it is important to know that servers identify the “newer” ACL by their timestamp.
Server A is a manager on the database, Server B is not.
User Donald, who is a manager, accesses the database on server B and changes the ACL.
After this, the databases will no longer replicate. During replication, server A will note that server B has a newer ACL (determined by the ACLs time-stamp) than itself. As server B is not a manager, server A will assume that server B has tampered illegally with the ACL on server B and will thus stop replicating.
Consistent ACLs MUST be used on databases where one of the following scenarios is needed:
If a Database is replicated to user’s desktop/laptop computers and will be accessed locally, only a consistent ACL will guarantee that users will have their desired access level. If roles are used within the ACL of the database, the consistent ACL MUST be used else users of local replica copies will not be able to use their role-based rights.
Consistent ACLs also need to be used when one must make sure that a server administrator person with physical access to the server does not have access to the database (in conjunction with encrypting the database on the server for the server ID).
Databases with consistent ACLs should be managed centrally. There should be ONE server (preferably set as the “administration server” in the enhanced properties of the ACL) which is the databases’ manager, and all other servers should not have manager access to the database. Manager persons should only change ACL settings on that administration server.
This is the most straightforward scenario for database management. If for any reason a manager person changes the ACL on a non-manager-server, this one server will stop replicating while others are not affected. The situation then can be solved by just performing a change to the ACL on the manager server (the “administration server”).
If however, for example, the administration server itself is removed or changed from the ACL of the non-manager-server, first this ACL has to be cleaned up in order to again allow replication from the original manager server. Then (watch the time-stamp!) a final change (whatever) has to be done on the “administration server’s” ACL, so that the ACL of the administration server has a more current time-stamp than the ACL of the downstream server.
Important: Multiple hub replication scenario
In a multiple hub replication scenario, every hub server needs to have manager access to the database, else he will not be able to replicate ACL changes to his spokes! However, it is strongly recommended to instruct manager users to only change ACL settings on the “administration server” to have a clear “top-down” replication and access-rights structure.
After updating the ACL on the administration server in order to correct a downstream server problem, do NOT directly replicate with the affected downstream server, but have the administration server first replicate the ACL update to its “secondary hub” spoke servers, then have those “first level spokes” replicate with their spokes, and so on. If you directly replicate with the affected server, you will run into more problems with upstream servers which then suddenly might stop replicating because the spoke’s ACL is more current than the hubs’. This may or may not happen, depending on the replication schedule and infrastructure.
Alternative, but less-desired scenario:
Give all servers manager access to the database. While this probably ensures smooth replication, it offers a lower degree of security. As databases with consistent ACLs normally use this feature for a reason (in order to enhance security), this scenario is less desired.
Additionally, especially in a multiple-hub replication infrastructure, managing problems with consistent ACLs becomes a little bit more challenging, because there might be no “single point of repair” which then guarantees smooth replication throughout the domain. In this case, the “administration server” should be used exactly the same way as it was in the scenario above.
In the first scenario, solving a consistent ACL problem always involves the “administration server”, then replicating with its spokes, and have them again replicate with their spokes. This is more straightforward.
Typical “workaround” solutions and why they don’t work or are not desired
#1 – NOT DESIRED: delete the database on the server which no longer replicates, and re-create a replica from another server. This is not desirable as all documents which have been created on the deleted replica since their last replication will be lost, too. Additionally, slow lines might prohibit replication of the full database. Also, if the server affected itself is a hub server with serving additional spokes, in turn those spokes might stop replicating after “repairing” the hub. The database will have to be re-deployed over the affected server plus all of its spokes.
#2 – NOT DESIRED: restore the database on the server which no longer replicates from a backup before the ACL has been changed. This is not desirable as all documents which have been created on the deleted replica since the backup will be lost, too. Additionally, the same problems with the affected servers’ spokes as described in workaround #1 above might occur.
#3 – DOES NOT WORK: have the manager person change the ACL on the database copy on the server which no longer replicates back to its original state. Even though the ACLs of both servers are identical, the replicas won’t replicate because the non-manager servers timestamp is more current and the ACL thus will be considered illegal.
Sidenote: another reason for consistent ACLs not working is tampering with the server or manager person computer’s time-date or timezone settings. Though this is a very rare issue, it should be noted. If a server computer is set to an incorrect date (spoke: a date in the future, hub: a date in the past), consistent ACLs might run into problems as they determine being current by looking at their respective timestamps. Server computers should always run with correct date-time settings. Lotus Notes / Domino also supports timezone settings.