Complete work to fix #90, fix #46, and fix #101.

This ended up being a major overhaul of the code that processes Adobe users who don't match users on the customer side.  The old code was overly complicated, because it was trying to carefully sequence deleting membership in groups with membership in organizations.  But in fact this is not necessary, because removing a user from an organization will remove that user from all the groups in the organization.  So the new code just has to be careful to remove users from accessor (aka trustee) orgs before removing or deleting them in the owning org.

Another major piece of cleanup/overhaul had to do with enumerating users in accessor organizations.  Since the conceptual framework for accessors vs. owners is that the owner org owns the domains that the users are in, whereas the accessors are just granting entitlements to the users in those domains, the whole structure assumes that all users must be in the owner org and only some of them are in the accessor org.  This means that, when we go to enumerate accessors, we want to ignore any dashboard users we find that weren't in the owner org.  In addition, we never want to update attributes in the accessor orgs, because trustees are not allowed to do that.  Finally, when we find an unmatched user that we want to operate on, we need to keep track which accessors we've actually seen that user in, so that we don't just go trying to remove every user from every accessor.  (We could do this, of course, but it's overkill, and currently the server gives errors if you try to delete a user who is not actually in the org.)

The work done here included (in no particular order):

* move to umapi_client 2.1, because it has compatibility fixes for changes in the wire protocol around removing users from organizations and deleting them.
* take advantage of the umapi_client upgrade to correctly delete user accounts.
* update the format of the unmatched users list, so that we also track the org of the unmatched user as well as the user key.  This allows being precise about which users need to be removed from which orgs.
* rename "nonexistent" users to be "unmatched" users in all public contexts.  @phil-levy may or may not like this change.
* rename "orphan users" (the internal word in the code for unmatched) to be "strays".
* make sure that the default new account type is inherited and used by the ldap and csv connectors.
* fix the stats to make sense in all cases
* rework the way we track excluded and included users so it works for accessors as well as the owner org.
* rework the way we track unmatched users by keeping a global map from organizations to the user keys that were deleted in those organizations.  This works even when you haven't loaded the directory or dashboard, as when you are removing users on an input list.
* changed input delete list handling so we don't bother loading directory or dashboard users.

The code now seems dead stable and works with multiple organizations quite well.  The one problem is a strange umapi crash on an unexpected server response when you do --remove-entitlements-for-non-existent-users on multiple orgs.  The crash happens *after* all the server-side processing has been done, so essentially the run completes successfully and then umapi_client immediately throws a ServerError.  I'll have to figure this out for the GM build, but for now testing can proceed on RC2.

Author: adobeDan

docs/success-guide/setup_config_files.md

@@ -112,7 +112,7 @@ A more realistic example is:
 
 ![](images/setup_config_group_map.png)
 
-#### Delete Limits 
+#### Unmatched User Limits 
 
 Limits on deletion prevent accidental account deletion in the event of misconfiguration or some other problem that results in User Sync not getting proper data from the directory system.
 
@@ -121,8 +121,8 @@ Limits on deletion prevent accidental account deletion in the event of misconfig
 ☐ If you expect the number of directory users to drop by more than 200 between User Sync runs, then you will need to raise the max\_missing\_users value.  These config file entries are to prevent runaway deletion in case of misconfiguration or other problems.
 
 	limits:
-	    max_strays_to_process: 10   # ceiling on disable/remove/delete
-	    max_strays_hard_limit: 200      # abort if this many directory users disappear
+	    max_removed_users: 10   # ceiling on disable/remove/delete
+	    max_unmatched_users: 200      # abort if this many directory users disappear
 
 
 

docs/user-manual/index.md

@@ -561,13 +561,13 @@ groups:
 User accounts are removed from the Adobe dashboard when
 corresponding users are not present in the directory and the tool
 is invoked with the `--remove-nonexistent-users` option. The
-`max_strays_to_process` and `max_strays_hard_limit` values in
+`max_removed_users` and `max_unmatched_users` values in
 the `limits` section of the configuration file set limits on
 how many users can be removed at any one time. These limits
 prevent accidental removal of a large number of accounts in case
 of misconfiguration or other errors:
 
-- The value of `max_strays_to_process` sets a limit on the number
+- The value of `max_removed_users` sets a limit on the number
 of account removals in a single run. If more users are flagged
 for removal, they are left for the next run.
 
@@ -577,7 +577,7 @@ raise this value.
 - If your organization has a large number of users in the
 enterprise directory and the number of users read during a sync
 is suddenly small, this could indicate a misconfiguration or
-error situation.  The value of `max_strays_hard_limit` is a threshold
+error situation.  The value of `max_unmatched_users` is a threshold
 which causes the run to exit and report an error if there are
 this many fewer users in the enterprise directory than in the
 Adobe admin console.
@@ -589,8 +589,8 @@ For example:
 
 ```YAML
 limits:
-  max_strays_to_process: 10
-  max_strays_hard_limit: 200
+  max_removed_users: 10
+  max_unmatched_users: 200
 ```
 
 This configuration causes User Sync to remove no more than 10
@@ -693,8 +693,8 @@ directory:
         - "Default Adobe Enterprise Support Program configuration"
 
 limits:
-  max_strays_to_process: 10
-  max_strays_hard_limit: 200
+  max_removed_users: 10
+  max_unmatched_users: 200
 
 logging:
   log_to_file: True

examples/config files - basic/1 user-sync-config.yml

@@ -19,7 +19,8 @@ dashboard:
   # accessor_config_filename_format: "dashboard-accessor-{organization_name}-config.yml"
 
 directory:
-  # (optional) Default country code to use if directory doesn't provide one for a user [Must be two-letter ISO-3166 code - see https://en.wikipedia.org/wiki/ISO_3166-1]
+  # (optional) Default country code to use if directory doesn't provide one for a user
+  # [Must be a two-letter ISO-3166 code - see https://en.wikipedia.org/wiki/ISO_3166-1]
   #
   # example:
   # default_country_code: US
@@ -87,14 +88,14 @@ directory:
   #   - enterpriseID
   #   - federatedID
 
-limits:   # provide processing controls over stray Adobe users (that is, that are unmatched on customer side)
-          # these controls only apply when strays are being processed for removal or deletion from the Adobe side;
-          # if strays are just being output to a file for later processing, these settings do not apply.
-    max_strays_hard_limit: 200   # If more than this many stray users are found, no processing of them is done.
-                                 # Instead, user sync will abort at the point it would remove or delete them.
-    max_strays_to_process: 10    # If more than this many stray users are found, only this many of them will be
-                                 # removed or deleted, and the the run will terminate normally.  Later runs
-                                 # can be used to process the next batch of them.
+limits:   # provide processing controls over Adobe users that do not match users on the customer side
+          # these controls only apply when unmatched users are being removed or deleted from the Adobe side;
+          # if the users are just being output to a file for later processing, these settings do not apply.
+    max_unmatched_users: 200  # If more than this many unmatched users are found, no processing of them is done.
+                              # Instead, user sync will abort at the point it would remove or delete them.
+    max_removed_users: 10     # If more than this many stray users are found, only this many of them will be
+                              # removed or deleted, and the the run will terminate normally.  Later runs
+                              # can be used to process the next batch of them.
 
 logging:
   # specifies whether you wish to generate a log file

examples/config files - custom attributes and mappings/1 user-sync-config.yml

@@ -19,7 +19,8 @@ dashboard:
   # accessor_config_filename_format: "dashboard-accessor-{organization_name}-config.yml"
 
 directory:
-  # (optional) Default country code to use if directory doesn't provide one for a user [Must be two-letter ISO-3166 code - see https://en.wikipedia.org/wiki/ISO_3166-1]
+  # (optional) Default country code to use if directory doesn't provide one for a user
+  # [Must be a two-letter ISO-3166 code - see https://en.wikipedia.org/wiki/ISO_3166-1]
   #
   # example:
   # default_country_code: US
@@ -121,9 +122,14 @@ extensions:
       elif subco == 'Company 2':
         target_groups.add('Company 2 Users')
 
-limits:
-    max_strays_to_process: 10    # if --remove-nonexistent-users is specified, this is the most users that will be removed.  Others will be left for a later run.  A critical message will be logged.
-    max_strays_hard_limit: 200       # if more than this number of user accounts are not found in the directory, user sync will abort with an error and a critical message will be logged.
+limits:   # provide processing controls over Adobe users that do not match users on the customer side
+          # these controls only apply when unmatched users are being removed or deleted from the Adobe side;
+          # if the users are just being output to a file for later processing, these settings do not apply.
+    max_unmatched_users: 200  # If more than this many unmatched users are found, no processing of them is done.
+                              # Instead, user sync will abort at the point it would remove or delete them.
+    max_removed_users: 10     # If more than this many stray users are found, only this many of them will be
+                              # removed or deleted, and the the run will terminate normally.  Later runs
+                              # can be used to process the next batch of them.
 
 logging:
   # specifies whether you wish to generate a log file

setup.py

@@ -43,7 +43,7 @@
           'pycrypto',
           'python-ldap==2.4.25',
           'PyYAML',
-          'umapi-client>=2.0.2',
+          'umapi-client>=2.1',
           'psutil',
       ],
       setup_requires=['nose>=1.0'],

tests/config_test.py

@@ -108,11 +108,11 @@ def test_get_rule_options(self, mock_id_type,mock_get_dict,mock_get_list,mock_ge
             'exclude_users': [],
             'extended_attributes': None,
             'manage_groups': False,
-            'max_strays_hard_limit': 1,
-            'max_strays_to_process': 1,
+            'max_removed_users': 1,
+            'max_unmatched_users': 1,
             'new_account_type': 'new_acc',
             'remove_strays': False,
-            'stray_key_list': None,
+            'stray_key_map': None,
             'stray_list_output_path': None,
             'update_user_info': True,
             'username_filter_regex': None,

user_sync/app.py

@@ -71,33 +71,35 @@ def process_args():
                              'the group membership is updated on the Adobe side so that the memberships in mapped '
                              'groups matches the customer side.',
                         action='store_true', dest='manage_groups')
-    parser.add_argument('--remove-entitlements-for-strays',
-                        help="any 'stray' Adobe users (that is, Adobe users that don't match users on the customer "
-                             "side) are removed from all user groups and product configurations on the Adobe side, "
+    parser.add_argument('--remove-entitlements-for-unmatched-users',
+                        help="any Adobe users that don't match users on the customer "
+                             "side are removed from all user groups and product configurations, "
                              "but they are left visible in the Users list in the Adobe console.",
                         action='store_true', dest='disentitle_strays')
-    parser.add_argument('--remove-strays',
-                        help='like --remove-entitlements-for-strays, but additionally removes stray users '
+    parser.add_argument('--remove-unmatched-users',
+                        help='like --remove-entitlements-for-unmatched-users, but additionally removes unmatched users '
                              'from the Users list in the Adobe console.  The user account is left intact, with all'
                              'of its associated storage, and can be re-added to the Users list if desired.',
                         action='store_true', dest='remove_strays')
-    parser.add_argument('--delete-strays',
-                        help='like --remove-strays, but additionally deletes the '
+    parser.add_argument('--delete-unmatched-users',
+                        help='like --remove-unmatched-users, but additionally deletes the '
                              '(Enterprise or Federated ID) user account for any '
-                             'stray users, so that all of their associated storage is reclaimed and their email '
+                             'unmatched users, so that all of their associated storage is reclaimed and their email '
                              'address is freed up for re-allocation to a new user.',
                         action='store_true', dest='delete_strays')
-    parser.add_argument('--output-stray-list',
-                        help="any 'stray' Adobe users (that is, Adobe users that don't match users on the customer "
-                             "side) are written to a file with the given pathname. "
-                             "This file can then be given in the --stray-list argument in a subsequent run.",
+    parser.add_argument('--output-unmatched-users',
+                        help="all Adobe users that don't match users on the customer "
+                             "side are written to a file with the given pathname, "
+                             "but are not otherwise processed. "
+                             "The output file can then be used with --input-unmatched-users in a subsequent run.",
                         metavar='output_path', dest='stray_list_output_path')
-    parser.add_argument('--input-stray-list',
-                        help='causes stray Adobe users to be read from a file with the given pathname rather than'
-                             'being computed by matching Adobe users with customer users, '
-                             'see --output-stray-list.  When using this option, you must also specify the type'
-                             'of stray processing you want done by specifying one of the options '
-                             '--remove-entitlements-for-strays, --remove-strays or --delete-strays.',
+    parser.add_argument('--input-unmatched-users',
+                        help='instead of computing unmatched users by comparing Adobe users with directory users, '
+                             'the list of unmatched Adobe users is read from a file (see --output-unmatched-users). '
+                             'When using this option, you must also specify what you want done with unmatched users by'
+                             'specifying one of the arguments '
+                             '--remove-entitlements-for-unmatched-users, --remove-unmatched-users '
+                             'or --delete-unmatched-users.',
                         metavar='input_path', dest='stray_list_input_path')
     return parser.parse_args()
 
@@ -251,63 +253,61 @@ def create_config_loader_options(args):
             raise user_sync.error.AssertionException("Bad regular expression for --user-filter: %s reason: %s" % (username_filter_pattern, e.message))
         config_options['username_filter_regex'] = compiled_expression
 
-    # --input-stray-list
+    # --input-unmatched-users
     stray_list_input_path = args.stray_list_input_path
     if (stray_list_input_path != None):
         if users_args is not None:
-            raise user_sync.error.AssertionException('You cannot specify both --users and --input-stray-list')
+            raise user_sync.error.AssertionException('You cannot specify both --users and --input-unmatched-users')
         # don't read the directory when processing from the stray list
         config_options['directory_connector_module_name'] = None
-        logger.info('--input-stray-list specified, so not reading and comparing directory and Adobe users')
-        logger.info('Reading stray list from: %s', stray_list_input_path)
-        stray_key_list = user_sync.rules.RuleProcessor.read_remove_list(stray_list_input_path, logger = logger)
-        logger.info('Total users in stray list: %d', len(stray_key_list))
-        config_options['stray_key_list'] = stray_key_list
+        logger.info('--input-unmatched-users specified, so not reading and comparing directory and Adobe users')
+        stray_key_map = user_sync.rules.RuleProcessor.read_stray_key_map(stray_list_input_path, logger = logger)
+        config_options['stray_key_map'] = stray_key_map
 
-    # --output-stray-list
+    # --output-unmatched-users
     stray_list_output_path = args.stray_list_output_path
     if (stray_list_output_path != None):
         if stray_list_input_path:
             raise user_sync.error.AssertionException('You cannot specify both '
-                                                     '--input-stray-list and --output-stray-list')
-        logger.info('Writing stray list to: %s', stray_list_output_path)
+                                                     '--input-unmatched-users and --output-unmatched-users')
+        logger.info('Writing unmatched users to: %s', stray_list_output_path)
         config_options['stray_list_output_path'] = stray_list_output_path
 
     # keep track of the number user removal type commands
     stray_processing_command_count = 0
 
-    # --remove-strays
+    # --remove-unmatched-users
     remove_strays = args.remove_strays
     if remove_strays:
         if stray_list_output_path:
             remove_strays = False
-            logger.warn('--remove-strays ignored when --output-stray-list is specified')
+            logger.warn('--remove-unmatched-users ignored when --output-unmatched-users is specified')
         stray_processing_command_count += 1
     config_options['remove_strays'] = remove_strays
 
-    # --delete-strays
+    # --delete-unmatched-users
     delete_strays = args.delete_strays
     if delete_strays:
         if stray_list_output_path:
             delete_strays = False
-            logger.warn('--delete-strays ignored when --output-stray-list is specified')
+            logger.warn('--delete-unmatched-users ignored when --output-unmatched-users is specified')
         stray_processing_command_count += 1
     config_options['delete_strays'] = delete_strays
 
-    # --remove-entitlements-for-strays
+    # --remove-entitlements-for-unmatched-users
     disentitle_strays = args.disentitle_strays
     if disentitle_strays:
         if stray_list_output_path:
             disentitle_strays = False
-            logger.warn('--remove-entitlements-for-strays ignored when --output-stray-list is specified')
+            logger.warn('--remove-entitlements-for-unmatched-users ignored when --output-unmatched-users is specified')
         stray_processing_command_count += 1
     config_options['disentitle_strays'] = disentitle_strays
 
     # ensure the user has only entered one "remove user" type command.    
     if stray_processing_command_count > 1:
         raise user_sync.error.AssertionException('You cannot specify more than one of '
-                                                 '--remove-entitlements-for-strays, --remove-strays '
-                                                 'and --delete-strays')
+                                                 '--remove-entitlements-for-unmatched-users, --remove-unmatched-users '
+                                                 'and --delete-unmatched-users')
 
     source_filter_args = args.source_filter_args
     if (source_filter_args != None):

user_sync/config.py

@@ -69,7 +69,7 @@ def __init__(self, caller_options):
             'main_config_filename': DEFAULT_MAIN_CONFIG_FILENAME,
             'manage_groups': False,
             'remove_strays': False,
-            'stray_key_list': None,
+            'stray_key_map': None,
             'stray_list_output_path': None,
             'test_mode': False,
             'update_user_info': True,
@@ -334,8 +334,8 @@ def get_rule_options(self):
             exclude_groups.append(group.get_group_name())
 
         limits_config = self.main_config.get_dict_config('limits')
-        max_strays_hard_limit = limits_config.get_int('max_strays_hard_limit')
-        max_strays_to_process = limits_config.get_int('max_strays_to_process')
+        max_unmatched_users = limits_config.get_int('max_unmatched_users')
+        max_removed_users = limits_config.get_int('max_removed_users')
 
         after_mapping_hook = None
         extended_attributes = None
@@ -383,11 +383,11 @@ def get_rule_options(self):
             'exclude_users': exclude_users,
             'extended_attributes': extended_attributes,
             'manage_groups': options['manage_groups'],
-            'max_strays_hard_limit': max_strays_hard_limit,
-            'max_strays_to_process': max_strays_to_process,
+            'max_removed_users': max_removed_users,
+            'max_unmatched_users': max_unmatched_users,
             'new_account_type': new_account_type,
             'remove_strays': options['remove_strays'],
-            'stray_key_list': options['stray_key_list'],
+            'stray_key_map': options['stray_key_map'],
             'stray_list_output_path': options['stray_list_output_path'],
             'update_user_info': options['update_user_info'],
             'username_filter_regex': options['username_filter_regex'],