Merge pull request #320 from lyndsey-ferguson/ga-3.15.0

Ga 3.15.0

Author: lyndsey-ferguson

docs/feature_details/multi_scan.md

@@ -33,13 +33,6 @@ If you have a large number of tests, and you want to inspect the overall status
 
 <center><img src="./images/retrying_tests.png" alt="retrying_failed_tests" /></center>
 
-> **Note**: I'm making the control of subsequent runs of `multi_scan` via the `:testrun_complete_block` available to Supporters first to show them appreciation. I'll open it up to everyone else on October 5th, 2020.
->
-> Interested in joining? Click [here ♥️](https://github.com/sponsors/lyndsey-ferguson) and select a tier that gives you early access to new features.
->
-> **Bonus**: if your organization (👨‍👩‍👧‍) becomes a Sponsor, every member of that org gets that same early access!
-
-
 ## Better Results Reporting
 
 Do you have multiple test targets and the normal operation of `:scan` is providing you a junit report that implies that all the tests ran in just one test target? Don't worry, `:multi_scan` has fixed that in v3.14.0 by adding a `package` attribute to each testsuite to let you which test target the test was run in. `:multi_scan` can handle JUnit, HTML, JSON, and Apple's `xcresult` bundles.
@@ -53,20 +46,6 @@ Do you have multiple test targets and the normal operation of `:scan` is providi
 >
 > 3. The `result_bundle` option used to provide `.result_bundle` files. `scan` recently changed that file extension to `.xcresult` to match what Apple is naming the file, and this broke builds expecting the `.result_bundle` extension. As a fix, `multi_scan` creates a `.result_bundle` symbolic link to the `.xcresult` bundle.
 
-****
-
-> **Note**: I'm making the following features available to the plugin's Supporters first. I'll open it up to everyone else on December 18th, 2020:
-> - A new option to pass in a callback when a simulator is started before iOS tests, `:simulator_started_callback`, that will be sent the device's udid.
-> - A new option to allow users to "reuse" a cloned simulator in parallel test runs via the `:reuse_simulators_for_parallel_testruns` option. This improves the speed of startup because apps do not have to be re-installed.
-> - Hardware keyboards are disabled for cloned iOS Simulators.
-> - A new option, `:override_scan_options_block`, to pass in a callback to override the options that will be passed into scan just before it is called for those special cases.
->
-> Interested in joining? Click [here ♥️](https://github.com/sponsors/lyndsey-ferguson) and select a tier that gives you early access to new features.
->
-> **Bonus**: if your organization (👨<U+200D>👩<U+200D>👧<U+200D>) becomes a Sponsor, every member of that org gets that same early access!
-
-
-
 
 ## Examples
 
@@ -111,7 +90,9 @@ multi_scan(
   batch_count: 4,
   fail_build: false,
   parallel_testrun_count: 4,
-  testrun_completed_block: test_run_block
+  testrun_completed_block: test_run_block,
+  simulator_started_callback: sim_callback,
+  override_scan_options_block: override_scan_options_callback
 )
 
 ```
@@ -174,8 +155,10 @@ In addition to the parameters supported by [`scan`](https://docs.fastlane.tools/
 |collate_reports|Whether or not to collate the reports generated by multiple retries, batches, and parallel test runs|true|
 |parallel_testrun_count|Run simulators each batch of tests and/or each test target in parallel on its own Simulator|1|
 |pre_delete_cloned_simulators|Delete left over cloned simulators before running a parallel testrun|true|
-|_reuse_simulators_for_parallel_testruns_<br/>♥️ Coming Dec 18th, 2020|Find simulators (or clone new ones) that match the requested device for the parallel test runs|false|
-|_override_scan_options_block_<br/>♥️ Coming Dec 18th, 2020|A block invoked with a Hash of the scan options that will be used when test run is about to start. This allows your code to modify the arguments that will be sent to scan||
+|reuse_simulators_for_parallel_testruns|Find simulators (or clone new ones) that match the requested device for the parallel test runs|false|
+|override_scan_options_block|A block invoked with a Hash of the scan options that will be used when test run is about to start. This allows your code to modify the arguments that will be sent to scan||
+|reuse_simulators_for_parallel_testruns|Find simulators (or clone new ones) that match the requested device for the parallel test runs. This option sets :pre_delete_cloned_simulators to false|false|
+|override_scan_options_block|A block invoked with a Hash of the scan options that will be used when test run is about to start. This allows your code to modify the arguments that will be sent to scan||
 |testrun_completed_block|A block invoked each time a test run completes. When combined with :parallel_testrun_count, will be called separately in each child process. Return a Hash with :continue set to false to stop retrying tests, or :only_testing to change which tests will be run in the next try||
-|_simulator_started_callback_<br/>♥️ Coming Dec 18th, 2020|A block invoked after the iOS simulators have started||
+|simulator_started_callback|A block invoked after the iOS simulators have started||
 <!-- multi_scan parameters: end -->

lib/fastlane/plugin/test_center/actions/multi_scan.rb

@@ -13,12 +13,7 @@ module Actions
 
     class MultiScanAction < Action
       def self.run(params)
-        params[:quit_simulators] = params._values[:force_quit_simulator] if params._values[:force_quit_simulator]
-        if params[:try_count] < 1
-          UI.important('multi_scan will not test any if :try_count < 0, setting to 1')
-          params[:try_count] = 1
-        end
-
+        update_interdependent_params(params)
         strip_leading_and_trailing_whitespace_from_output_types(params)
 
         warn_of_xcode11_result_bundle_incompatability(params)
@@ -53,6 +48,14 @@ def self.run(params)
         summary
       end
 
+      def self.update_interdependent_params(params)
+        params[:quit_simulators] = params._values[:force_quit_simulator] if params._values[:force_quit_simulator]
+        if params[:try_count] < 1
+          UI.important('multi_scan will not test any if :try_count < 0, setting to 1')
+          params[:try_count] = 1
+        end
+      end
+
       def self.warn_of_parallelism_with_circle_ci(params)
         if params[:parallel_testrun_count] > 1 && Helper.is_circle_ci?
           UI.important("Warning: problems have occurreed when running parallel simulators on Circle CI.")
@@ -456,13 +459,37 @@ def self.available_options
             is_string: false,
             default_value: true
           ),
+          FastlaneCore::ConfigItem.new(
+            key: :override_scan_options_block,
+            description: 'A block invoked with a Hash of the scan options that will be used when test run is about to start. This allows your code to modify the arguments that will be sent to scan',
+            optional: true,
+            is_string: false,
+            default_value: nil,
+            type: Proc
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :reuse_simulators_for_parallel_testruns,
+            description: 'Find simulators (or clone new ones) that match the requested device for the parallel test runs. This option sets :pre_delete_cloned_simulators to false',
+            optional: true,
+            is_string: false,
+            type: Boolean,
+            default_value: false
+          ),
           FastlaneCore::ConfigItem.new(
             key: :testrun_completed_block,
             description: 'A block invoked each time a test run completes. When combined with :parallel_testrun_count, will be called separately in each child process. Return a Hash with :continue set to false to stop retrying tests, or :only_testing to change which tests will be run in the next try',
             optional: true,
             is_string: false,
             default_value: nil,
             type: Proc
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :simulator_started_callback,
+            description: 'A block invoked after the iOS simulators have started',
+            optional: true,
+            is_string: false,
+            default_value: nil,
+            type: Proc
           )
         ]
       end
@@ -492,14 +519,19 @@ def self.example_code
             }
           end
 
+          sim_callback = lambda do |simulator_device_udid|
+            puts \"Start streaming system log for device \#{simulator_device_udid}\"
+          end
+
           multi_scan(
             project: File.absolute_path('../AtomicBoy/AtomicBoy.xcodeproj'),
             scheme: 'AtomicBoy',
             try_count: 3,
             batch_count: 4,
             fail_build: false,
             parallel_testrun_count: 4,
-            testrun_completed_block: test_run_block
+            testrun_completed_block: test_run_block,
+            simulator_started_callback: sim_callback
           )
           ",
           "

lib/fastlane/plugin/test_center/helper/html_test_report.rb

@@ -8,6 +8,12 @@ def self.verbose(message)
         FastlaneCore::UI.verbose(message)
       end
 
+      def self.error(message)
+        return if ENV.fetch('COLLATE_HTML_REPORTS_VERBOSITY', 1).to_i.zero?
+
+        FastlaneCore::UI.error(message)
+      end
+
       class Report
         require 'rexml/formatters/transitive'
 

lib/fastlane/plugin/test_center/helper/multi_scan_manager/device_manager.rb

@@ -18,6 +18,17 @@ def rename(newname)
         self.name = newname
       end
 
+      def disable_hardware_keyboard
+        UI.verbose("Disabling hardware keyboard for #{self.udid}")
+        plist_filepath = File.expand_path("~/Library/Preferences/com.apple.iphonesimulator.plist")
+        keyboard_pref_key = ":DevicePreferences:#{self.udid}:ConnectHardwareKeyboard"
+
+        command = "/usr/libexec/PlistBuddy -c \"Set #{keyboard_pref_key} false\" #{plist_filepath} 2>/dev/null || "
+        command << "/usr/libexec/PlistBuddy -c \"Add #{keyboard_pref_key} bool false\" #{plist_filepath}"
+
+        `#{command}`
+      end
+
       def boot
         return unless is_simulator
         return unless os_type == "iOS"

lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan.rb

@@ -7,57 +7,6 @@ def initialize(options = {})
           @retrying_scan_helper = RetryingScanHelper.new(@options)
         end
 
-        # :nocov:
-        def scan_config
-          Scan.config
-        end
-
-        def scan_cache
-          Scan.cache
-        end
-        # :nocov:
-
-        def prepare_scan_config
-          # this allows multi_scan's `destination` option to be picked up by `scan`
-          scan_config._values.delete(:device)
-          ENV.delete('SCAN_DEVICE')
-          scan_config._values.delete(:devices)
-          ENV.delete('SCAN_DEVICES')
-          # this prevents double -resultBundlePath args to xcodebuild
-          if ReportNameHelper.includes_xcresult?(@options[:output_types])
-            scan_config._values.delete(:result_bundle)
-            ENV.delete('SCAN_RESULT_BUNDLE')
-          end
-          scan_config._values.delete(:skip_testing)
-          scan_cache.clear
-        end
-
-        def update_scan_options
-          valid_scan_keys = Fastlane::Actions::ScanAction.available_options.map(&:key)
-          scan_options = @options.select { |k,v| valid_scan_keys.include?(k) }
-                                  .merge(@retrying_scan_helper.scan_options)
-
-          prepare_scan_config
-          scan_options[:build_for_testing] = false
-          scan_options.delete(:skip_testing)
-          FastlaneCore::UI.verbose("retrying_scan #update_scan_options")
-          scan_options.each do |k,v|
-            next if v.nil?
-
-            scan_config.set(k,v) unless v.nil?
-            FastlaneCore::UI.verbose("\tSetting #{k.to_s} to #{v}")
-          end
-          if @options[:scan_devices_override]
-            scan_device_names = @options[:scan_devices_override].map { |device| device.name }
-            FastlaneCore::UI.verbose("\tSetting Scan.devices to #{scan_device_names}")
-            if Scan.devices
-              Scan.devices.replace(@options[:scan_devices_override])
-            else
-              Scan.devices = @options[:scan_devices_override]
-            end
-          end
-        end
-
         # :nocov:
         def self.run(options)
           RetryingScan.new(options).run
@@ -68,12 +17,6 @@ def run
           try_count = @options[:try_count] || 1
           begin
             @retrying_scan_helper.before_testrun
-            update_scan_options
-
-            values = scan_config.values(ask: false)
-            values[:xcode_path] = File.expand_path("../..", FastlaneCore::Helper.xcode_path)
-            ScanHelper.print_scan_parameters(values)
-
             Scan::Runner.new.run
             @retrying_scan_helper.after_testrun
             true

lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan_helper.rb

@@ -23,9 +23,78 @@ def before_testrun
           delete_xcresults # has to be performed _after_ moving a *.test_result
           quit_simulator
           set_json_env
+          set_scan_config
           print_starting_scan_message
         end
 
+        def set_scan_config
+          valid_scan_keys = Fastlane::Actions::ScanAction.available_options.map(&:key)
+          new_scan_options = @options.select { |k,v| valid_scan_keys.include?(k) }
+                                 .merge(scan_options)
+
+          prepare_scan_config
+          new_scan_options[:build_for_testing] = false
+          new_scan_options.delete(:skip_testing)
+
+          new_scan_options = send_callback_override_scan_options_block(new_scan_options)
+
+          FastlaneCore::UI.verbose("retrying_scan #update_scan_options")
+          new_scan_options.each do |k,v|
+            next if v.nil?
+
+            scan_config.set(k,v) unless v.nil?
+            FastlaneCore::UI.verbose("\tSetting #{k.to_s} to #{v}")
+          end
+          if @options[:scan_devices_override]
+            scan_device_names = @options[:scan_devices_override].map { |device| device.name }
+            FastlaneCore::UI.verbose("\tSetting Scan.devices to #{scan_device_names}")
+            if Scan.devices
+              Scan.devices.replace(@options[:scan_devices_override])
+            else
+              Scan.devices = @options[:scan_devices_override]
+            end
+          end
+ 
+          values = scan_config.values(ask: false)
+          values[:xcode_path] = File.expand_path("../..", FastlaneCore::Helper.xcode_path)
+          ScanHelper.print_scan_parameters(values)
+        end
+
+        # :nocov:
+        def scan_config
+          Scan.config
+        end
+
+        def scan_cache
+          Scan.cache
+        end
+        # :nocov:
+
+        def prepare_scan_config
+          # this allows multi_scan's `destination` option to be picked up by `scan`
+          scan_config._values.delete(:device)
+          ENV.delete('SCAN_DEVICE')
+          scan_config._values.delete(:devices)
+          ENV.delete('SCAN_DEVICES')
+          # this prevents double -resultBundlePath args to xcodebuild
+          if ReportNameHelper.includes_xcresult?(@options[:output_types])
+            scan_config._values.delete(:result_bundle)
+            ENV.delete('SCAN_RESULT_BUNDLE')
+          end
+          scan_config._values.delete(:skip_testing)
+          scan_cache.clear
+        end
+
+        def send_callback_override_scan_options_block(new_scan_options)
+          return new_scan_options unless @options[:override_scan_options_block]
+
+          callback_result = @options[:override_scan_options_block].call(new_scan_options)
+          if callback_result.kind_of?(Hash)
+            return callback_result
+          end
+          new_scan_options
+        end
+
         def quit_simulator
           return unless @options[:quit_simulators]
 

lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb

@@ -21,6 +21,8 @@ def initialize(multi_scan_options)
             update_options_to_use_xcresult_output
           end
           @batch_count = 1 # default count. Will be updated by setup_testcollector
+          @options[:parallel_testrun_count] ||= 1
+          @initial_parallel_testrun_count = @options[:parallel_testrun_count]
           setup_testcollector
           setup_logcollection
           FastlaneCore::UI.verbose("< done in TestCenter::Helper::MultiScanManager.initialize")
@@ -67,6 +69,7 @@ def setup_testcollector
           @test_collector = TestCollector.new(@options)
           @options.reject! { |key| %i[testplan].include?(key) }
           @batch_count = @test_collector.batches.size
+          @options[:parallel_testrun_count] = @initial_parallel_testrun_count
           tests = @test_collector.batches.flatten
           if tests.size < @options[:parallel_testrun_count].to_i
             FastlaneCore::UI.important(":parallel_testrun_count greater than the number of tests (#{tests.size}). Reducing to that number.")
@@ -184,6 +187,8 @@ def run_tests_through_single_try
           options = @options.reject { |key| %i[device devices force_quit_simulator].include?(key) }
           options[:try_count] = 1
 
+          SimulatorHelper.call_simulator_started_callback(@options, Scan.devices)
+
           tests_passed = RetryingScan.run(options)
           @options[:try_count] -= 1
 
@@ -229,6 +234,11 @@ def run_test_batches
           pool_options[:test_batch_results] = test_batch_results
           pool_options[:xctestrun] = @test_collector.xctestrun_path
 
+          serial_test_batches = (@options.fetch(:parallel_testrun_count, 1) == 1)
+          if serial_test_batches && !@options[:invocation_based_tests]
+            SimulatorHelper.call_simulator_started_callback(@options, Scan.devices)
+          end
+
           pool = TestBatchWorkerPool.new(pool_options)
           pool.setup_workers