Creating extensions for ZAP files using .zapExtension files (#1552)

- .zapExtension files can now be used to extend the .zap files to add additional clusters, commands or attributes on an endoint
- Adding the logic to do this during generation or during file open using command line arguments.
- Adding unit tests
- Adding documentation on usage
- Added an attribute to the xml so updated one of the existing tests
- Fixing existing tests and adding tests for zap extensions with clusters and attributes on an endpoint type id
- Linking the extension file to the endpoint identifiers and not endpoint type ids
- Github: ZAP#1254

Author: brdandu

README.md

@@ -39,6 +39,7 @@ This software is licensed under [Apache 2.0 license](LICENSE.txt).
 
 - [ZAP Template Helpers](docs/helpers.md)
 - [ZAP External Template Helpers](docs/external-helpers.md)
+- [ZAP file Extensions](docs/zap-file-extensions.md)
 - [FAQ/Developer dependencies](docs/faq.md)
 - [Release instructions](docs/release.md)
 - [Development Instructions](docs/development-instructions.md)

docs/api.md

@@ -16929,11 +16929,11 @@ Utility module for ZAP UI
 
 * [JS API: Utility module for ZAP UI](#module_JS API_ Utility module for ZAP UI)
     * [~showErrorMessage(title, err)](#module_JS API_ Utility module for ZAP UI..showErrorMessage)
-    * [~openFileConfiguration(db, filePath, httpPort)](#module_JS API_ Utility module for ZAP UI..openFileConfiguration)
+    * [~openFileConfiguration(db, filePath, httpPort, zapFileExtensions)](#module_JS API_ Utility module for ZAP UI..openFileConfiguration)
     * [~openNewConfiguration(httpPort, options:)](#module_JS API_ Utility module for ZAP UI..openNewConfiguration)
     * [~toggleDirtyFlag(browserWindow, dirty)](#module_JS API_ Utility module for ZAP UI..toggleDirtyFlag)
     * [~openFileDialogAndReportResult(browserWindow, options)](#module_JS API_ Utility module for ZAP UI..openFileDialogAndReportResult)
-    * [~enableUi(port, zapFiles, uiMode, standalone)](#module_JS API_ Utility module for ZAP UI..enableUi) ⇒
+    * [~enableUi(port, zapFiles, uiMode, standalone, zapFileExtensions)](#module_JS API_ Utility module for ZAP UI..enableUi) ⇒
 
 <a name="module_JS API_ Utility module for ZAP UI..showErrorMessage"></a>
 
@@ -16949,7 +16949,7 @@ Simple dialog to show error messages from electron renderer scope.
 
 <a name="module_JS API_ Utility module for ZAP UI..openFileConfiguration"></a>
 
-### JS API: Utility module for ZAP UI~openFileConfiguration(db, filePath, httpPort)
+### JS API: Utility module for ZAP UI~openFileConfiguration(db, filePath, httpPort, zapFileExtensions)
 Process a single file, parsing it in as JSON and then possibly opening
 a new window if all is good.
 
@@ -16960,6 +16960,7 @@ a new window if all is good.
 | db | <code>\*</code> |  |
 | filePath | <code>\*</code> |  |
 | httpPort | <code>\*</code> | Server port for the URL that will be constructed. |
+| zapFileExtensions | <code>\*</code> | Extend a zap file with zapExtension |
 
 <a name="module_JS API_ Utility module for ZAP UI..openNewConfiguration"></a>
 
@@ -17001,7 +17002,7 @@ reports result back through the API.
 
 <a name="module_JS API_ Utility module for ZAP UI..enableUi"></a>
 
-### JS API: Utility module for ZAP UI~enableUi(port, zapFiles, uiMode, standalone) ⇒
+### JS API: Utility module for ZAP UI~enableUi(port, zapFiles, uiMode, standalone, zapFileExtensions) ⇒
 Enable the UI open using the given arguments.
 
 **Kind**: inner method of [<code>JS API: Utility module for ZAP UI</code>](#module_JS API_ Utility module for ZAP UI)  
@@ -17013,6 +17014,7 @@ Enable the UI open using the given arguments.
 | zapFiles | <code>\*</code> | 
 | uiMode | <code>\*</code> | 
 | standalone | <code>\*</code> | 
+| zapFileExtensions | <code>\*</code> | 
 
 <a name="module_JS API_ Window module for ZAP UI"></a>
 

docs/zap-file-extensions.md

@@ -0,0 +1,25 @@
+# ZAP File Extensions
+
+## Overview
+
+ZAP file extensions allow users to extend the functionality of existing ZAP configuration files(.zap) by merging additional data from extension files(.zapExtension). This feature is particularly useful for adding new clusters, attributes, or other configuration elements to an existing endpoint identifier without modifying the base ZAP file. Note that if the extension element is already present in the base configuration then it will not be modified. Also note that if you open a .zap baseline file along with a .zapExtension extension file and then save the configuration then this will always produce a single .zap file which includes the extension configuration.
+
+## How It Works
+
+1. **Base ZAP File(.zap)**: The primary configuration file containing the initial setup. eg [zapFile](../test/resource/lighting-matter.zap)
+2. **Extension ZAP File(.zapExtension)**: A supplementary file that adds to elements in the base file. eg [zapFileExtension](../test/resource/zapExtension1.zapExtension). This file shows how everything is wrapped within endpoints object and linked to the endpoint identifier.
+3. **Merging Process**: During the import process, the extension file's content is merged into the base file content. However, if you save this imported file along with its extension then this saves the content in one explicit file.
+
+## Usage
+
+### Generation
+
+[ZAP executable] generate --noUi --noServer -o [output directory path] --packageMatch fuzzy --zcl [path to zcl.json in SDK] --generationTemplate [path to generation templates.json in SDK] --in [path to input zap file] --inE [path to zap extension file] --noLoadingFailure --appendGenerationSubdirectory
+
+### Launching UI
+
+[ZAP executable] --packageMatch fuzzy --zcl [path to zcl.json in SDK] --generationTemplate [path to generation templates.json in SDK] --in [path to input zap file] --inE [path to zap extension file] --noLoadingFailure --appendGenerationSubdirectory
+
+### References
+
+- ZAP executable -> [ZAP binary](https://github.com/project-chip/zap/releases) or `node src-script/zap-start.js`

src-electron/importexport/import.js

@@ -81,6 +81,68 @@ async function executePostImportScript(db, sessionId, scriptFile) {
   )
 }
 
+/**
+ * Merge extension file content into the base zap file content.
+ *
+ * @param {Object} baseState - The state object of the base .zap file.
+ * @param {Object} extensionState - The state object of the extension .zap file.
+ */
+function mergeZapExtension(baseState, extensionState) {
+  if (extensionState.endpoints) {
+    extensionState.endpoints.forEach((extEndpoint) => {
+      // Finding an endpoint using endpoint identifier linked to extension endpoint Id
+      const baseEndpoint = baseState.endpoints.find(
+        (et) => et.endpointId === extEndpoint.id
+      )
+      if (!baseEndpoint) {
+        return
+      }
+
+      // Finding the appropriate endpointType in the .zap file/base State
+      const baseEndpointType =
+        baseState.endpointTypes[baseEndpoint.endpointTypeIndex]
+
+      if (baseEndpointType) {
+        extEndpoint.clusters.forEach((extCluster) => {
+          const baseCluster = baseEndpointType.clusters.find(
+            (bc) => bc.code === extCluster.code
+          )
+
+          if (baseCluster) {
+            // Merge attributes
+            if (extCluster.attributes) {
+              baseCluster.attributes = baseCluster.attributes || []
+              extCluster.attributes.forEach((extAttr) => {
+                const existingAttr = baseCluster.attributes.find(
+                  (attr) => attr.code === extAttr.code
+                )
+                if (!existingAttr) {
+                  baseCluster.attributes.push(extAttr)
+                }
+              })
+            }
+            // Merge commands
+            if (extCluster.commands) {
+              baseCluster.commands = baseCluster.commands || []
+              extCluster.commands.forEach((extAttr) => {
+                const existingCmd = baseCluster.commands.find(
+                  (attr) => attr.code === extAttr.code
+                )
+                if (!existingCmd) {
+                  baseCluster.commands.push(extAttr)
+                }
+              })
+            }
+          } else {
+            // Add new cluster if it doesn't exist
+            baseEndpointType.clusters.push(extCluster)
+          }
+        })
+      }
+    })
+  }
+}
+
 /**
  * Writes the data from the file into a new session.
  * NOTE: This function does NOT initialize session packages.
@@ -101,6 +163,20 @@ async function importDataFromFile(
   }
 ) {
   let state = await readDataFromFile(filePath, options.defaultZclMetafile)
+  // Merge extension files into the state
+  if (
+    options.extensionFiles &&
+    Array.isArray(options.extensionFiles) &&
+    options.extensionFiles.length > 0
+  ) {
+    for (const extensionFile of options.extensionFiles) {
+      const extensionState = await readDataFromFile(
+        extensionFile,
+        options.defaultZclMetafile
+      )
+      mergeZapExtension(state, extensionState)
+    }
+  }
 
   state = ff.convertFromFile(state)
   try {

src-electron/main-process/startup.js

@@ -834,7 +834,8 @@ async function generateSingleFile(
       packageMatch: options.packageMatch,
       defaultTemplateFile: options.template,
       upgradeZclPackages: upgradeZclPackages,
-      upgradeTemplatePackages: upgradeTemplatePackages
+      upgradeTemplatePackages: upgradeTemplatePackages,
+      extensionFiles: options.extensionFiles
     })
     sessionId = importResult.sessionId
     output = outputFile(zapFile, outputPattern, index)
@@ -910,11 +911,13 @@ async function startGeneration(argv, options) {
   let zclProperties = argv.zclProperties
   let genResultFile = argv.genResultFile
   let skipPostGeneration = argv.skipPostGeneration
+  let zapFileExtensions = argv.zapFileExtension ? [argv.zapFileExtension] : []
 
   let hrstart = process.hrtime.bigint()
   options.logger(
     `🤖 ZAP generation started:
     🔍 input files: ${zapFiles}
+    🔍 input Extension files: ${zapFileExtensions}
     🔍 output pattern: ${output}
     🔍 using templates: ${templateMetafile}
     🔍 using zcl data: ${zclProperties}
@@ -942,6 +945,10 @@ async function startGeneration(argv, options) {
   let globalTemplatePackageId = ctx.packageId
 
   let files = gatherFiles(zapFiles, { suffix: '.zap', doBlank: true })
+  let extensionFiles = gatherFiles(zapFileExtensions, {
+    suffix: '.zapExtension',
+    doBlank: false
+  })
   if (files.length == 0) {
     options.logger(`    👎 no zap files found in: ${zapFiles}`)
     throw `👎 no zap files found in: ${zapFiles}`
@@ -959,6 +966,11 @@ async function startGeneration(argv, options) {
   // Used to upgrade the zap file during generation. Makes sure packages are
   // updated in .zap file during project creation in Studio.
   options.upgradeZapFile = argv.upgradeZapFile
+  // Used for extending all the .zap files. Users can extend cluster
+  // configurations on an endpoint type id mentioned
+  if (extensionFiles && extensionFiles.length > 0) {
+    options.extensionFiles = extensionFiles
+  }
 
   let nsDuration = process.hrtime.bigint() - hrstart
   options.logger(`🕐 Setup time: ${util.duration(nsDuration)} `)
@@ -1174,8 +1186,18 @@ async function startUpMainInstance(argv, callbacks) {
     let uiEnabled = !argv.noUi
     let zapFiles = argv.zapFiles
     let port = await startNormal(quitFunction, argv)
+    let zapFileExtensions = null
+    if ('zapFileExtension' in argv) {
+      zapFileExtensions = [argv.zapFileExtension]
+    }
     if (uiEnabled && uiFunction != null) {
-      uiFunction(port, zapFiles, argv.uiMode, argv.standalone)
+      uiFunction(
+        port,
+        zapFiles,
+        argv.uiMode,
+        argv.standalone,
+        zapFileExtensions
+      )
     } else {
       if (argv.showUrl) {
         // NOTE: this is parsed/used by Studio as the default landing page.

src-electron/rest/file-ops.js

@@ -45,6 +45,8 @@ function httpPostFileOpen(db) {
     let search = req.body.search
     const query = new URLSearchParams(search)
     let file = query.get('filePath')
+    // Gather .zapExtension files
+    let zapFileExtensions = query.get('zapFileExtensions')
     if (file) {
       zapFilePath = file
       ideProjectPath = query.get('studioProject')
@@ -76,10 +78,15 @@ function httpPostFileOpen(db) {
           dbEnum.sessionKey.ideProjectPath,
           ideProjectPath
         )
-
-        let importResult = await importJs.importDataFromFile(db, zapFilePath, {
-          sessionId: req.zapSessionId
-        })
+        let options = { sessionId: req.zapSessionId }
+        if (zapFileExtensions) {
+          options.extensionFiles = [zapFileExtensions]
+        }
+        let importResult = await importJs.importDataFromFile(
+          db,
+          zapFilePath,
+          options
+        )
 
         let response = {
           sessionId: importResult.sessionId,

src-electron/rest/initialize.js

@@ -40,6 +40,7 @@ function sessionAttempt(db) {
 
     const query = new URLSearchParams(search)
     let filePath = query.get('filePath')
+    let filePathExtension = query.get('zapFileExtensions')
     if (filePath) {
       if (filePath.includes('.zap')) {
         let data = await fsp.readFile(filePath)
@@ -72,7 +73,8 @@ function sessionAttempt(db) {
             sessions,
             filePath,
             zapFilePackages,
-            open
+            open,
+            filePathExtension
           })
         } else {
           let open = true

src-electron/ui/ui-util.js

@@ -49,11 +49,18 @@ function showErrorMessage(title, err) {
  * @param {*} db
  * @param {*} filePath
  * @param {*} httpPort Server port for the URL that will be constructed.
+ * @param {*} zapFileExtensions Extend a zap file with zapExtension
  */
-function openFileConfiguration(filePath, httpPort, standalone = false) {
+function openFileConfiguration(
+  filePath,
+  httpPort,
+  standalone = false,
+  zapFileExtensions = null
+) {
   window.windowCreate(httpPort, {
     filePath,
-    standalone
+    standalone,
+    zapFileExtensions
   })
 }
 
@@ -145,9 +152,10 @@ function openFileDialogAndReportResult(browserWindow, options) {
  * @param {*} zapFiles
  * @param {*} uiMode
  * @param {*} standalone
+ * @param {*} zapFileExtensions
  * @returns promise of a file open configuration
  */
-function enableUi(port, zapFiles, uiMode, standalone) {
+function enableUi(port, zapFiles, uiMode, standalone, zapFileExtensions) {
   window.initializeElectronUi(port)
   if (zapFiles.length == 0) {
     return openNewConfiguration(port, {
@@ -157,7 +165,7 @@ function enableUi(port, zapFiles, uiMode, standalone) {
     })
   } else {
     return util.executePromisesSequentially(zapFiles, (f) =>
-      openFileConfiguration(f, port)
+      openFileConfiguration(f, port, standalone, zapFileExtensions)
     )
   }
 }

src-electron/ui/window.js

@@ -67,7 +67,14 @@ export function windowCreateIfNotThere(port) {
  * @param {*} restPort
  * @returns String
  */
-function createQueryString(uiMode, standalone, isNew, filePath, restPort) {
+function createQueryString(
+  uiMode,
+  standalone,
+  isNew,
+  filePath,
+  restPort,
+  zapFileExtensions
+) {
   const params = new Map()
 
   if (!arguments.length) {
@@ -88,6 +95,9 @@ function createQueryString(uiMode, standalone, isNew, filePath, restPort) {
   if (filePath !== undefined) {
     params.set('filePath', filePath)
   }
+  if (Array.isArray(zapFileExtensions) && zapFileExtensions.length > 0) {
+    params.set('zapFileExtensions', zapFileExtensions)
+  }
 
   // Electron/Development mode
   if (
@@ -145,7 +155,8 @@ export function windowCreate(port, args) {
     args?.standalone,
     args?.new,
     args?.filePath,
-    httpServer.httpServerPort()
+    httpServer.httpServerPort(),
+    args?.zapFileExtensions
   )
 
   // @ts-ignore

src-electron/util/args.js

@@ -86,6 +86,12 @@ export function processCommandLineArguments(argv) {
       type: 'string',
       default: null
     })
+    .option('zapExtensionFile', {
+      desc: 'input .zapExtension file to read in to extend the given .zap file.',
+      alias: ['zapExtension', 'inE', 'iE', 'zapFileExtension'],
+      type: 'string',
+      default: null
+    })
     .option('zclProperties', {
       desc: 'zcl.properties file to read in.',
       alias: ['zcl', 'z'],
@@ -271,11 +277,20 @@ For more information, see ${commonUrl.projectUrl}`
     if (typeof arg == 'number') return false
     if (arg.endsWith('.js')) return false
     if (commands.has(arg)) return false
+    if (commands.has('.zapExtension')) return false
     return true
   })
+  // Collect all zapExtensions
+  let allZapFileExtensions = ret._.filter((arg) => {
+    return arg.endsWith('.zapExtension')
+  })
   if (ret.zapFile != null) allFiles.push(ret.zapFile)
   ret.zapFiles = allFiles
 
+  if (allZapFileExtensions && allZapFileExtensions.length > 0) {
+    ret.zapFileExtensions = allZapFileExtensions
+  }
+
   if (ret.tempState) {
     let tempDir = fs.mkdtempSync(`${os.tmpdir()}${path.sep}zap.`)
     console.log(