Fix JSON parameter parsing in warnet bitcoin rpc command

[b-l-u-e]

Description

This PR addresses issue #714 by implementing JSON parameter handling for the warnet bitcoin rpc command. Currently, the command fails when users attempt to pass JSON parameters due to missing parameter processing logic.

Problem Analysis

The issue manifests when users attempt to pass JSON parameters to Bitcoin Core RPC methods:

$ warnet bitcoin rpc tank-0000 getblock '["146cc34515ae4ad2e938c4b8a899ac2889b37f603e9bba51854eb14540250805"]' 'true'
error code: -8
error message: blockhash must be of length 64 (not 66, for '[146cc34515ae4ad2e938c4b8a899ac2889b37f603e9bba51854eb14540250805]')
command terminated with exit code 8

Root Cause: The current _rpc() function has incorrect parameter handling:

1. Function signature expects params: str but should be params: List[str]
2. No JSON parameter processing logic exists
3. Shell parsing breaks JSON arrays into separate arguments
4. Bitcoin Core receives malformed input like [146cc34515ae4ad2e938c4b8a899ac2889b37f603e9bba51854eb14540250805] instead of the actual hash

Technical Solution

Parameter Type Fix

• Function Signature: Change params: str to params: List[str] to properly handle multiple arguments
• Argument Processing: Add proper parameter reconstruction and processing logic

JSON Reconstruction Logic

Add _reconstruct_json_params() function that:

1. Detects Split JSON: Identifies when JSON structures are fragmented across multiple arguments
2. Reconstructs Parameters: Joins split JSON parts back together
3. Quote Restoration: Adds quotes to unquoted string arrays ([network] → ["network"])
4. Validation: Ensures reconstructed JSON is syntactically valid

RPC Parameter Processing

Enhance _rpc() function with:

• JSON Array Handling: Properly process JSON arrays for bitcoin-cli
• Type Preservation: Handle boolean, numeric, and string values correctly
• Backward Compatibility: Maintain support for non-JSON parameters

Verification

Current Behavior (Broken)

$ warnet bitcoin rpc tank-0000 getblock '["146cc34515ae4ad2e938c4b8a899ac2889b37f603e9bba51854eb14540250805"]' 'true'
error code: -8
error message: blockhash must be of length 64 (not 66, for '[146cc34515ae4ad2e938c4b8a899ac2889b37f603e9bba51854eb14540250805]')

Expected Behavior (After Fix)

$ warnet bitcoin rpc tank-0000 getblock '["146cc34515ae4ad2e938c4b8a899ac2889b37f603e9bba51854eb14540250805"]' 'true'
{
  "hash": "146cc34515ae4ad2e938c4b8a899ac2889b37f603e9bba51854eb14540250805",
  "confirmations": 1,
  ...
}

Impact

This fix will enable:

• Proper JSON Parameter Handling: Support for arrays, objects, and complex parameters
• Bitcoin Core RPC Compatibility: Correct parameter formatting for bitcoin-cli
• Shell Safety: Robust handling of shell argument parsing edge cases
• Backward Compatibility: Existing non-JSON RPC calls continue to work

Files Modified

• src/warnet/bitcoin.py: Fix parameter type and add JSON reconstruction logic

Fixes #714

[pinheadmz]

Expected this to work:

 --> warnet bitcoin rpc tank-0001 logging '["http","rpc"]'
error: Error parsing JSON: http
command terminated with exit code 1

It did work with slashes:
warnet bitcoin rpc tank-0001 logging '[\"http\"]'

[pinheadmz]

Another thing that may be out of scope for this PR is there can be a swallowed error
bash: line 1: /usr/local/bin/kubectl: Argument list too long

for example this beast, which I'd love to see working at some point:

warnet bitcoin rpc tank-0001 importdescriptors '"descriptors":[{"desc":"pkh(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/44h/1h/0h/0/)#ulcldqpa","timestamp":1750949660,"active":true,"internal":false,"range":[0,999],"next":0,"next_index":0},{"desc":"pkh(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/44h/1h/0h/1/)#dta7s439","timestamp":1750949660,"active":true,"internal":true,"range":[0,999],"next":0,"next_index":0},{"desc":"sh(wpkh(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/49h/1h/0h/0/))#ygm43e8l","timestamp":1750949660,"active":true,"internal":false,"range":[0,999],"next":0,"next_index":0},{"desc":"sh(wpkh(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/49h/1h/0h/1/))#ztns25vt","timestamp":1750949660,"active":true,"internal":true,"range":[0,999],"next":0,"next_index":0},{"desc":"tr(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/86h/1h/0h/0/)#6pett65n","timestamp":1750949660,"active":true,"internal":false,"range":[0,999],"next":0,"next_index":0},{"desc":"tr(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/86h/1h/0h/1/)#t4u2k0yt","timestamp":1750949660,"active":true,"internal":true,"range":[0,999],"next":0,"next_index":0},{"desc":"wpkh(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/84h/1h/0h/0/)#8rwp50x9","timestamp":1750949660,"active":true,"internal":false,"range":[0,1000],"next":1,"next_index":1},{"desc":"wpkh(tprv8ZgxMBicQKsPdDAbWAtdMmeLt8H9t8aTwHARZ2paraAf3D167mWwqvxUcpmFmPb538Ymepz1cPLjiwzGkR7ep7yxFATw4sX2BFmqsPJYnoW/84h/1h/0h/1/)#khtqf6ka","timestamp":1750949660,"active":true,"internal":true,"range":[0,999],"next":0,"next_index": 0}]'

[pinheadmz]

This seems like a lot of work, implementing our own JSON parsing? Did you try any other approaches? Is there a better way to get the raw input string from click perhaps?

[b-l-u-e]

i will adjust the changes..thank you for the review

[pinheadmz]

I cherry-picked the lint fix to main, if you wanna rebase to run ci

[b-l-u-e]

The following is fixed:

• JSON arrays for logging: '["http","rpc"]'
• JSON arrays for getblock: '["blockhash"]' → extracts blockhash
• Plain string parameters still work: getblock 'blockhash'
• Empty arrays handled correctly

[b-l-u-e]

#725 (comment)

For the long importdescriptors command you mentioned, a few potential solutions could be explored in a future PR:
maybe file-based approach writing the descriptor data to a temporary file and pass the file path instead of the raw JSON
or perhaps chunking like split large descriptor arrays into smaller batches or use a different method to pass large data (like environment variables or stdin)

[pinheadmz]

I still don't think we need to maintain our own JSON parsing code like this. I think the issue in #714 is that @click.argument("params", type=str, nargs=-1) returns a List of what it thinks are separate arguments separated by spaces. I'm wondering if its easier to just get sys.argv and try to pass all the bitcoin-cli args as one single string. The commands are executed inside the pod itself , currently like f"bitcoin-cli {method} {params}" so we should be able to rely on bitcoin-cli interpreting the argument string correctly, as long as it is sent to the shell inside the pod correctly.

[b-l-u-e]

@pinheadmz - You were absolutely right! After researching this further. The issue wasn't with JSON parsing itself, but with how arguments were being passed through the shell layers.

What I implemented:

• Removed all custom JSON parsing logic
• Used shlex.quote() to properly shell-escape parameters
• Let bitcoin-cli handle all argument interpretation
• Users must provide arguments in the exact format bitcoin-cli expects

The solution:

# Shell-escape each param to preserve quotes and special characters
bitcoin_cli_args = " ".join(shlex.quote(p) for p in params)

Results:

• ✅ warnet bitcoin rpc tank-0001 logging '["http","rpc"]' - works perfectly
• ❌ warnet bitcoin rpc tank-0000 getblock '["blockhash"]' - fails (as expected, user must use getblock blockhash)
• ❌ warnet bitcoin rpc tank-0001 logging '[\"http\"]' - fails (as expected, user must use proper JSON)

This eliminates all method-specific logic while ensuring proper quoting through kubectl exec to bitcoin-cli. Users now provide arguments in the exact format that bitcoin-cli expects, which is much simpler and more maintainable.

Thanks for the guidance.

[pinheadmz]

The signet test is failing. I ran it locally with extra logging and saw this:

2025-07-08 07:44:45 | DEBUG   | test     | Executing warnet command: bitcoin rpc miner importdescriptors '[{"desc": "wpkh(tprv8ZgxMBicQKsPfH87iaMtrpzTkWiyFDW7SVWqfsKAhtyEBEqMV6ctPdtc5pNrb2FpSmPcDe8NrxEouUnWj1ud7LT1X1hB1XHKAgB2Z5Z4u2s/84h/1h/0h/0/*)#5j6mshps", "timestamp": 0, "active": true, "internal": false, "range": [0, 999], "next": 0, "next_index": 0}, {"desc": "wpkh(tprv8ZgxMBicQKsPfH87iaMtrpzTkWiyFDW7SVWqfsKAhtyEBEqMV6ctPdtc5pNrb2FpSmPcDe8NrxEouUnWj1ud7LT1X1hB1XHKAgB2Z5Z4u2s/84h/1h/0h/1/*)#9xl6dz3g", "timestamp": 0, "active": true, "internal": true, "range": [0, 999], "next": 0, "next_index": 0}]'
2025-07-08 07:44:45 | INFO    | test     | importdescriptors response:
error: Error parsing JSON: '[{"desc":
command terminated with exit code 1

extra logging added like this:

diff --git a/test/signet_test.py b/test/signet_test.py
index 93fa2e48..82de5f92 100755
--- a/test/signet_test.py
+++ b/test/signet_test.py
@@ -32,10 +32,12 @@ class SignetTest(TestBase):
         self.wait_for_all_edges()
 
     def check_signet_miner(self):
-        self.warnet("bitcoin rpc miner createwallet miner")
-        self.warnet(
+        createwallet = self.warnet("bitcoin rpc miner createwallet miner")
+        self.log.info("createwallet response:\n" + createwallet)
+        importdescriptors = self.warnet(
             f"bitcoin rpc miner importdescriptors '{json.dumps(self.signer_data['descriptors'])}'"
         )
+        self.log.info("importdescriptors response:\n" + importdescriptors)
         self.warnet(
             f"run resources/scenarios/signet_miner.py --tank=0 generate --max-blocks=8 --min-nbits --address={self.signer_data['address']['address']}"
         )

[pinheadmz]

Ok this is getting better but I can still break it :-) We might want to just start adding all these edge cases to a test, including the long descriptors one that fails (assert that it fails) just to keep track of what works and what doesn't:

RPC send accepts a json argument and then a few non-json arguments like conf_target and estimate_mode.

So this works inside the container: (expected error case)

$ bitcoin-cli send '[{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}]' 1

error code: -8
error message:
Specify estimate_mode

but fails from warnet:

$ warnet bitcoin rpc tank-0027 send '[{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}]' 1
error: Error parsing JSON: [{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}] 1
command terminated with exit code 1

Succeeds in container (proper syntax):

$ bitcoin-cli send '[{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}]' 1 "economical"
{
  "txid": "cd029354dfab6c1b17dd9cf8a96da56c51e7b02e15eb69215b9356de9bd5a6fc",
  "complete": true
}

fails from warnet:

$ warnet bitcoin rpc tank-0027 send '[{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}]' 1 "economical"
error: Error parsing JSON: [{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}] 1 economical
command terminated with exit code 1

$ warnet bitcoin rpc tank-0027 send '[{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}]' 1 economical
error: Error parsing JSON: [{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}] 1 economical
command terminated with exit code 1


$ warnet bitcoin rpc tank-0027 send '[{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}]' 1 'economical'
error: Error parsing JSON: [{"bcrt1qsrsmr7f77kcxggk99yp2h8yjzv29lxhet4efwn":0.1}] 1 economical
command terminated with exit code 1

What I'd look at next is, the startswith("[") kind of stuff -- a little JSON parsing in warnet code might be ok for certain edge cases but clearly we cant expect the entire params to be one single json object. I wonder if shlex has other methods that can help parse the shell input to warnet before rendering to the shell output in the container ?