diff --git a/eip-0047.md b/eip-0047.md new file mode 100644 index 00000000..b14dc468 --- /dev/null +++ b/eip-0047.md @@ -0,0 +1,698 @@ +# EIP-0048: Pooled Transaction Inputs + +- Author: aslesarenko +- Status: Proposed +- Created: 07.07.2024 +- Last edited: 07.07.2024 +- License: CC0 +- Forking: soft-fork required + +## Contents +- [Background and Motivation](#background-and-motivation) + - [The Problem](#the-problem) + - [Illustrative Example](#illustrative-example) + - [The Solution Idea](#the-solution-idea) +- [Proposed Solution](#proposed-solution) + - [Key Components of the Solution](#key-components-of-the-solution) + - [Double-Spending Problem with PooledInputs](#double-spending-problem-with-pooledinputs) + - [Solution to Double-Spending Problem](#solution-to-double-spending-problem) + - [Wallet Standard for PooledInputs](#wallet-standard-for-pooledinputs) +- [Example: Alice's Multi-Step DeFi Workflow](#example-alices-multi-step-defi-workflow) + - [Context](#context) + - [Current Step-by-Step Workflow with explicit UTXO selection](#current-step-by-step-workflow-with-explicit-utxo-selection) + - [New step-by-step Workflow with Pooled Inputs](#new-step-by-step-workflow-with-pooled-inputs) +- [Implementation Details](#implementation-details) + - [PooledInput Data Structure](#pooledinput-data-structure) + - [Transaction Format Extension](#transaction-format-extension) + - [Validation and Processing Logic](#validation-and-processing-logic) + - [Deterministic Box Selection](#deterministic-box-selection) + - [Ordered access to UTXO Storage](#ordered-access-to-utxo-storage) + - [Obtaining UTXO Ordering Details](#obtaining-utxo-ordering-details) + - [Adding and Removing UTXOs](#adding-and-removing-utxos) + - [Extension of UTXO Storage](#extension-of-utxo-storage) + - [Selecting UTXOs for Unpooling](#selecting-utxos-for-unpooling) + - [Change Output Handling](#change-output-handling) + - [Transaction Concretization](#transaction-concretization) +- [Benefits of Pooled Transaction Inputs](#benefits-of-pooled-transaction-inputs) +- [Related Work](#related-work) +- [Conclusion](#conclusion) + +## Background and Motivation + +In the current implementation of the Ergo blockchain, each transaction spends a list of +UTXOs (called Boxes) given by their boxId. These input boxes need to be selected by an +off-chain selector, which is part of the transaction builder in the wallet. Once the +transaction is built and signed, these concrete Boxes are set in stone and cannot be +changed without rebuilding the transaction and re-signing it again. + +### The Problem + +This creates challenges when multiple transactions need to be created from the same wallet +in short period of time in response to external events (e.g. when participating in +multiple DeFi protocols). Two transactions cannot share the same input box, so the wallet +must track which boxes are already spent and which are not, selecting non-overlapping +subsets of UTXOs. + +However, this is not always feasible, especially when UTXOs are consolidated into a single +box, resulting in a situation where only one transaction can be created from that single +UTXO and other transactions has to be chained. The chaining by itself creates undesirable +dependency on the parent transaction so that cancellation/delaying/queueing of the parent +transaction will also cancel/delay/queue the child transaction. + +This problem is exacerbated in scenarios requiring responsiveness. For example, if a user +participates in two or more DeFi protocols involving some form of transaction queueing and +needs to react to multiple events quickly, user cannot chain transactions belonging to +different DeFi protocols (to avoid dependencies), but has to submit them simultaneously in +response to the events. In such cases, the consolidation of UTXOs can become a bottleneck, +limiting the number of transactions that can be executed in parallel without undesirable +chaining. + +### Illustrative Example + +Consider Alice, a user who actively participates in multiple DeFi protocols on the Ergo +blockchain. Each protocol requires Alice to submit transactions in response to various +events, such as staking rewards, yield farming updates, or automated trading triggers. +These interactions need to be executed promptly to maximize her returns and maintain her +positions, which can lead to multiple simultaneous, but independent transactions. + +Scenario: + - Alice stakes some ERGs in a DeFi staking pool, which periodically pays out rewards. + - She also participates in a yield farming protocol, which requires her to move funds + between different liquidity pools based on real-time market conditions. + - Additionally, Alice uses an automated trading bot that submits transactions based on + market signals. + +Each of these DeFi protocols requires Alice to submit transactions spending from her +wallet in response to the protocol-specific events. However, chaining of these transactions +together is undesirable as they are built independently by different dApps. If Alice's wallet has +consolidated its funds into a single UTXO for storage efficiency, she can only process one +transaction at a time, leading to delays and potentially missed opportunities. +Alternatively, Alice can chain transactions, dealing with undesirable dependencies and +risks of cancellation/delaying/queueing of the parent transaction affecting the whole chain. + +Alternatively, Alice can de-consolidate her UTXOs, but this additional step will worsen +user experience and increase time and cost of transactions. Even if implemented by a wallet, +such add-hoc consolidation/de-consolidation once in a while will bottleneck Alice in her +DeFi experience. Moreover, not being implemented on network protocol level, automatic +de-consolidation has to be enforced across all wallets to enable consistent user experience. + +### The Solution Idea + +To address these challenges, we propose the introduction of _Pooled Transaction Inputs_. +This new mechanism allows transactions to specify input pools from which UTXOs can be +selected dynamically, rather than pre-selecting specific UTXOs. By introducing a new data +structure called `PooledInput`, we enable transactions to define _inputs to spend_ without +specifying concrete UTXOs at the time of transaction creation. + +With `PooledInput`, Alice can define her transactions to spend from a pool of UTXOs +associated with her wallet's address, specifying the amount to be spent. This way, +multiple transactions can be created by different dApps and processed in parallel, each +spending from the same pool of UTXOs and without pre-determined boxIds. The Ergo node will +select (or _unpool_) the appropriate UTXOs for spending during transaction processing, +ensuring deterministic and conflict-free execution on L1 protocol level. + +In Alice's case, this means she can handle staking rewards, yield farming updates, and +automated trades simultaneously, improving her DeFi interaction efficiency and +responsiveness. This solution allows her to create transactions simultaneously across +multiple dApps without the risk of transaction conflicts even in the +presence of queuing, delays or cancellations at individual applications. + +By implementing Pooled Transaction Inputs, we aim to enhance the flexibility and +efficiency of UTXO management in Ergo, supporting higher throughput and more responsive +user experience especially in the context of cross-DeFi interactions. More on this in the +Benefits section. + +## Proposed Solution + +To address the challenges identified in the Background and Motivation section, we propose +the introduction of a new transaction input type called `PooledInput`. This new data +structure and its associated protocol modifications will enable transactions to +dynamically select UTXOs from a specified pool, enhancing the flexibility and efficiency +of transaction creation and execution on the Ergo blockchain. + +### Key Components of the Solution + +1. **PooledInput Data Structure** which contains the following fields: + - `address: SigmaProp`: Specifies the address from which UTXOs should be selected. + - `value: Long`: The total value to be spent from the pool. + +2. **Transaction Format Extension:** + - Extend the existing transaction format to include a new field `pools: List[PooledInput]`, + which is a list of `PooledInput`. + +3. **Validation and Processing Logic:** + - Modify the Ergo node to support the new transaction format with the `pools` field. + - During transaction processing, the Ergo node will select UTXOs from the specified + address pool and have a total value greater than or equal to the `value` field of the + `PooledInput`. + +4. **Deterministic Box Selection:** + - Implement a deterministic selection algorithm to ensure all nodes select the same UTXOs + from the pool. This can be achieved by using FIFO selector [1], i.e. selecting the + pool boxes by their order of inclusion into the UTXO set (which is defined by the + global order of transactions in the blockchain). The boxes from the ordered pool + are selected until the total value meets or exceeds the required `value`. + +5. **LevelDB Integration for Efficient Box Selection:** + + - To efficiently manage UTXOs and ensure deterministic box selection, we propose leveraging + LevelDB's ordered key-value storage. This approach maintains UTXOs in an ordered fashion + based on `(address, blockHeight, outputIndex)`, where the value stored is the `boxId` of + the UTXO: + - Record Key: `(address, blockHeight, outputIndex)` formatted as a string to maintain order. + - Record Value: `boxId` stored as bytes. + - New Database Instance `indexDb` to store the ordered keys with the UTXO `boxId` as the value. + +6. **Change Output Handling:** + - After selecting the required UTXOs, the change output will be added with + the remaining value (total value of selected UTXOs minus the `value` field of the + `PooledInput`). This change output will be appended to the list of transaction + outputs and the change amount will be returned to the address associated with the pool and + thus can be consumed by the next transaction spending from the same pool (respecting FIFO + ordering). + +7. **Transaction Concretization:** + - Introduce the process of `input unpooling` during block + validation. This involves determining the exact UTXOs (concrete boxes) to be spent based + on the state of the UTXO set at the transaction's position in the block. This also + includes adding the change output to the transaction outputs. + +8. **Support for Multiple PooledInputs:** + - Allow transactions to include multiple `PooledInputs` in the `pools` list. The unpooling + process will handle each `PooledInput` sequentially, in the order they appear in the + list. + +### Double-Spending Problem with PooledInputs + +While PooledInputs offer flexibility, they also introduce a potential risk (if +not addressed) of double-spending. Specifically, when a transaction contains only +PooledInputs, it can be submitted multiple times to the blockchain through different +network nodes, potentially spending from the same UTXO pools. This is possible because the +`messageToSign` for different submissions of the same transaction will be the same, making +the signature valid for each attempt. This loophole can be exploited for double-spending +attacks. + +The classical solution to this problem involves incorporating a _nonce_ in the +`messageToSign` that increments with each transaction submission. However, this approach +is impractical for UTXO blockchains due to several reasons: +- The nonce is not part of the UTXO. +- There is no place to store the previous nonce in the UTXO state. +- Introducing a nonce would require significant changes to the UTXO model. +- The nonce would need to be persistently stored, adding inefficiency and requiring additional storage. +- Each transaction submission would require updating the nonce, which introduces a shared state. +- Wallet software would need to track the nonce for each UTXO pool, adding complexity. +- The transaction format and signing process would need to change to include the nonce, + necessitating wallet software updates. + +In this proposal, instead of using nonce, we address this issue by imposing a protocol +level constraint on the usage of PooledInputs and describe a wallet level convention for +UTXO management that will make this proposal both safe and also practical to implement. +More on this in the Proposed Solution section. + +### Solution to Double-Spending Problem + +To address the possibility of double-spending attacks with PooledInputs, we propose a +constraint: _a transaction with PooledInputs must include at least one concrete input_. +This ensures that each transaction has a unique piece of data in its `messageToSign` +(concrete box id which is unique), preventing duplicate submissions. The Ergo node will +enforce this constraint during transaction validation. + +However, in many practical use cases, transactions with PooledInputs may not naturally have +concrete inputs, as seen in the examples described earlier. To address this, we propose a +convention for UTXO management supported by wallet software, which can be regarded as a +wallet standard (can be published as separate EIP motivated by this proposal). + +### Wallet Standard for PooledInputs + +1. **PoolAccessBox**: + - When creating a transaction with PooledInputs (when there is no concrete input), + the wallet should include a dummy input box called `PoolAccessBox` with a minimal value + (pre-defined constant `PoolAccessBoxValue` of nanoERG). This box is spendable from the + wallet's address and serves solely as a placeholder to ensure the transaction has at least + one concrete input and thus can have pooled inputs (have access to the _input pools_ + feature). + +2. **Maintaining PoolAccessBoxes**: + - Every transaction spending `PoolAccessBox` should also create a new one in its outputs. + This maintains a continuous supply of `PoolAccessBoxes` for future transactions and + preserves the input/output balance. + - Initially, the wallet software should create a sufficient number of `PoolAccessBoxes` to + enable the PooledInputs feature. Subsequent management of these boxes will be + handled internally by the wallet, ensuring user-friendly operation. + +3. **User Experience**: + - The wallet can prompt the user to enable the "Pooled Inputs" feature once enough ERGs + are available, creating the necessary `PoolAccessBoxes` in a single transaction. + - To distinguish `PoolAccessBoxes` from other boxes, they can have a specific register + value (e.g., `R4 = 0x012345`). The wallet can identify available `PoolAccessBoxes` + using (wallet's address, `R4 = 0x012345`, value == `PoolAccessBoxValue`). + +## Example: Alice's Multi-Step DeFi Workflow +Let's create a detailed example involving Alice's participation in multiple DeFi +protocols, requiring her to sign transactions and manage her UTXOs efficiently using the +Pooled Transaction Inputs mechanism. + +### Context +Alice is an active participant in the DeFi ecosystem on the Ergo blockchain. She engages +in multiple protocols, including staking, yield farming, and automated trading. She needs +to handle these interactions efficiently to maximize her returns. + +### Current Step-by-Step Workflow with explicit UTXO selection + +1. **Initial Setup:** + - **Wallet:** Alice's wallet contains the following UTXOs: + UTXO1: 100 ERG + +2. **Staking:** + - **Event:** Alice decides to stake 40 ERG in a DeFi staking pool that periodically pays out rewards. + - **Transaction 1:** Alice creates a transaction to stake 40 ERG. + - **Inputs:** UTXO1 (100 ERG) + - **Outputs:** + - StakeOutput: 40 ERG to the staking pool + - ChangeOutput: 60 ERG back to Alice's wallet + - Wallet: UTXO2: 60 ERG (unconfirmed) + +3. **Yield Farming:** + - **Event:** Alice wants to participate in a yield farming protocol that requires 35 ERG. + - **Transaction 2:** Alice creates chained transaction to deposit 35 ERG into the yield farming pool + referencing the change output from the previous transaction. + - **Inputs:** UTXO2 (60 ERG) (unconfirmed) + - **Outputs:** + - YieldFarmOutput: 35 ERG to the yield farming pool + - ChangeOutput: 25 ERG back to Alice's wallet + - Wallet: UTXO3: 25 ERG (unconfirmed) + +4. **Automated Trading:** + - **Event:** Alice's automated trading bot detects a market signal and needs to execute a trade with 20 ERG. + - **Transaction 3:** The bot creates a transaction to execute the trade. + - **Inputs:** UTXO3 (25 ERG) (unconfirmed) + - **Outputs:** + - TradeOutput: 20 ERG to the trading pool + - ChangeOutput: 5 ERG back to Alice's wallet + - Wallet: UTXO4: 5 ERG (unconfirmed) + +Note, in order for Alice to execute the above transaction at the same time, they all need to be +chained together. This creates hard dependence between the transactions and requires all +the previous (parent) transactions to be successful for child transactions to also be +successful. This rules out any possibility for cancellation or transaction queueing on +DeFi protocol side. If Transaction 1 is cancelled/delayed by the DeFi protocol, then +the whole chain of transactions will be also cancelled/delayed. + +### New step-by-step Workflow with Pooled Inputs + +Note: +- to ensure the security and uniqueness of transactions involving PooledInputs, + Alice's wallet will include a `PoolAccessBox` in each transaction. +- many access boxes can be created initially to enable PooledInputs feature in the wallet +- the number of access boxes defines the maximum number of transactions that can be created + in parallel without using chaining. + +1. **Initial Setup**: + - **Wallet**: Alice's wallet contains the following UTXOs: + - UTXO1: 100 ERG + - PoolAccessBox1: 0.001 ERG (pre-defined constant `PoolAccessBoxValue`) + - PoolAccessBox2: 0.001 ERG + - PoolAccessBox3: 0.001 ERG + +2. **Staking**: + - **Transaction 1**: Alice creates a transaction to stake 40 ERG. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 40 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolAccessBox1 (0.001 ERG) + - **Outputs**: + - StakeOutput: 40 ERG to the staking pool + - PoolAccessBox4: 0.001 ERG to Alice's wallet + +3. **Yield Farming**: + - **Transaction 2**: Alice creates a transaction to deposit 35 ERG into the yield farming pool. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 35 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolAccessBox2 (0.001 ERG) + - **Outputs**: + - YieldFarmOutput: 35 ERG to the yield farming pool + - PoolAccessBox5: 0.001 ERG to Alice's wallet + +4. **Automated Trading**: + - **Transaction 3**: The bot creates a transaction to execute a trade with 20 ERG. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 20 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolAccessBox3 (0.001 ERG) + - **Outputs**: + - TradeOutput: 20 ERG to the trading pool + - PoolAccessBox6: 0.001 ERG to Alice's wallet + - Wallet: UTXO: 5 ERG; PAB4: 0.001 ERG; PAB5: 0.001 ERG; PAB6: 0.001 ERG (unconfirmed) + +This workflow ensures that each transaction has a unique `messageToSign`, preventing +double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. +All transaction are independent and can be submitted simultaneously without the risk of +cancellation/delaying/queueing of the parent transaction affecting them all. + +## Implementation Details + +### PooledInput Data Structure + The `PooledInput` structure can be defined in Scala as follows: + ```scala + case class PooledInput( + address: SigmaProp, // defines the pool where UTXOs are selected from + value: Long, // Total value to be spent from the pool + signature: Array[Byte] // Signature for the `address` sigma proposition proof + ) + ``` + - `SigmaProp` type: aka Sigma Proposition, specifies the source address of the UTXOs (in + its simplest form it is P2PK, but can also be multisig and more complex [sigma trees + supported by Ergo](https://hackernoon.com/sigma-protocols-for-the-working-programmer)). + - `value`: The total amount of ERGs to be covered by inputs spent from the pool. + - `signature`: The signature for the `address` sigma proposition, ensuring the transaction + is authorized to spend from the pool. + + The `address` field defines the UTXO _pool_ as a list of boxes as if returned by the following query: + ```sql + SELECT * FROM UTXO as U + WHERE U.ergoTree = ErgoTree.fromSigmaProp(address) + ``` +The ordering of the boxes in the pool is defined below. + +### Transaction Format Extension + Extend the transaction format to include the `pools` field: + ```scala + case class ErgoTransaction( + inputs: Seq[Input], // List of concrete inputs to spend + dataInputs: Seq[DataInput], // List of data inputs (readonly or reference inputs) + pools: Seq[PooledInput], // List of pooled inputs (UTXO pools) to spend from + outputs: Seq[BoxCandidate] // List of output boxes (not including change outputs created for each pooled input) + ) + ``` + - `pools`: A list of `PooledInput` that each specify the list of UTXOs to be spent from + each pool during transaction processing. + - When the `pools` is empty (i.e., no pooled inputs), the transaction behaves as a + standard transaction with concrete inputs and data inputs. + - When the `pools` list contain a simple P2PK address, then the corresponding signature + should be present in the `signature` field of the `PooledInput`. + - When the `pools` list contain addresses belonging to different wallets, +then properly generated signatures has to be provided in the `signature` fields of the +corresponding `PooledInput`s. + - Technically, the format versioning can be done by reusing one of the counts (e.g. data + input count). When the count is greater than MaxArrayLength protocol constant + (e.g. 100000) then it means the transaction contains `pools` field, otherwise the list is empty, + which means the transaction is indistinguishable from the current protocol version. This + is similar to how versioning is done in BlockTransactionsSerializer. + +### Validation and Processing Logic + + During transaction validation, the Ergo node will: + - **Get the list of PooledInputs**: + ```scala + val pooledInputs = transaction.pools + ``` + - **For each `PooledInput`, select UTXOs** from the specified address that sum up to the required value: + ```scala + pooledInputs.foreach { pooledInput => + val selectedUtxos = selectUtxos(pooledInput.address, pooledInput.value) + transaction.inputs.appendAll(selectedUtxos.map(createInput)) + } + ``` + - `selectUtxos(address, value)`: Selects UTXOs from the pool (by the address) until the total + value meets or exceeds the specified value (see the `def selectUtxos` below). + +### Deterministic Box Selection + +To ensure deterministic and efficient selection of UTXOs for a given `PooledInput`, we use +LevelDB to efficiently (i.e. logarithmic time) store and retrieve UTXOs in an ordered manner +(using lexically ordered keys). This is an implementation detail and just one of the +possible ways to implement simple FIFO selection of UTXOs [1] on-chain, as part of Layer 1 +protocol. + +#### Ordered access to UTXO Storage + +Each UTXO is stored with a key consisting of `(address, blockHeight, outputIndex)` and the +`boxId` as the value. The UTXO data itself is stored separately in the +`PersistentBatchAVLProver` (field in `UtxoState`) with the `boxId` as the key. + +The following method can be used to create the ordered key for a given UTXO: +```scala +def createKey(address: String, blockHeight: Long, outputIndex: Int): String = { + f"$address%s-$blockHeight%010d-$outputIndex%05d" +} +``` +Thus, the keys are ordered first by `address`, then by `blockHeight`, and finally by `outputIndex`. +Such defined keys will allow to create an iterator and seek to the first UTXO with the +given address (see below). + +#### Obtaining UTXO Ordering Details + +To support the deterministic selection of UTXOs from a pool, we need to obtain for each UTXO +the additional information about its address and ordering. Specifically, we need to compute +`address`, `blockHeight` and `outputIndex` for each UTXO in order to be able to determine +the order of UTXOs in the pool. + +```scala +case class UtxoInfo(addressOpt: Option[String], blockHeight: Int, outputIndex: Int) +``` + +- `addressOpt`: This is the optional pool's address associated with the UTXO. It is + optional because not all UTXOs can be related to a pool (i.e. most boxes with PayToScript + addresses). + +- `blockHeight`: This is calculated as the height of the block where UTXO is created. + +- `outputIndex`: This is calculated as the index of the output in the list of outputs of + the full block that created the UTXO. + +These fields can be calculated in the `UtxoState.stateChanges` method during the processing +of the full block (see below). + +#### Adding and Removing UTXOs + +UTXOs are added to and removed from both the `indexDb` and the `PersistentBatchAVLProver`. +The logic is as follows: +1. Block is processed and a list of state changes is obtained. Each state change is a pair + of `(Operation, UtxoInfo)`, where `Operation` is either `Insert` or `Remove` and + `UtxoInfo` is a case class containing the UTXO details, which can be unambiguously + obtained during the block processing. + +```scala +val stateChanges = ErgoState.stateChanges(block) // compute state changes for the block +``` + +2. Once the block is validated and the state changes are obtained, they are processed as + follows: + +```scala +stateChanges.foreach { + case (Insert(boxId, boxBytes), utxo) => + utxo.addressOpt match { + case Some(address) => + // When UTXO can be related to a pool, add it to the indexDb + val key = createKey(address, utxo.blockHeight, utxo.outputIndex) + indexDb.put(key, utxo.boxId) + case _ => + } + // add UTXO to the persistent storage (existing code of the current version) + persistentProver.performOneOperation(Insert(boxId, boxBytes)) + case (Remove(boxId), utxo) => + utxo.addressOpt match { + case Some(address) => + // When UTXO is spent from a pool, remove it from the indexDb + val key = createKey(address, utxo.blockHeight, utxo.outputIndex) + indexDb.delete(key) + case _ => + } + // remove UTXO from the persistent storage (existing code of the current version) + persistentProver.performOneOperation(Remove(boxId)) +} +``` + +#### Extension of UTXO Storage + +To efficiently obtain UtxoInfo for each UTXO that is being spent, we need to extend the UTXO +storage with additional information. Specifically, we need to store the +`blockHeight` and `outputIndex` for each new UTXO that is created. + +These fields can be retrieved later in the `UtxoState.stateChanges` method for each input +box, during the processing of a full block. + +To store this pair of parameters, the `value` field of the `Insert` operation, which is +submitted to the `persistentProver.performOneOperation` method need to be extended with +the above information (serialized `blockHeight` and `outputIndex`). + +### Selecting UTXOs for Unpooling + +To select UTXOs for a given PooledInput, we use an iterator to retrieve UTXOs in the order +until the required value is met. The logic can be implemented as follows: + +```scala +def selectUtxos(address: String, requiredValue: Long): Seq[UTXO] = { + val iter = indexDb.iterator() + val prefix = f"$address%s-" + iter.seek(prefix.getBytes) // position the iterator at the first UTXO with the given address + var selectedUtxos = Seq.empty[UTXO] + var totalSelectedValue = 0L + + while (iter.hasNext && totalSelectedValue < requiredValue) { + val entry = iter.next() + val keyStr = new String(entry.getKey) + if (keyStr.startsWith(prefix)) { + val boxId = new String(entry.getValue) + val utxoBytes = persistentProver.unauthenticatedLookup(boxId).get + val utxo = deserialize(utxoBytes) + + selectedUtxos = selectedUtxos :+ utxo + totalSelectedValue += utxo.value + } + } + + iter.close() + + if (totalSelectedValue >= requiredValue) { + selectedUtxos + } else { + Seq.empty[UTXO] // Or throw an error if the required value cannot be met + } +} +``` +By leveraging LevelDB's ordered storage, we can implement efficient and deterministic FIFO +selection [1] of UTXOs for unpooling. + +### Change Output Handling + After selecting the UTXOs, an optional change output is created to return the remaining + coins to the pool. This is done as follows: + ```scala + def addChangeOutput(selectedUtxos: Seq[Utxo], pooledInput: PooledInput, transaction: ErgoTransaction): Unit = { + val totalSelectedValue = selectedUtxos.map(_.value).sum + val changeValue = totalSelectedValue - pooledInput.value + if (changeValue > 0) { + val changeOutput = createChangeOutput(changeValue, pooledInput.address) + transaction.outputs.append(changeOutput) + } + } + ``` + - `totalSelectedValue`: Sum of values of the selected UTXOs. + - `changeValue`: Difference between the total selected value and the required value. + - `createChangeOutput(value, address)`: Creates a change output box with the specified + value and address, note the use of the same address as the pool. + +### Transaction Concretization + When a transaction have at least one `PooledInput`, the transaction needs to be + concretized, i.e. the exact UTXOs to be spent need to be determined along with the change + outputs. This process is called `input unpooling` and is done during block application and + validation. The concretization process can be implemented as follows: + ```scala + def concretizeTransaction(tx: ErgoTransaction): Unit = { + tx.pools.flatMap { pooledInput => + val selectedUtxos = selectUtxos(pooledInput.address, pooledInput.value) + val concretizedInputs = selectedUtxos.map(u => createInput(u)) + transaction.inputs.appendAll(concretizedInputs) + addChangeOutput(selectedUtxos, pooledInput, transaction) + } + } + ``` + - `concretizeTransaction`: Resolves `PooledInputs` into concrete inputs + based on FIFO selector and adds change outputs. + - `createInput(utxo)`: Converts a UTXO into a transaction input. + - Note, the method can handle multiple `PooledInput` by sequentially processing each + entry in the `pools` list: + - This ensures that each `PooledInput` is processed in the order they appear, with + deterministic FIFO UTXO selection and conflict-free execution. + +Transaction concretization ensures that each transaction is processed with concrete inputs +as if it was a standard transaction fully constructed and signed off-chain. This also +means PooledInput mechanism is fully transparent to the smart contracts evaluation as the +concretization happens before any contract is executed. + +## Benefits + +Here are the key benefits of the proposed Pooled Transaction Inputs mechanism: +- wallet doesn't need to chain transactions and select UTXOs explicitly +- no need to add change outputs explicitly (the change is handled by the network on the protocol level) +- transactions can be sent in any order (or all at once) and don't depend on each other +- any transaction can be cancelled/delayed/queued without affecting others +- resolves the trade-off between UTXO consolidation and transaction efficiency, it became + possible to consolidate UTXOs to a single box and still be able to participate in multiple + DeFi protocols simultaneously and independently +- combined with atomic transaction chains (see EIP-46) enables atomic transaction + composition, which will allow to avoid transaction conflicts in a systematic and generic + way +- enables complex atomic transaction DAGs to be submitted to the mempool + and then process by the network without requiring wallet to manage the dependencies, or + even be connected + +Here are some of the additional benefits of the proposed protocol level extensions: +- Box.isPoolable property: can be used in contracts to require/forbid specific boxes + (input or output) to be poolable. +- Box.blockHeight property: can be used in contracts to determine exact height when the + box was created (in contrast to creationHeight, which doesn't guarantee the exact height). +- Box.outputIndex property: can be used in contracts to implement ordering logic between + boxes relying on the exact "created before" relation between UTXOs. + +## Related Work + +To answer the question regarding prior work that proposes to move coin selection from +off-chain to on-chain and make it part of Layer 1 protocol, we conducted a search for +relevant publications in the blockchain space. Here are some notable examples and +discussions: + +**Research Papers and Proposals** + + Various academic papers discuss coin selection strategies, focusing on optimizing + transaction costs and sizes. These discussions often highlight the limitations of + off-chain coin selection and suggest improvements, some of which could potentially be + integrated into the protocol layer. For example, the paper "A Survey on Coin Selection + Algorithms in UTXO-based Blockchains" provides a comprehensive review and categorizes + existing algorithms, potentially paving the way for on-chain implementations [1]. + +**UTXO Management and Coin Selection**: + + *MACS Algorithm*: This algorithm, although not explicitly moving coin selection +on-chain, addresses many foundational issues with UTXO management by optimizing +transaction costs and sizes, enhancing privacy, and efficiently managing the UTXO pool. It +considers the age of transactions to promote the spending of older UTXOs, which can be +seen as a step towards more dynamic and protocol-level coin management [2]. + +**Dynamic UTXO Selection**: + + Utreexo by MIT Digital Currency Initiative: Utreexo introduces a dynamic hash-based +accumulator for UTXO management that operates in a more efficient manner compared to +traditional methods. It doesn't explicitly move coin selection on-chain but proposes +significant changes to how UTXOs are managed and verified, indirectly affecting the coin +selection process [3]. + +Despite these advancements, it appears that explicitly moving the entire coin selection +process on-chain as a part of the Layer 1 protocol is still a relatively novel idea. The +existing research and proposals primarily focus on optimizing the current off-chain +processes or introducing hybrid approaches. + +For a deeper dive into the specifics, you can explore the links listed in the References +section. These sources provide foundational insights and potential directions for future +research and implementation in the area of coin selection. + +## Conclusion + +The introduction of Pooled Transaction Inputs and the use of `PoolAccessBox` significantly +streamline the process of managing multiple DeFi simultaneous interactions for users like +Alice. By allowing transactions to specify input pools rather than fixed UTXOs, and +ensuring each transaction includes at least one concrete input, users can engage in +various DeFi protocols simultaneously without chaining transactions or managing UTXO +dependencies. This approach not only enhances transaction flexibility and efficiency, but +also prevents double-spending attacks, enabling a more secure and responsive DeFi +experience on UTXO blockchain platform such as Ergo. + +Furthermore, Pooled Transaction Inputs and `PoolAccessBox` mitigate challenges associated +with UTXO consolidation, allowing users to maximize their participation in the DeFi +ecosystem without transaction processing limitations. This proposal represents a +step forward in the scalability and usability of the Ergo blockchain, paving +the way for more complex and dynamic DeFi applications. + +The implementation of Pooled Transaction Inputs and `PoolAccessBox` as a wallet convention +simplifies UTXO management, ensuring that transactions are both secure and efficient. This +user-friendly approach enhances the DeFi experience on the Ergo blockchain. + +## References +1. [A Survey on Coin Selection Algorithms in UTXO-based Blockchains](https://cs.paperswithcode.com/paper/a-survey-on-coin-selection-algorithms-in-utxo) +2. [MACS: A new approach to multi-asset coin selection](https://cardanofoundation.org/blog/macs-a-new-approach-to-multi-asset-coin-selection) +3. [Utreexo: A dynamic hash-based accumulator optimized for the Bitcoin UTXO set](https://dci.mit.edu/dci-news/2019/6/6/utreexo-a-dynamic-hash-based-accumulator-optimized-for-the-bitcoin-utxo-set)