Skip to content

universe+tapdb: add new block_height field to universe_leaves use that to implement block height based supply tree queries #1596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 24, 2025

Conversation

Roasbeef
Copy link
Member

In this PR, we add a new method FetchSupplyLeavesByHeight, that can be used to query for the leaves of a supply tree by height. This will be useful for writing the new syncing state machine and the sub-system that serves the supply tree syncer. Such a syncer would start from the initially created pre-commitment, then fetch all the leaves for the future updates, assemble those into the supply tree, then verify that everything matches up.

Prompts used during development (aider+Gemini 2.5-pro-06-5)

Initial draft creation

Based on by working memory of this new feature, I scanned the relevant files, then made mental nodes of what needed to be updated. I then drafted this fairly detailed prompt that guided aider w.r.t which files to update, and the set of high level changes to make.

#### Today we assemble the individual updates (burn, ignore, mint) for a given asset  
#### specifier (group key is what we care about) into distinct trees for each type.  
#### We then further aggregrate each of those roots into a unified supply tree for  
#### the entire asset specifier. This tree commits to a root sum of the supply (you  
#### can subtract out any of the roots from the root sum to derive a snapshot of the  
#### supply such as the amt burnt, etc).  
####   
#### We then also have a system to assemble these updates, then commit then in a  
#### batch, this exists in: tapdb/supply_commit.go and tapdb/supply_tree.go.  
####   
#### For each of the supply sub-trees, we actually re-use the existing  
#### universe_leaves table to point to each of the respective leaves.  
####   
#### In this task, we want to add a block height to this leaf, as this'll allow us  
#### to scan the leaves in order, based on the type, so we can send out the set of  
#### leaves added in a state transition to a party that wants to sync the set of  
#### leaves.  
####   
####   
#### For this, we may need to update the set of queries in:  
#### tapdb/sqlc/queries/universe.sql. Of interest is also the  
#### applySupplyUpdatesInternal helper function, as we'll want to make sure that the  
#### helper functions that it calls properly inserts the height into the ultimately  
#### universe leaf. For the mint and burn types, we can obtain this height from the  
#### issuance proof.  
####   
#### For the ignore type, we'll need to add a block height field to  
#### universe.IgnoreTuple (universe/ignore_records.go).  
####   
#### For the leaves, we'll likely need to add a new migration (38) which adds the  
#### new height field (can be NULL) to the universe_leaves table.  
####   
#### We'll likely also want to add a new BlockHeight() method to the  
#### SupplyUpdateEvent interface.  
####   
#### Next, we'll want a new method on the SupplyTreeStore that given a start+end  
#### height and a specifier, the set of updates whose universe_leaves have a block  
#### height between that param. This may need to update  
#### tapdb/sqlc/queries/supply_tree.sql.  
####   
#### Also see tapdb/querier.go.  
####   
#### Ruminate on my sketch above, then let's create an execution plan remembering  
#### the ultimate goal. From there we'll implement each portion.  

Subsequent iterations

From here I made an edit or two to get things compiling. Then I executed a series of prompts to refine the code, refactor slightly, then add unit tests.

inside multiverse.UpsertProof Leaf, let's use SparseDecode instead along with BlockHeightRecord as we don't need to decode the entire proof just to get the block height

let's use the same sparse decode in BaseUniverseStore.UpsertProofLeaf

let's do the same for SupplyTreeStore.RegisterMintSupply (sparse decode)

can we morph that into a function closure invocation that itself returns lfn.Option[uint32]? so we don't leak that intermediate variable to the higher scope

great! but now we have a bit of duplication, can we make a helper function can call that intead of having the same duplication everywhere? we can put the helper function in tapdb/sqlutils.go

nice! now let's extend the unit tests in tapdb/supply_tree_test.go to cover the new FetchSupplyLeavesByHeight method.

I messed when committing and dropped a commit entirely (lol), so I had aider re-add it:

help me ree-add QuerySupplyLeavesByHeight back to tapdb/sqlc/queries/supply_tree.sql, I deleted it accidentally

@Roasbeef Roasbeef added enhancement New feature or request commitments supply commit Work on the supply commitment feature, enabling issuers to attest to total asset supply on-chain. labels Jun 13, 2025
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 3e7ed65 to c57d19d Compare June 18, 2025 01:51
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from c2c394f to b1173b9 Compare June 18, 2025 01:51
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from c57d19d to bd5d0b4 Compare June 18, 2025 02:06
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from b1173b9 to c6f9962 Compare June 18, 2025 02:06
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from bd5d0b4 to 37f23fa Compare June 18, 2025 23:02
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from c6f9962 to da0df38 Compare June 18, 2025 23:02
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 37f23fa to be0a52c Compare June 18, 2025 23:25
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch 2 times, most recently from ee045a1 to 598067d Compare June 18, 2025 23:51
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from be0a52c to f2a69aa Compare June 18, 2025 23:57
@levmi levmi moved this from 🆕 New to 🏗 In progress in Taproot-Assets Project Board Jun 19, 2025
@levmi levmi moved this from 🏗 In progress to 👀 In review in Taproot-Assets Project Board Jun 19, 2025
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from f2a69aa to 93ed469 Compare June 20, 2025 23:49
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from 598067d to 9f03d81 Compare June 20, 2025 23:49
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 93ed469 to 633373a Compare June 21, 2025 00:08
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from 9f03d81 to e77d0cf Compare June 21, 2025 00:08
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 633373a to 3ca1517 Compare June 21, 2025 00:12
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from e77d0cf to 3044a87 Compare June 21, 2025 00:12
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 3ca1517 to 988837b Compare June 21, 2025 00:20
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from 3044a87 to 9854971 Compare June 21, 2025 00:20
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 988837b to a9ef139 Compare June 21, 2025 01:22
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from 9854971 to 37bdcfa Compare June 21, 2025 01:22
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from a9ef139 to ca7db9a Compare June 25, 2025 00:27
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 16f9f9a to 059ed3f Compare June 27, 2025 00:08
@Roasbeef Roasbeef force-pushed the supply-commit-machine branch from 99529ae to 87ba694 Compare June 27, 2025 00:08
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 059ed3f to 5025c8c Compare July 3, 2025 01:45
@Roasbeef Roasbeef changed the base branch from supply-commit-machine to main July 3, 2025 01:45
@Roasbeef
Copy link
Member Author

Roasbeef commented Jul 3, 2025

Rebased.

@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 5025c8c to acafde1 Compare July 3, 2025 02:20
@coveralls
Copy link

coveralls commented Jul 3, 2025

Pull Request Test Coverage Report for Build 16428331342

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 159 of 215 (73.95%) changed or added relevant lines in 10 files are covered.
  • 61 unchanged lines in 15 files lost coverage.
  • Overall coverage increased (+0.01%) to 56.472%

Changes Missing Coverage Covered Lines Changed/Added Lines %
tapdb/sqlc/universe.sql.go 1 2 50.0%
tapdb/universe.go 18 20 90.0%
tapdb/sqlutils.go 11 14 78.57%
universe/supplycommit/states.go 8 11 72.73%
tapdb/multiverse.go 12 16 75.0%
universe/ignore_records.go 9 13 69.23%
tapdb/sqlc/supply_tree.sql.go 19 27 70.37%
tapdb/supply_tree.go 62 93 66.67%
Files with Coverage Reduction New Missed Lines %
fn/context_guard.go 1 88.71%
asset/asset.go 2 79.23%
rfqmsg/records.go 2 70.25%
tapdb/multiverse.go 2 82.48%
tapdb/sqlc/assets.sql.go 2 74.29%
universe_rpc_diff.go 2 76.0%
universe/syncer.go 2 86.63%
tapdb/sqlerrors.go 3 67.86%
tapdb/assets_common.go 4 78.19%
tapdb/mssmt.go 4 90.45%
Totals Coverage Status
Change from base Build 16427162624: 0.01%
Covered Lines: 57863
Relevant Lines: 102463

💛 - Coveralls

@Roasbeef Roasbeef requested a review from ffranr July 3, 2025 20:07
@lightninglabs-deploy
Copy link

@ffranr: review reminder

@ffranr ffranr requested a review from guggero July 10, 2025 14:21
Copy link
Member

@guggero guggero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Got a couple of suggestions and the linter isn't fully happy yet. Other than that LGTM 🎉


readTx := NewBaseUniverseReadTx()
dbErr := s.db.ExecTx(ctx, &readTx, func(db BaseUniverseStore) error {
for _, treeType := range []supplycommit.SupplySubTree{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: feels like this would make a good package-level variable with the list of tree types we often iterate over? Would also help a bit with the formatting/readability.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, added.

Copy link
Contributor

@ffranr ffranr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty much there. One question on SQL query re-use.

Comment on lines +46 to +57
-- name: QuerySupplyLeavesByHeight :many
SELECT
leaves.script_key_bytes,
gen.gen_asset_id,
nodes.value AS supply_leaf_bytes,
nodes.sum AS sum_amt,
gen.asset_id,
leaves.block_height
FROM universe_leaves AS leaves
JOIN mssmt_nodes AS nodes
ON leaves.leaf_node_key = nodes.key
AND leaves.leaf_node_namespace = nodes.namespace
Copy link
Contributor

@ffranr ffranr Jul 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a where clause in QueryUniverseSupplyLeaves. Instead of adding a new query here, can't we just extend QueryUniverseSupplyLeaves with an optional block height?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good eye. These queries differ slightly though: the above is based on always knowing the root_id to pass in (hence the join with universe supply roots). Here we query based on just the namespace, so we can skip the extra join on the root.

Roasbeef added 4 commits July 21, 2025 15:38
In this commit, we add a migration to add a block_height to the universe
leaf table. We'll use this in the future to be able to allow clients to
sync block by block for the new supply universe trees.
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from acafde1 to beb58bb Compare July 21, 2025 20:51
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from beb58bb to 3ff47a4 Compare July 21, 2025 21:03
In this commit, we add a new method to the SupplyTreeStore that is able
to read out the leaves of a supply tree based on a start and end height.
This will be useful for writing the new syncing state machine and the
sub-system that serves the supply tree syncer.
@Roasbeef Roasbeef force-pushed the supply-leaf-height-query branch from 3ff47a4 to 4e7a385 Compare July 21, 2025 21:15
@Roasbeef Roasbeef requested review from guggero and ffranr July 23, 2025 18:59
@Roasbeef
Copy link
Member Author

This should be ready for a final pass. The unit test failure is being addressed in another PR to improve the set of property based tests.

Copy link
Member

@guggero guggero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎉

Copy link
Contributor

@ffranr ffranr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ffranr ffranr added this pull request to the merge queue Jul 24, 2025
Merged via the queue into main with commit f1db7c9 Jul 24, 2025
33 of 36 checks passed
@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in Taproot-Assets Project Board Jul 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
commitments enhancement New feature or request supply commit Work on the supply commitment feature, enabling issuers to attest to total asset supply on-chain.
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

5 participants