feat: support horizon

[suchapalaver]

V2 TAP Testing Guide

This guide covers running and debugging V2 (Horizon) TAP functionality in the indexer-rs test environment.

The testing here uses the semiotic-ai/local-network fork based on the horizon branch of edgeandnode/local-network.

Table of Contents

• Prerequisites
• Initial Setup
• Running V2 Tests
• Known Issues and Workarounds
• Understanding the Logs
• Questions for Upstream Teams
• Troubleshooting

Prerequisites

Before running V2 tests, ensure you have:

• Docker and Docker Compose installed
• Rust development environment set up
• just command runner installed
• sqlx-cli installed (cargo install sqlx-cli)
• At least 10GB of free disk space

Initial Setup

0. Set Up Local Development Database (Required for Compilation)

The indexer-rs codebase uses SQLx with compile-time checked queries. You need a local PostgreSQL database for the code to compile (but also see the alternative below):

# Create and start a local PostgreSQL container
docker run \
  --env POSTGRES_HOST_AUTH_METHOD=trust \
  --publish "127.0.0.1:5433":5432 \
  --detach \
  --name postgres_container \
  postgres

# Create .env file with DATABASE_URL
echo "DATABASE_URL=postgres://[email protected]:5433/postgres" > .env

# Wait for PostgreSQL to be ready
docker exec postgres_container pg_isready

# Run migrations

```sqlx migrate run

Alternative: If you want to skip the database requirement, you can use SQLx's offline mode:

export SQLX_OFFLINE=true

This requires that the .sqlx directory with prepared queries is up to date, which if CI is passing they are.

1. Start the Local Test Network

just setup

This command runs setup-test-network.sh which:

• Sets up a local blockchain (chain ID 1337)
• Deploys Graph Protocol contracts (including Horizon V2 contracts)
• Starts graph-node with test subgraphs
• Deploys TAP contracts and aggregator services
• Starts indexer-agent, indexer-service, and tap-agent
• Creates funded escrow accounts

Note: The setup script checks for existing services and skips those already running. If you encounter conflicts:

just down
docker rm -f indexer-service tap-agent gateway

2. Critical Post-Setup Step: Restart Services

IMPORTANT: After just setup completes successfully, you MUST restart the indexer-service and tap-agent:

just reload

This is necessary because:

• The subgraphs may not be fully deployed when services first start
• Both indexer-service and tap-agent check the network subgraph on startup to detect if Horizon is enabled
• If the subgraph isn't ready, they won't detect Horizon and will run in legacy-only mode
• The dev docker-compose mounts the local compiled binaries with latest fixes
• Without this restart, you may see collection_id parsing errors or missing V2 functionality

Running V2 Tests

Basic V2 Test

just test-local-v2

This runs the V2 RAV (Receipt Aggregate Voucher) integration test which:

1. Sends V2 TAP receipts to the indexer-service
2. Verifies receipts are stored in tap_horizon_receipts table
3. Checks if RAVs are generated in tap_horizon_ravs table

Load Test

just load-test-v2 1000  # Send 1000 V2 receipts

Known Issues and Workarounds

1. Collection ID Parsing Error

Symptom:

thread 'main' panicked at crates/tap-agent/src/agent/sender_accounts_manager.rs:830:29:
Invalid collection_id '74a9df11e27c1f3579b2950b02e4dc54eb40ea28                        ': invalid string length

Solution: Run just reload to use the updated tap-agent binary with collection_id parsing fixes.

2. Missing Allocation Warnings

What you'll see:

WARN indexer_tap_agent::agent::sender_account: Missing allocation was not closed yet, 
allocation_id: 0x00000000000000000000000074a9df11e27c1f3579b2950b02e4dc54eb40ea28

Explanation: V2 allocations must be closed before receipts can be aggregated into RAVs. This is expected behavior in the test environment where allocations remain open.

3. Attestation Derivation Failures

What you'll see:

WARN indexer_attestation: Cannot derive attestation signer for allocation 0xbf46488Af7C03adAC9221077ff10B04d9A6190E1 
after trying 200 key combinations. The reason is unclear - this could be a configuration issue, 
but we're not certain.

Explanation: Some allocations have deployment IDs that exceed BIP32 path length limits. The system falls back to V2 derivation but some allocations may still fail. This affects attestation signing but not TAP receipt processing.

Understanding the Logs

Successful V2 Processing

When V2 is working correctly, you should see:

1. Horizon Detection:

INFO indexer_monitor::horizon_detection: Horizon (V2) contracts detected - found 1 PaymentsEscrow accounts
INFO indexer_tap_agent::agent: TAP Agent Mode: Process existing V1 receipts for RAVs, accept new V2 receipts

2. V2 Actor Creation:

Actor with id: "0.1", name: "horizon:0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"

3. Receipt Processing (in database):

-- Check V2 receipts
SELECT COUNT(*) FROM tap_horizon_receipts;  -- Should show receipt count

-- Check V2 RAVs (will be 0 until allocations are closed)
SELECT COUNT(*) FROM tap_horizon_ravs;

Expected Warnings

1. Missing Allocation Not Closed: Normal for test environment
2. Attestation Failures: Expected for certain deployment IDs

Questions for Upstream Teams

For indexer-agent Team

1. Allocation Lifecycle: In the test environment, how should V2 allocations be properly closed to enable RAV generation? The current test setup creates allocations but doesn't close them.

2. Migration Period Handling: The tap-agent expects collection_ids in the database to be either:

   • 20-byte addresses (during migration from V1)
   • 32-byte CollectionIds (native V2)

   Is this the correct approach for the migration period?

3. Mnemonic Configuration: Some allocations cannot be derived even after trying 200 combinations. Are test allocations being created with a different mnemonic than what indexer-service uses? When I look it seems like everything is correct but maybe I'm missing something.

For Smart Contracts Team

1. Collection ID Format: The system converts 20-byte allocation IDs to 32-byte CollectionIds using zero-padding. Is this the intended behavior for Horizon V2?

For Gateway Team

1. V2 Service Paths: The aggregator uses service paths (tap_aggregator.v1.* vs tap_aggregator.v2.*) for version routing. See the fork we're using in testing - although I'm unsure whether I need or am using at all the gateway here apart from as a docker compose dependency.

Troubleshooting

Check Service Health

docker ps | grep -E "indexer-service|tap-agent|tap-aggregator"

View Logs

just logs  # All services
docker logs tap-agent -f  # Specific service

Database Queries

# Check V2 receipts
docker exec postgres psql -U postgres -d indexer_components_1 -c \
  "SELECT collection_id, COUNT(*) FROM tap_horizon_receipts GROUP BY collection_id;"

# Check for V2 RAVs
docker exec postgres psql -U postgres -d indexer_components_1 -c \
  "SELECT * FROM tap_horizon_ravs;"

# Check PostgreSQL notifications
docker exec postgres psql -U postgres -d indexer_components_1 -c \
  "LISTEN tap_horizon_receipt_notification;"

Force Service Rebuild

cargo build --release -p indexer-service-rs -p indexer-tap-agent
just reload

Clean Restart

just down
docker volume prune -f  # WARNING: Removes all data
just setup
just reload

Architecture Notes

V2 Receipt Flow

1. Gateway sends V2 TAP receipts to indexer-service
2. indexer-service validates and stores in tap_horizon_receipts
3. PostgreSQL trigger fires notification
4. tap-agent receives notification and processes receipts
5. When allocation is closed, receipts aggregate into RAVs
6. RAVs are stored in tap_horizon_ravs for redemption

Key Differences from V1

• Allocation IDs: V1 uses 20-byte, V2 uses 32-byte CollectionIds
• Database Tables: Separate tables for V1 (scalar_tap_*) and V2 (tap_horizon_*)
• Actor System: Separate actor hierarchies with "horizon:" prefix
• Service Paths: Different gRPC paths for V1/V2 in tap-aggregator

Current Status

As of the latest testing:

• ✅ V2 receipt ingestion working
• ✅ PostgreSQL notifications functioning
• ✅ Actor system creating V2 actors
• ⚠️ RAV generation pending allocation closure
• ❌ Automatic RAV redemption not functioning (out of scope for indexer-rs?)

The V2 TAP pipeline is operational but requires proper allocation lifecycle management and escrow configuration for full end-to-end functionality.

---

Signed off by Joseph Livesey [email protected]

[github-actions]

Pull Request Test Coverage Report for Build 15981239840

Details

• 333 of 1096 (30.38%) changed or added relevant lines in 27 files are covered.
• 11 unchanged lines in 5 files lost coverage.
• Overall coverage decreased (-3.2%) to 69.453%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/tap-agent/src/tap/context/checks/allocation_id.rs	5	6	83.33%
crates/tap-agent/src/tap/context/receipt.rs	30	31	96.77%
crates/indexer-receipt/src/lib.rs	8	10	80.0%
crates/query/src/lib.rs	1	3	33.33%
crates/service/src/tap/checks/allocation_eligible.rs	2	4	50.0%
crates/monitor/src/attestation.rs	2	6	33.33%
crates/tap-agent/src/tap/context/rav.rs	12	16	75.0%
crates/service/src/middleware/sender.rs	4	12	33.33%
crates/tap-agent/src/tap/context.rs	8	16	50.0%
crates/service/src/tap/checks/sender_balance_check.rs	5	14	35.71%

Files with Coverage Reduction	New Missed Lines	%
crates/tap-agent/src/tap/context/rav.rs	1	86.89%
crates/attestation/src/lib.rs	2	80.56%
crates/service/src/service.rs	2	0.0%
crates/tap-agent/src/agent/sender_account.rs	2	72.96%
crates/watcher/src/lib.rs	4	92.71%

Totals
Change from base Build 15977955855:	-3.2%
Covered Lines:	8997
Relevant Lines:	12954

---

💛 - Coveralls

[neithanmo]

The documentation is great!!

[neithanmo]

nice!

[neithanmo]

We can remove this setting later, after the testing phase

[neithanmo]

but it is already smart enough to fallback to V1, really good!

[suchapalaver]

Basically, what I want to be clear is that v1 uses a separate escrow subgraph. If we're already on v2/horizon then we only have an escrow subgraph apart from the network subgraph to provide legacy support for already existing receipts.

[neithanmo]

Incredible work, a lot of changes and little details!
All looks good to me!