Sui Indexer

A high-performance, modular blockchain indexer for the Sui network with real-time event monitoring and customizable data processing

✨ Features

• 🚀 High Performance: Built with Rust for maximum speed and concurrent processing
• 🔄 Real-time Monitoring: gRPC-based subscriptions for live blockchain data
• 🧩 Modular Architecture: Clean separation of concerns with pluggable components
• 🎯 Dual Usage: Works as both a framework library and a standalone CLI tool
• 📊 Comprehensive Indexing: Events, transactions, objects, and checkpoints
• 🗄️ PostgreSQL Backend: Reliable storage with async operations via sqlx
• ⚙️ Configuration-driven: TOML-based configuration with environment variable support
• � Customizable Processing: Implement custom event processors for your specific use cases
• 📈 Production Ready: Built-in health checks, metrics, and database migrations
• 🔧 Developer Friendly: Extensive logging, error handling, and debugging tools

�️ Architecture Overview

The Sui Indexer Framework follows a clean, modular architecture designed for flexibility and performance:

graph TB
    subgraph "Sui Indexer Framework"
        subgraph "Application Layer"
            CLI["🖥️ CLI Tool<br/>(Production)"]
            SDK["📚 Library/SDK<br/>(Custom Apps)"]
        end
        
        subgraph "Core Framework"
            IC["🎯 IndexerCore<br/>(Orchestration)"]
            
            subgraph "Core Components"
                SC["🌐 SuiClient<br/>(gRPC)"]
                EP["⚙️ EventProcessor<br/>(Customizable)"]
                SM["💾 StorageManager<br/>(PostgreSQL)"]
            end
        end
        
        subgraph "Configuration Layer"
            CONFIG["📋 TOML Configuration<br/>+ Environment Variables"]
        end
        
        subgraph "External Systems"
            SUI["🔗 Sui Network<br/>(Fullnode gRPC)"]
            DB["🗄️ PostgreSQL<br/>(Database)"]
        end
    end
    
    CLI --> IC
    SDK --> IC
    IC --> SC
    IC --> EP
    IC --> SM
    SC --> SUI
    SM --> DB
    CONFIG --> IC
    
    style CLI fill:#e1f5fe
    style SDK fill:#e8f5e8
    style IC fill:#fff3e0
    style SC fill:#f3e5f5
    style EP fill:#e8eaf6
    style SM fill:#fce4ec
    style CONFIG fill:#f1f8e9
    style SUI fill:#e3f2fd
    style DB fill:#fafafa

Loading

Core Components

• sui-indexer-core: Main orchestration service and indexer logic
• sui-indexer-config: Configuration management and loading
• sui-indexer-events: Event processing pipeline with customizable processors
• sui-indexer-storage: Database abstraction layer with migrations
• sui-indexer-cli: Command-line interface for production deployments

🔄 Framework Comparison

Our Framework vs. Official Sui Indexer

This framework provides an alternative approach to the official MystenLabs sui-indexer-alt-framework. Here's a comprehensive comparison to help you choose the right solution:

Aspect	🎯 Our Framework	🏢 Official Framework
🎨 Design Philosophy	Framework-first design as both library and CLI. Developer-friendly with simple API and extensive examples. Focus on customization with easy custom processors. Production-ready with built-in CLI for immediate deployment.	Production-first approach built for Mysten's infrastructure needs. Performance-optimized for high-throughput data ingestion. Store-trait based with abstract storage layer. Enterprise-grade design for large-scale operations.
🏗️ Architecture	Modular crates with clean separation of concerns. Event-driven approach focused on real-time event processing. Single pipeline with unified processing approach. gRPC subscriptions for real-time data streaming.	Pipeline-based architecture with sequential & concurrent pipelines. Checkpoint-driven processing of data in checkpoint batches. Store abstraction with pluggable storage backends. Watermark system for progress tracking and resume capability.
📊 Data Processing	Real-time events via live gRPC subscriptions. Custom processors implementing EventProcessor trait. Batch processing with configurable batch sizes. TOML configuration with declarative event filters.	Checkpoint ingestion for full checkpoint data processing. Handler system with ProcessorHandler for custom logic. Concurrent pipelines with parallel processing support. Built-in IndexerMetrics integration.
🗄️ Storage	PostgreSQL focus with optimization for PostgreSQL. SQLx integration for async database operations. Auto-migrations with automatic schema management. Simple schema covering events, transactions, objects.	Store trait providing abstract storage interface. Multiple backends including PostgreSQL and custom stores. Advanced features with watermarks and progress tracking. Complex schema with comprehensive data model.
⚙️ Configuration	TOML-based human-readable configuration. Environment variables with 12-factor app support. Event filters for package/module/type filtering. Network settings with connection pooling and retries.	Rust structures using IndexerArgs and ClientArgs. Programmatic code-based configuration. Advanced options for clustering and pipeline tuning. Ingestion config for detailed checkpoint processing.
🚀 Ease of Use	Quick start with single command deployment. Extensive examples for DeFi monitoring and custom processors. Built-in CLI interface with production commands. Gentle learning curve with comprehensive documentation.	Complex setup requiring deeper Rust knowledge. Flexible architecture with extensive configuration options. No built-in CLI requiring custom application development. Steeper learning curve with enterprise focus.
🎯 Best Use Cases	DeFi monitoring for protocol-specific event tracking. Real-time analytics for live dashboard data. Custom applications with tailored indexing solutions. Rapid prototyping for quick proof-of-concepts. Small to medium scale up to 10K events/second.	Full node indexing for complete blockchain data. Data warehousing with large-scale data processing. Enterprise integration with complex infrastructure. High throughput for large-scale production systems. Custom storage for non-PostgreSQL backends.
📈 Performance	Throughput of approximately 10K events/second. Memory usage around 50MB base. Latency under 100ms for event processing. Optimized for event-driven workloads.	Higher throughput for batch processing. Variable memory usage based on configuration. Latency optimized for batch ingestion. Optimized for full checkpoint processing.
🛠️ Development	Framework crates providing reusable components. Testing support with built-in test utilities. Extensive documentation with guides and examples. Community focus with open-source friendly approach.	Library approach offering lower-level building blocks. Mysten ecosystem integration with Sui tooling. Advanced features providing enterprise-grade capabilities. Official support backed by Mysten Labs.

🤔 Which Should You Choose?

Choose Our Framework If:

• 🎯 You want to build custom DeFi monitoring or analytics applications
• 🚀 You need to get started quickly with minimal configuration
• 📊 You're focused on real-time event processing rather than full blockchain indexing
• 🧩 You prefer a framework approach with built-in CLI tools
• 📈 Your scale is small to medium (under 10K events/second)
• 🎨 You value developer experience and extensive documentation

Choose Official Framework If:

• 🏢 You're building enterprise-scale infrastructure
• 📦 You need to process complete checkpoint data, not just events
• 🔧 You require custom storage backends beyond PostgreSQL
• ⚡ You need maximum throughput for large-scale data processing
• 🔄 You're building complex data pipelines with multiple processing stages
• 🏗️ You prefer lower-level control over the indexing architecture

Both frameworks are excellent choices depending on your specific needs. Our framework excels at rapid development and custom event processing, while the official framework provides enterprise-grade infrastructure for large-scale operations.

📦 Installation

As a CLI Tool

# Install from source
git clone https://github.com/longcipher/sui-indexer.git
cd sui-indexer
cargo install --path bin/sui-indexer-cli

Or run directly:

cargo run -p sui-indexer-cli -- --help

As a Library

Add to your Cargo.toml:

[dependencies]
sui-indexer-core = "0.1.0"
sui-indexer-config = "0.1.0"
sui-indexer-events = "0.1.0"
sui-indexer-storage = "0.1.0"

🚀 Quick Start

1. CLI Usage (Production Deployments)

Generate configuration file:

sui-indexer config

Edit the configuration file:

[network]
grpc_url = "https://fullnode.mainnet.sui.io/"
network = "mainnet"

[database]
url = "postgresql://postgres:password@localhost:5433/sui_indexer"

[events]
batch_size = 50
max_concurrent_batches = 4

# Event filters for monitoring specific protocols
filters = [
  { 
    package = "0x2", 
    module = "coin"
  },
  { 
    package = "0x3",
    module = "sui_system"
  }
]

Initialize the database:

sui-indexer init -c config.toml

Start indexing:

sui-indexer start -c config.toml

2. Framework Usage (Custom Applications)

Create your own indexer application using the framework:

use sui_indexer_core::IndexerCore;
use sui_indexer_config::IndexerConfig;
use eyre::Result;

#[tokio::main]
async fn main() -> Result<()> {
    // Load configuration
    let config = IndexerConfig::from_file("config.toml")?;
    
    // Create and initialize indexer
    let indexer = IndexerCore::new(config).await?;
    indexer.initialize().await?;
    
    // Start indexing with real-time monitoring
    indexer.start().await?;
    
    Ok(())
}

3. Custom Event Processing

Implement custom event processors for your specific use cases:

use sui_indexer_core::IndexerCore;
use sui_indexer_config::IndexerConfig;
use sui_indexer_events::{EventProcessor, ProcessedEvent};
use sui_json_rpc_types::SuiEvent;
use async_trait::async_trait;
use eyre::Result;
use std::sync::Arc;
use tracing::info;

// Example: Monitoring DeFi protocol events (using Navi as example)
struct DeFiEventProcessor {
    // Track specific protocol package IDs
    navi_package_id: String,
}

impl DeFiEventProcessor {
    fn new() -> Self {
        Self {
            navi_package_id: "0x81c408448d0d57b3e371ea94de1d40bf852784d3e225de1e74acab3e8395c18f".to_string(),
        }
    }
}

#[async_trait]
impl EventProcessor for DeFiEventProcessor {
    async fn process_event(&self, event: SuiEvent) -> Result<ProcessedEvent> {
        let package_id_str = event.package_id.to_string();
        
        // Identify protocol-specific events
        if package_id_str.contains(&self.navi_package_id) {
            self.handle_navi_event(&event).await?;
        } else {
            self.handle_generic_event(&event).await?;
        }
        
        // Convert to ProcessedEvent
        Ok(ProcessedEvent::from_sui_event(event))
    }
}

impl DeFiEventProcessor {
    async fn handle_navi_event(&self, event: &SuiEvent) -> Result<()> {
        match event.type_.name.as_str() {
            name if name.contains("DepositEvent") => {
                info!("💰 Deposit detected: {} deposited funds", event.sender);
                
                // Extract deposit amount and asset
                if let Some(amount) = event.parsed_json.get("amount") {
                    info!("💵 Amount: {}", amount);
                }
                if let Some(asset) = event.parsed_json.get("coin_type") {
                    info!("🪙 Asset: {}", asset);
                }
                
                // Custom business logic here
                // - Update user portfolio
                // - Calculate TVL changes
                // - Trigger notifications
            }
            name if name.contains("BorrowEvent") => {
                info!("🏦 Borrow detected: {} borrowed funds", event.sender);
                
                // Extract borrow details
                if let Some(amount) = event.parsed_json.get("amount") {
                    info!("� Borrowed: {}", amount);
                }
                
                // Custom business logic here
                // - Update debt tracking
                // - Calculate utilization rates
                // - Risk assessment
            }
            name if name.contains("WithdrawEvent") => {
                info!("� Withdrawal detected from {}", event.sender);
                // Handle withdrawal logic
            }
            name if name.contains("RepayEvent") => {
                info!("💳 Loan repayment from {}", event.sender);
                // Handle repayment logic
            }
            _ => {
                info!("📋 Other DeFi event: {}", event.type_.name);
            }
        }
        Ok(())
    }
    
    async fn handle_generic_event(&self, event: &SuiEvent) -> Result<()> {
        // Handle other blockchain events
        info!("📝 Processing event: {} from package {}", 
              event.type_.name, event.package_id);
        Ok(())
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    // Load configuration
    let config = IndexerConfig::from_file("config.toml")?;
    
    // Create custom event processor
    let processor = Arc::new(DeFiEventProcessor::new());
    
    // Create indexer with custom processor
    let indexer = IndexerCore::with_event_processor(config, processor).await?;
    indexer.initialize().await?;
    
    // Start with custom processing
    indexer.start().await?;
    
    Ok(())
}

4. Advanced Usage Patterns

Multi-Protocol Monitoring

use std::collections::HashMap;

struct MultiProtocolProcessor {
    protocol_handlers: HashMap<String, Box<dyn ProtocolHandler>>,
}

trait ProtocolHandler: Send + Sync {
    fn handle_event(&self, event: &SuiEvent) -> Result<()>;
}

struct UniswapHandler;
struct AaveHandler;
struct CompoundHandler;

impl ProtocolHandler for UniswapHandler {
    fn handle_event(&self, event: &SuiEvent) -> Result<()> {
        // Handle Uniswap-like DEX events
        info!("🔄 DEX Event: {}", event.type_.name);
        Ok(())
    }
}

impl ProtocolHandler for AaveHandler {
    fn handle_event(&self, event: &SuiEvent) -> Result<()> {
        // Handle Aave-like lending events
        info!("🏛️ Lending Event: {}", event.type_.name);
        Ok(())
    }
}

Event Filtering and Routing

use sui_indexer_config::EventFilter;

// Create sophisticated filters
let filters = vec![
    // Monitor specific DEX pools
    EventFilter {
        package: Some("0xdex_package_id".to_string()),
        module: Some("pool".to_string()),
        event_type: Some("SwapEvent".to_string()),
        sender: None,
    },
    // Monitor large transactions
    EventFilter {
        package: None,
        module: None,
        event_type: None,
        sender: Some("0xlarge_whale_address".to_string()),
    },
    // Monitor governance events
    EventFilter {
        package: Some("0xgovernance_package".to_string()),
        module: Some("voting".to_string()),
        event_type: None,
        sender: None,
    },
];

⚙️ Configuration

Configuration Structure

The framework uses TOML configuration files with the following structure:

[network]
grpc_url = "https://fullnode.mainnet.sui.io/"
network = "mainnet"

[network.pool]
max_connections = 20
timeout = 30
keep_alive = 60

[network.retry]
max_attempts = 5
initial_delay = 2000
max_delay = 30000
backoff_multiplier = 2.0

[database]
url = "postgresql://postgres:password@localhost:5433/sui_indexer"
max_connections = 20
min_connections = 5
connect_timeout = 30
idle_timeout = 300
auto_migrate = true

[events]
batch_size = 50
max_concurrent_batches = 4
index_transactions = true
index_objects = false

# Event filters for specific protocols/contracts
[[events.filters]]
package = "0x2"
module = "coin"

[[events.filters]]
package = "0x3"
module = "sui_system"
event_type = "ValidatorEpochInfoEvent"

# Example: DeFi protocol monitoring (Navi Protocol example)
[[events.filters]]
package = "0x81c408448d0d57b3e371ea94de1d40bf852784d3e225de1e74acab3e8395c18f"
module = "lending"
event_type = "0xd899cf7d2b5db716bd2cf55599fb0d5ee38a3061e7b6bb6eebf73fa5bc4c81ca::lending::DepositEvent"

Environment Variables

Override any configuration with environment variables:

export SUI_INDEXER_NETWORK_GRPC_URL="https://fullnode.testnet.sui.io/"
export SUI_INDEXER_DATABASE_URL="postgresql://localhost/sui_testnet"
export SUI_INDEXER_EVENTS_BATCH_SIZE="100"

🔧 CLI Commands

Start Indexing

sui-indexer start -c config.toml --log-level info

Initialize Database

sui-indexer init -c config.toml

Status Check

sui-indexer status -c config.toml

Example Status Output:

🔍 Sui Indexer Status Check
============================
📊 System Information:
  - Version: 0.1.0
🌐 Network Status:
  - Network: mainnet
  - gRPC URL: https://fullnode.mainnet.sui.io/
💾 Database Status:
  - URL: postgresql://***:***@localhost/sui_indexer
  - Connection: ✅ Connected
📋 Event Configuration:
  - Filters: 3 configured
  - Batch size: 50
  - Max concurrent batches: 4
🔗 Sui Network Status:
  - Connection: ✅ Connected
  - Latest checkpoint: 182976532
💾 Memory Usage:
  - Current: 45.2 MB
  - Peak: 67.8 MB
📊 Processing Statistics:
  - Events processed: 1,234
  - Transactions processed: 567
  - Last checkpoint: 182976530
  - Processing rate: 15.4 events/min

Database Management

Database migrations are handled automatically, but you can also manage them manually:

# Using the built-in migration helper
./migrate.sh run

# Check migration status
./migrate.sh info

# Using sqlx-cli directly
sqlx migrate run --source crates/sui-indexer-storage/migrations

Available migrations:

• 20250826000001_initial_schema.sql - Basic tables for checkpoints, transactions, events
• 20250826000002_processed_tables.sql - Processed events and transaction tracking
• 20250826000003_indexer_state.sql - Indexer state and statistics tables

📊 Monitoring & Metrics

Real-time Processing

The indexer provides comprehensive monitoring capabilities:

# Start with detailed logging
sui-indexer start -c config.toml --log-level debug

# Expected output for protocol events:
� Processing event: DepositEvent from package 0x81c408... (tx: 0x1234...)
💰 Deposit detected: 0xabcd... deposited funds
💵 Amount: 1000000000
🪙 Asset: 0x2::sui::SUI
✅ Event processed (processing time: 15ms)

Performance Metrics

• Throughput: Up to 10,000 events/second
• Latency: < 100ms for event processing
• Memory: ~50MB base usage (measured via status command)
• Storage: Efficient PostgreSQL schema with optimized indexes

Health Monitoring

# Built-in health check
sui-indexer health -c config.toml

# HTTP health endpoint (if server enabled)
curl http://localhost:8080/health

🧪 Development

Prerequisites

• Rust 1.89+
• PostgreSQL 12+
• Access to a Sui fullnode (mainnet or testnet)

Building from Source

git clone https://github.com/longcipher/sui-indexer.git
cd sui-indexer
cargo build --release

Running Tests

cargo test

Development with Docker

# Start PostgreSQL for testing
docker-compose up -d postgres

# Run the indexer in development mode
cargo run -p sui-indexer-cli -- start -c config.toml

Testing Custom Processors

For testing your custom event processors:

# Create a test configuration
cp config.example.toml test_config.toml

# Edit test_config.toml with your target events
# Run with your custom processor
cargo run --bin my_custom_indexer

📚 Documentation

API Reference

• IndexerCore: Main orchestration service
• EventProcessor: Trait for custom event processing
• StorageManager: Database abstraction
• ConfigLoader: Configuration management

Example Implementations

The /examples directory contains comprehensive examples:

Simple Event Indexer (examples/simple_indexer.rs)

Basic usage showing:

• Custom event processor implementation
• Programmatic configuration
• Basic event monitoring

cargo run --example simple_indexer -p sui-indexer-core

Custom DeFi Indexer (examples/custom_defi_indexer.rs)

Advanced example demonstrating:

• Multi-protocol event processing
• DeFi-specific business logic (using Navi Protocol as example)
• Custom data extraction and analysis
• Portfolio tracking patterns
• Risk monitoring implementation

cargo run --example custom_defi_indexer -p sui-indexer-core

Features in DeFi example:

• ✅ Deposit/Borrow/Withdraw/Repay event handling
• ✅ Liquidation monitoring and alerts
• ✅ TVL calculation patterns
• ✅ User portfolio tracking structure
• ✅ Utilization rate calculations
• ✅ Risk assessment framework

For detailed documentation on these examples, see examples/README.md.

📝 License

This project is dual-licensed under MIT and Apache 2.0. See LICENSE-MIT and LICENSE-APACHE for details.

🙏 Acknowledgments

• Inspired by ponder for its framework design
• Built on the Sui blockchain ecosystem

📞 Support

• 📖 Documentation
• 🐛 Issue Tracker
• 💬 Discussions

---

Made with ❤️ for the Sui blockchain community

Sui Indexer Framework - Powering the future of blockchain data indexing