Feat/salesforce bulk upload

[etsenake]

Summary

This MR implements a Salesforce Bulk Upload destination as an open source contribution to RudderStack. This enables high-volume bulk synchronization of customer data to Salesforce using the Bulk API 2.0, optimized for RETL (Reverse ETL) warehouse syncs.

Motivation

Fullscript needs to sync large volumes of data (100K-500K+ records) from our data warehouse to Salesforce. The existing real-time Salesforce destination uses REST API for per-event syncs, which is inefficient for bulk operations due to:

• High API call volume (one call per record)
• Rate limiting issues at scale
• Poor throughput for bulk operations
• Increased API usage costs

What This Implements

Core AsyncDestinationManager Implementation

New package: router/batchrouter/asyncdestinationmanager/salesforce-bulk/

Key Features

Salesforce Bulk API 2.0 integration

• Job creation, CSV upload, job closing
• Status polling and result retrieval
• Handles all job states (Open, InProgress, JobComplete, Failed, Aborted)

OAuth 2.0 authentication

• Integrates with RudderStack's OAuth v2 service
• Token caching with expiry tracking
• Automatic token refresh on 401 errors

CSV generation

• Extracts transformed Salesforce fields from processor output
• 100MB file size limit (Bulk API 2.0 max)
• Overflow handling for large datasets
• Hash-based job tracking for success/failure matching

Error handling

• Categorized errors: RefreshToken, RateLimit, BadRequest, ServerError
• Proper retry vs abort logic
• Detailed error messages

Dual use case support

• RETL warehouse syncs (primary): Uses VDM field mapping, extracts object type from context
• Event streams (secondary): Uses config object type, batch processing

Integration Points

Registered in:

• asyncdestinationmanager/manager.go - Manager factory routing
• asyncdestinationmanager/common/utils.go - Added to asyncDestinations list
• utils/misc/misc.go - Added to BatchDestinations() list

Testing

✅ All tests passing

• Transform method (payload extraction)
• Config parsing with validation
• Object info extraction (VDM and event stream)
• CSV file creation
• Hash calculation for tracking
• Poll status handling (all job states)
• API error categorization
• Manager factory

Test approach:

• Table-driven tests following warehouse testing guidelines
• Uses require for assertions
• t.Parallel() for concurrent execution
• Proper cleanup with t.Cleanup()
• Manual mocks (no external dependencies)

Transformer Reuse

We reuse the existing Salesforce transformer instead of creating a new one. This is a proven pattern in RudderStack:

// Already exists:
salesforce_oauth → 'salesforce' transformer
salesforce_oauth_sandbox → 'salesforce' transformer  

// We add:
salesforce_bulk_upload → 'salesforce' transformer

Benefits:

• ✅ VDM v1 support for free (visual field mapping UI)
• ✅ VDM v2 support for free (with recordTransform.js in companion MR)
• ✅ Zero transformer code duplication
• ✅ Consistent UX with regular Salesforce destination
• ✅ Future transformer improvements automatically benefit bulk upload

What's NOT in This MR

Requires RudderStack Team Collaboration

Control Plane changes (not in this repo):

1. Backend config destination definition
2. OAuth Connected App setup (can likely reuse existing Salesforce OAuth)
3. UI configuration schema with VDM flags:
   • supportsVisualMapper: true
   • supportedSourceTypes: ["warehouse", "cloud"]
   • supportedMessageTypes: {"cloud": ["identify", "track", "record"]}
4. Destination metadata (icon, description, category)

Companion MR in rudder-transformer: See eng/hercules/rudder-transformer!1

Testing Instructions

Run Unit Tests

cd router/batchrouter/asyncdestinationmanager/salesforce-bulk
go test -v

Expected: All tests pass ✅

Integration Testing

Requires:

• Salesforce sandbox environment
• OAuth account configured in Control Plane
• Backend config with destination definition

Related Documentation

• RFC: Salesforce Bulk Upload - Complete technical design
• Salesforce Bulk API 2.0 Docs
• AsyncDestinationManager README

Checklist

• Code follows AsyncDestinationManager pattern
• All unit tests passing
• Follows warehouse testing guidelines
• OAuth v2 service integration
• Error handling for all scenarios
• Proper logging and metrics hooks
• Documentation (RFC) included
• Integration tests (requires Control Plane setup)
• Code review from RudderStack team
• Control Plane configuration coordinated

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.