Skip to content

Feature: Add multi-config #918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 218 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1373,6 +1373,224 @@ Listed below are all configuration options.
corresponding table in DynamoDB at model persisting if the table
doesn't exist yet. Default is `true`

## Multi-Configuration Support

Dynamoid supports multiple configurations to connect to different DynamoDB instances
across multiple AWS accounts or regions. This is useful when you need to:

- Connect to DynamoDB tables in different AWS accounts
- Use different regions for different models
- Separate production/staging data across different AWS setups
- Implement cross-account data access patterns

### Setting up Multiple Configurations

Configure multiple DynamoDB connections in your application initializer:

```ruby
# config/initializers/dynamoid.rb
Dynamoid.multi_configure do |config|
# Primary configuration (e.g., main application data)
config.add_config(:primary) do |c|
c.access_key = ENV.fetch('PRIMARY_AWS_ACCESS_KEY', nil)
c.secret_key = ENV.fetch('PRIMARY_AWS_SECRET_KEY', nil)
c.region = 'us-east-1'
c.namespace = 'myapp_primary'
end

# Secondary configuration (e.g., analytics data)
config.add_config(:analytics) do |c|
c.access_key = ENV.fetch('ANALYTICS_AWS_ACCESS_KEY', nil)
c.secret_key = ENV.fetch('ANALYTICS_AWS_SECRET_KEY', nil)
c.region = 'us-west-2'
c.namespace = 'myapp_analytics'
end

# Cross-account configuration (e.g., partner data)
config.add_config(:partner) do |c|
c.credentials = Aws::AssumeRoleCredentials.new(
role_arn: ENV.fetch('PARTNER_ROLE_ARN', nil),
role_session_name: 'dynamoid-cross-account'
)
c.region = 'eu-west-1'
c.namespace = 'partner_shared'
end
end
```

### Using Multiple Configurations in Models

Specify which configuration a model should use with the `dynamoid_config` method:

```ruby
# Models using primary configuration
class User
include Dynamoid::Document

dynamoid_config :primary

field :name, :string
field :email, :string

has_many :orders
end

class Order
include Dynamoid::Document

dynamoid_config :primary

field :total, :number
field :status, :string

belongs_to :user
end

# Models using analytics configuration
class PageView
include Dynamoid::Document

dynamoid_config :analytics

field :url, :string
field :user_id, :string
field :timestamp, :datetime

global_secondary_index hash_key: :user_id, range_key: :timestamp
end

class Report
include Dynamoid::Document

dynamoid_config :analytics

field :name, :string
field :data, :serialized
field :generated_at, :datetime
end

# Models using partner configuration
class SharedData
include Dynamoid::Document

dynamoid_config :partner

field :partner_id, :string
field :content, :serialized
field :sync_status, :string
end

# Models using default configuration (fallback to main Dynamoid.configure)
class SystemLog
include Dynamoid::Document

# No dynamoid_config specified - uses default configuration

field :level, :string
field :message, :string
field :timestamp, :datetime
end
```

### Configuration Inheritance

Models automatically inherit the correct configuration for all operations:

- **Table operations**: `create_table`, `delete_table`
- **CRUD operations**: `create`, `save`, `update`, `delete`, `find`
- **Queries**: `where`, `all`, `first`, `last`
- **Batch operations**: `import`, `batch_write`
- **Scanning**: `scan`

```ruby
# Each model uses its own DynamoDB connection
User.create(name: "John", email: "[email protected]") # Uses :primary config
PageView.create(url: "/home", user_id: "123") # Uses :analytics config
SharedData.create(partner_id: "partner1", content: {}) # Uses :partner config
SystemLog.create(level: "info", message: "App started") # Uses default config
```

### Table Names and Namespaces

Each configuration can have its own namespace, resulting in different table prefixes:

```ruby
# With the configurations above:
User.table_name # => "myapp_primary_users"
PageView.table_name # => "myapp_analytics_page_views"
SharedData.table_name # => "partner_shared_shared_data"
SystemLog.table_name # => "dynamoid_system_logs" (uses default namespace)
```

### Configuration Management

```ruby
# List all configured names
Dynamoid::MultiConfig.configuration_names
# => [:primary, :analytics, :partner]

# Check if a configuration exists
Dynamoid::MultiConfig.configuration_exists?(:primary)
# => true

# Get a specific configuration
config = Dynamoid::MultiConfig.get_config(:primary)
config.region # => "us-east-1"

# Remove a configuration
Dynamoid::MultiConfig.remove_config(:analytics)

# Clear all configurations
Dynamoid::MultiConfig.clear_all
```

### Error Handling

If you specify a non-existent configuration, Dynamoid will raise an error:

```ruby
class InvalidModel
include Dynamoid::Document

dynamoid_config :nonexistent # This configuration doesn't exist

field :name, :string
end

InvalidModel.create(name: "test")
# => Dynamoid::Errors::UnknownConfiguration: Unknown configuration: nonexistent
```

### Best Practices

1. **Environment-based configuration**: Use environment variables for sensitive credentials
2. **Logical separation**: Group related models in the same configuration
3. **Namespace isolation**: Use distinct namespaces to avoid table name conflicts
4. **Role-based access**: Use IAM roles for cross-account access when possible
5. **Connection reuse**: Configurations create connection pools, so reuse them efficiently

```ruby
# Example: Environment-based setup
Dynamoid.multi_configure do |config|
# Production data
config.add_config(:production) do |c|
c.credentials = Aws::InstanceProfileCredentials.new
c.region = ENV.fetch('PRODUCTION_REGION', 'us-east-1')
c.namespace = "#{Rails.application.class.module_parent_name.downcase}_prod"
end

# Analytics warehouse
config.add_config(:warehouse) do |c|
c.credentials = Aws::AssumeRoleCredentials.new(
role_arn: ENV['WAREHOUSE_ROLE_ARN'],
role_session_name: "#{Rails.application.class.module_parent_name.downcase}-warehouse"
)
c.region = ENV.fetch('WAREHOUSE_REGION', 'us-west-2')
c.namespace = "warehouse_#{Rails.env}"
end
end
```


## Concurrency

Expand Down
5 changes: 5 additions & 0 deletions lib/dynamoid.rb
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
require 'dynamoid/components'
require 'dynamoid/document'
require 'dynamoid/adapter'
require 'dynamoid/multi_config'
require 'dynamoid/transaction_write'

require 'dynamoid/tasks/database'
Expand All @@ -51,6 +52,10 @@ def configure
end
alias config configure

def multi_configure(&block)
Dynamoid::MultiConfig.configure(&block)
end

def logger
Dynamoid::Config.logger
end
Expand Down
14 changes: 7 additions & 7 deletions lib/dynamoid/criteria/chain.rb
Original file line number Diff line number Diff line change
Expand Up @@ -233,18 +233,18 @@ def delete_all
ranges = []

if @key_fields_detector.key_present?
Dynamoid.adapter.query(source.table_name, query_key_conditions, query_non_key_conditions, query_options).flat_map { |i| i }.collect do |hash|
source.adapter.query(source.table_name, query_key_conditions, query_non_key_conditions, query_options).flat_map { |i| i }.collect do |hash|
ids << hash[source.hash_key.to_sym]
ranges << hash[source.range_key.to_sym] if source.range_key
end
else
Dynamoid.adapter.scan(source.table_name, scan_conditions, scan_options).flat_map { |i| i }.collect do |hash|
source.adapter.scan(source.table_name, scan_conditions, scan_options).flat_map { |i| i }.collect do |hash|
ids << hash[source.hash_key.to_sym]
ranges << hash[source.range_key.to_sym] if source.range_key
end
end

Dynamoid.adapter.delete(source.table_name, ids, range_key: ranges.presence)
source.adapter.delete(source.table_name, ids, range_key: ranges.presence)
end
alias destroy_all delete_all

Expand Down Expand Up @@ -575,7 +575,7 @@ def raw_pages
# @since 3.1.0
def raw_pages_via_query
Enumerator.new do |y|
Dynamoid.adapter.query(source.table_name, query_key_conditions, query_non_key_conditions, query_options).each do |items, metadata|
source.adapter.query(source.table_name, query_key_conditions, query_non_key_conditions, query_options).each do |items, metadata|
options = metadata.slice(:last_evaluated_key)

y.yield items, options
Expand All @@ -590,7 +590,7 @@ def raw_pages_via_query
# @since 3.1.0
def raw_pages_via_scan
Enumerator.new do |y|
Dynamoid.adapter.scan(source.table_name, scan_conditions, scan_options).each do |items, metadata|
source.adapter.scan(source.table_name, scan_conditions, scan_options).each do |items, metadata|
options = metadata.slice(:last_evaluated_key)

y.yield items, options
Expand All @@ -613,11 +613,11 @@ def issue_scan_warning
end

def count_via_query
Dynamoid.adapter.query_count(source.table_name, query_key_conditions, query_non_key_conditions, query_options)
source.adapter.query_count(source.table_name, query_key_conditions, query_non_key_conditions, query_options)
end

def count_via_scan
Dynamoid.adapter.scan_count(source.table_name, scan_conditions, scan_options)
source.adapter.scan_count(source.table_name, scan_conditions, scan_options)
end

def field_condition(key, value_before_type_casting)
Expand Down
25 changes: 23 additions & 2 deletions lib/dynamoid/document.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@ module Document
include Dynamoid::Components

included do
class_attribute :options, :read_only_attributes, :base_class, instance_accessor: false
class_attribute :options, :read_only_attributes, :base_class, :dynamoid_config_name, instance_accessor: false
self.options = {}
self.read_only_attributes = []
self.base_class = self
self.dynamoid_config_name = nil

Dynamoid.included_models << self unless Dynamoid.included_models.include? self
end
Expand All @@ -21,6 +22,26 @@ def attr_readonly(*read_only_attributes)
self.read_only_attributes.concat read_only_attributes.map(&:to_s)
end

# Set the DynamoDB configuration to use for this model
#
# @param [Symbol] config_name the name of the configuration
# @since 4.0.0
def dynamoid_config(config_name)
self.dynamoid_config_name = config_name.to_sym
end

# Get the adapter for this model's configuration
#
# @return [Dynamoid::Adapter] the adapter instance
# @since 4.0.0
def adapter
if dynamoid_config_name
Dynamoid::MultiConfig.get_adapter(dynamoid_config_name)
else
Dynamoid.adapter
end
end

# Returns the read capacity for this table.
#
# @return [Integer] read capacity units
Expand Down Expand Up @@ -80,7 +101,7 @@ def hash_key
# @return [Integer] items count in a table
# @since 0.6.1
def count
Dynamoid.adapter.count(table_name)
adapter.count(table_name)
end

# Initialize a new object.
Expand Down
6 changes: 6 additions & 0 deletions lib/dynamoid/errors.rb
Original file line number Diff line number Diff line change
Expand Up @@ -106,5 +106,11 @@ def initialize(_msg = nil)
super('Scan operations prohibited. Modify Dynamoid::Config.error_on_scan to change this behavior.')
end
end

class UnknownConfiguration < Error
def initialize(config_name)
super("Unknown configuration: #{config_name}")
end
end
end
end
Loading