Skip to content

Experiment with how to make YAML tests (and SQL more generally) able to encrypt records at rest #3557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Experiment with how to make YAML tests (and SQL more generally) able to encrypt records at rest #3557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fd082c0
Propagate options from connection into database.
MMcM Aug 22, 2025
396777b
Move serializer from RecordLayerConfig to StoreConfig
MMcM Aug 22, 2025
448a651
Add some more options related to encryption.
MMcM Aug 21, 2025
1ff759c
Use those options to generate encryption keys
MMcM Aug 22, 2025
ccc924e
Add a YAML test of that.
MMcM Aug 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion fdb-relational-api/src/main/java/com/apple/foundationdb/relational/api/Options.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,23 @@ public enum Name {
* operations interacting with FDB.
* Scope: Engine
*/
ASYNC_OPERATIONS_TIMEOUT_MILLIS
ASYNC_OPERATIONS_TIMEOUT_MILLIS,

/**
* A boolean indicating whether to encrypt records when saving and decrypt when loading.
*/
ENCRYPT_WHEN_SERIALIZING,

/**
* An AES encryption key in Base64.
*/
ENCRYPTION_KEY,

/**
* A text password to be used to generate an encryption key.
* Since a fixed salt is used, this is <em>not secure at all</em>.
*/
ENCRYPTION_PASSWORD,
Copy link
Contributor

@ohadzeliger ohadzeliger Aug 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be used for testing only? Maybe document it as such?

}

public enum IndexFetchMethod {
Expand Down Expand Up @@ -258,6 +274,7 @@ public enum IndexFetchMethod {
builder.put(Name.CASE_SENSITIVE_IDENTIFIERS, false);
builder.put(Name.CONTINUATIONS_CONTAIN_COMPILED_STATEMENTS, true);
builder.put(Name.ASYNC_OPERATIONS_TIMEOUT_MILLIS, 10_000L);
builder.put(Name.ENCRYPT_WHEN_SERIALIZING, false);
OPTIONS_DEFAULT_VALUES = builder.build();
}

Expand Down Expand Up @@ -291,6 +308,10 @@ public <T> T getOption(Name name) {
}
}

public Options withOption(@Nonnull Name name, Object value) throws SQLException {
return builder().fromOptions(this).withOption(name, value).build();
}

public Options withChild(@Nonnull Options childOptions) throws SQLException {
return Options.combine(this, childOptions);
}
Expand Down Expand Up @@ -425,6 +446,9 @@ private static Map<Name, List<OptionContract>> makeContracts() {
data.put(Name.VALID_PLAN_HASH_MODES, List.of(TypeContract.stringType()));
data.put(Name.CONTINUATIONS_CONTAIN_COMPILED_STATEMENTS, List.of(TypeContract.booleanType()));
data.put(Name.ASYNC_OPERATIONS_TIMEOUT_MILLIS, List.of(TypeContract.longType(), RangeContract.of(0L, Long.MAX_VALUE)));
data.put(Name.ENCRYPT_WHEN_SERIALIZING, List.of(TypeContract.booleanType()));
data.put(Name.ENCRYPTION_KEY, List.of(TypeContract.stringType()));
data.put(Name.ENCRYPTION_PASSWORD, List.of(TypeContract.stringType()));

return Collections.unmodifiableMap(data);
}
Expand Down
17 changes: 16 additions & 1 deletion ...al-core/src/main/java/com/apple/foundationdb/relational/recordlayer/AbstractDatabase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
package com.apple.foundationdb.relational.recordlayer;

import com.apple.foundationdb.record.provider.foundationdb.FDBRecordStoreBase;
import com.apple.foundationdb.relational.api.Options;
import com.apple.foundationdb.relational.api.Transaction;
import com.apple.foundationdb.relational.api.TransactionManager;
import com.apple.foundationdb.relational.api.catalog.RelationalDatabase;
Expand All @@ -34,6 +35,7 @@
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.net.URI;
import java.sql.SQLException;
import java.util.HashMap;
import java.util.Map;

Expand All @@ -49,13 +51,17 @@ public abstract class AbstractDatabase implements RelationalDatabase {
final Map<String, RecordLayerSchema> schemas = new HashMap<>();
@Nullable
private final RelationalPlanCache planCache;
@Nonnull
protected Options options;

public AbstractDatabase(@Nonnull final MetadataOperationsFactory metadataOperationsFactory,
@Nonnull DdlQueryFactory ddlQueryFactory,
@Nullable RelationalPlanCache planCache) {
@Nullable RelationalPlanCache planCache,
@Nonnull Options options) {
this.metadataOperationsFactory = metadataOperationsFactory;
this.ddlQueryFactory = ddlQueryFactory;
this.planCache = planCache;
this.options = options;
}

protected void setConnection(@Nonnull EmbeddedRelationalConnection conn) {
Expand Down Expand Up @@ -119,4 +125,13 @@ public DdlQueryFactory getDdlQueryFactory() {
public RelationalPlanCache getPlanCache() {
return planCache;
}

@Nonnull
public Options getOptions() {
return options;
}

public void setOption(@Nonnull Options.Name name, Object value) throws SQLException {
options = options.withOption(name, value);
}
}
3 changes: 2 additions & 1 deletion ...main/java/com/apple/foundationdb/relational/recordlayer/EmbeddedRelationalConnection.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,8 @@ public Options getOptions() {

@Override
public void setOption(Options.Name name, Object value) throws SQLException {
options = Options.builder().fromOptions(options).withOption(name, value).build();
options = options.withOption(name, value);
frl.setOption(name, value);
}

@Override
Expand Down
18 changes: 2 additions & 16 deletions ...l-core/src/main/java/com/apple/foundationdb/relational/recordlayer/RecordLayerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,36 +24,26 @@

import com.apple.foundationdb.record.IndexState;
import com.apple.foundationdb.record.provider.common.RecordSerializer;
import com.apple.foundationdb.record.provider.common.TransformedRecordSerializer;
import com.apple.foundationdb.record.provider.foundationdb.FDBRecordStoreBase;

import com.apple.foundationdb.record.provider.foundationdb.FormatVersion;
import com.apple.foundationdb.relational.recordlayer.storage.StoreConfig;
import com.google.protobuf.Message;

import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.zip.Deflater;

/**
* Holder object for RecordLayer-specific stuff that isn't directly tied to an actual FDB StorageCluster.
*/
@API(API.Status.EXPERIMENTAL)
public final class RecordLayerConfig {
private final FDBRecordStoreBase.UserVersionChecker userVersionChecker;
private final RecordSerializer<Message> serializer;
private final FormatVersion formatVersion;
private final Map<String, IndexState> indexStateMap;

private static final RecordSerializer<Message> DEFAULT_RELATIONAL_SERIALIZER = TransformedRecordSerializer.newDefaultBuilder()
.setEncryptWhenSerializing(false)
.setCompressWhenSerializing(true)
.setCompressionLevel(Deflater.DEFAULT_COMPRESSION)
.setWriteValidationRatio(0.0)
.build();

private RecordLayerConfig(RecordLayerConfigBuilder builder) {
this.userVersionChecker = builder.userVersionChecker;
this.serializer = builder.serializer;
this.formatVersion = builder.formatVersion;
this.indexStateMap = builder.indexStateMap;
}
Expand All @@ -62,10 +52,6 @@ public FDBRecordStoreBase.UserVersionChecker getUserVersionChecker() {
return userVersionChecker;
}

public RecordSerializer<Message> getSerializer() {
return serializer;
}

public FormatVersion getFormatVersion() {
return formatVersion;
}
Expand All @@ -86,7 +72,7 @@ public static class RecordLayerConfigBuilder {

public RecordLayerConfigBuilder() {
this.userVersionChecker = (oldUserVersion, oldMetaDataVersion, metaData) -> CompletableFuture.completedFuture(oldUserVersion);
this.serializer = DEFAULT_RELATIONAL_SERIALIZER;
this.serializer = StoreConfig.DEFAULT_RELATIONAL_SERIALIZER;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can probably be removed?

this.formatVersion = FormatVersion.getDefaultFormatVersion();
this.indexStateMap = Map.of();
}
Expand Down
7 changes: 2 additions & 5 deletions ...core/src/main/java/com/apple/foundationdb/relational/recordlayer/RecordLayerDatabase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,6 @@ public class RecordLayerDatabase extends AbstractDatabase {
private final RecordLayerConfig recordLayerConfig;

private final RelationalKeyspaceProvider.RelationalDatabasePath databasePath;
@Nonnull
private final Options options;

@Nullable
private final String defaultSchema;
Expand Down Expand Up @@ -83,14 +81,13 @@ public RecordLayerDatabase(FdbConnection fdbDb,
@Nullable RelationalPlanCache planCache,
@Nullable String defaultSchema,
@Nonnull Options options) {
super(metadataOperationsFactory, ddlQueryFactory, planCache);
super(metadataOperationsFactory, ddlQueryFactory, planCache, options);
this.fdbDb = fdbDb;
this.metaDataStore = new CachedMetaDataStore(metaDataStore);
this.storeCatalog = storeCatalog;
this.recordLayerConfig = config;
this.databasePath = databasePath;
this.defaultSchema = defaultSchema;
this.options = options;
}

@Override
Expand Down Expand Up @@ -129,7 +126,7 @@ public void close() throws RelationalException {
}

BackingRecordStore loadStore(@Nonnull Transaction txn, @Nonnull String schemaName, @Nonnull FDBRecordStoreBase.StoreExistenceCheck existenceCheck) throws RelationalException {
StoreConfig storeConfig = StoreConfig.create(recordLayerConfig, schemaName, databasePath, metaDataStore, txn);
StoreConfig storeConfig = StoreConfig.create(recordLayerConfig, schemaName, databasePath, metaDataStore, txn, options);
return BackingRecordStore.load(txn, storeConfig, existenceCheck);
}

Expand Down
5 changes: 1 addition & 4 deletions .../java/com/apple/foundationdb/relational/recordlayer/catalog/TransactionBoundDatabase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,6 @@
public class TransactionBoundDatabase extends AbstractDatabase {
BackingStore store;
URI uri;
@Nonnull
final Options options;

private static final MetadataOperationsFactory onlyTemporaryFunctionOperationsFactory = new AbstractMetadataOperationsFactory() {
@Nonnull
Expand All @@ -85,9 +83,8 @@ public ConstantAction getDropTemporaryFunctionConstantAction(final boolean throw
};

public TransactionBoundDatabase(URI uri, @Nonnull Options options, @Nullable RelationalPlanCache planCache) {
super(onlyTemporaryFunctionOperationsFactory, NoOpQueryFactory.INSTANCE, planCache);
super(onlyTemporaryFunctionOperationsFactory, NoOpQueryFactory.INSTANCE, planCache, options);
this.uri = uri;
this.options = options;
}

@Override
Expand Down
3 changes: 2 additions & 1 deletion .../apple/foundationdb/relational/recordlayer/ddl/RecordLayerCreateSchemaConstantAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
import com.apple.foundationdb.relational.recordlayer.RecordLayerConfig;
import com.apple.foundationdb.relational.recordlayer.RelationalKeyspaceProvider;
import com.apple.foundationdb.relational.recordlayer.catalog.CatalogMetaDataProvider;
import com.apple.foundationdb.relational.recordlayer.storage.StoreConfig;
import com.apple.foundationdb.relational.recordlayer.util.ExceptionUtil;

import java.net.URI;
Expand Down Expand Up @@ -101,7 +102,7 @@ public void execute(Transaction txn) throws RelationalException {
try {
FDBRecordStore.newBuilder()
.setKeySpacePath(databasePath)
.setSerializer(rlConfig.getSerializer())
.setSerializer(StoreConfig.DEFAULT_RELATIONAL_SERIALIZER)
.setMetaDataProvider(new CatalogMetaDataProvider(catalog, dbUri, schemaName, txn))
.setUserVersionChecker(rlConfig.getUserVersionChecker())
.setFormatVersion(rlConfig.getFormatVersion())
Expand Down
3 changes: 2 additions & 1 deletion ...apple/foundationdb/relational/recordlayer/ddl/RecordLayerSetStoreStateConstantAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import com.apple.foundationdb.relational.recordlayer.RecordLayerConfig;
import com.apple.foundationdb.relational.recordlayer.RelationalKeyspaceProvider;
import com.apple.foundationdb.relational.recordlayer.catalog.CatalogMetaDataProvider;
import com.apple.foundationdb.relational.recordlayer.storage.StoreConfig;
import com.apple.foundationdb.relational.recordlayer.util.ExceptionUtil;

import java.net.URI;
Expand Down Expand Up @@ -69,7 +70,7 @@ public void execute(Transaction txn) throws RelationalException {
FDBRecordStore recordStore =
FDBRecordStore.newBuilder()
.setKeySpacePath(databasePath)
.setSerializer(rlConfig.getSerializer())
.setSerializer(StoreConfig.DEFAULT_RELATIONAL_SERIALIZER)
.setMetaDataProvider(new CatalogMetaDataProvider(catalog, dbUri, schemaName, txn))
.setContext(txn.unwrap(FDBRecordContext.class))
.open();
Expand Down
81 changes: 75 additions & 6 deletions ...core/src/main/java/com/apple/foundationdb/relational/recordlayer/storage/StoreConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,42 +21,63 @@
package com.apple.foundationdb.relational.recordlayer.storage;

import com.apple.foundationdb.annotation.API;

import com.apple.foundationdb.record.RecordCoreException;
import com.apple.foundationdb.record.RecordMetaDataProvider;
import com.apple.foundationdb.record.metadata.MetaDataException;
import com.apple.foundationdb.record.provider.common.RecordSerializer;
import com.apple.foundationdb.record.provider.common.TransformedRecordSerializer;
import com.apple.foundationdb.record.provider.common.TransformedRecordSerializerJCE;
import com.apple.foundationdb.record.provider.foundationdb.FDBRecordStoreBase;
import com.apple.foundationdb.record.provider.foundationdb.FormatVersion;
import com.apple.foundationdb.record.provider.foundationdb.keyspace.KeySpacePath;
import com.apple.foundationdb.record.provider.foundationdb.keyspace.NoSuchDirectoryException;
import com.apple.foundationdb.relational.api.Options;
import com.apple.foundationdb.relational.api.Transaction;
import com.apple.foundationdb.relational.api.exceptions.ErrorCode;
import com.apple.foundationdb.relational.api.exceptions.RelationalException;
import com.apple.foundationdb.relational.recordlayer.RecordLayerConfig;
import com.apple.foundationdb.relational.recordlayer.RelationalKeyspaceProvider;
import com.apple.foundationdb.relational.recordlayer.catalog.RecordMetaDataStore;
import com.apple.foundationdb.relational.recordlayer.util.ExceptionUtil;

import com.google.protobuf.Message;

import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.security.NoSuchAlgorithmException;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.KeySpec;
import java.util.Base64;
import java.util.zip.Deflater;

@API(API.Status.EXPERIMENTAL)
public final class StoreConfig {
public static final RecordSerializer<Message> DEFAULT_RELATIONAL_SERIALIZER = TransformedRecordSerializer.newDefaultBuilder()
.setEncryptWhenSerializing(false)
.setCompressWhenSerializing(true)
.setCompressionLevel(Deflater.DEFAULT_COMPRESSION)
.setWriteValidationRatio(0.0)
.build();
private final RecordLayerConfig recordLayerConfig;
private final String schemaName;
private final KeySpacePath storePath;
private final RecordMetaDataProvider metaDataProvider;
private final RecordSerializer<Message> serializer;

private StoreConfig(RecordLayerConfig recordLayerConfig,
String schemaName,
KeySpacePath storePath,
RecordMetaDataProvider metaDataProvider) {
RecordMetaDataProvider metaDataProvider,
RecordSerializer<Message> serializer) {
this.recordLayerConfig = recordLayerConfig;
this.schemaName = schemaName;
this.storePath = storePath;
this.metaDataProvider = metaDataProvider;
this.serializer = serializer;
}

public String getSchemaName() {
Expand All @@ -72,7 +93,7 @@
}

public RecordSerializer<Message> getSerializer() {
return recordLayerConfig.getSerializer();
return serializer;
}

public FDBRecordStoreBase.UserVersionChecker getUserVersionChecker() {
Expand All @@ -87,7 +108,8 @@
String schemaName,
RelationalKeyspaceProvider.RelationalDatabasePath databasePath,
RecordMetaDataStore metaDataStore,
Transaction transaction) throws RelationalException {
Transaction transaction,
Options options) throws RelationalException {
//TODO(bfines) error handling if this store doesn't exist

RelationalKeyspaceProvider.RelationalSchemaPath schemaPath;
Expand All @@ -104,6 +126,53 @@
URI dbUri = databasePath.toUri();
RecordMetaDataProvider metaDataProvider = metaDataStore.loadMetaData(transaction, dbUri, schemaName);

return new StoreConfig(recordLayerConfig, schemaName, schemaPath, metaDataProvider);
RecordSerializer<Message> serializer = serializerFromOptions(options);

return new StoreConfig(recordLayerConfig, schemaName, schemaPath, metaDataProvider, serializer);
}

static RecordSerializer<Message> serializerFromOptions(Options options) throws RelationalException {
final boolean encrypted = options.getOption(Options.Name.ENCRYPT_WHEN_SERIALIZING);
if (!encrypted) {
return DEFAULT_RELATIONAL_SERIALIZER;
}
final SecretKey key;
final String keyBase64 = options.getOption(Options.Name.ENCRYPTION_KEY);
if (keyBase64 != null) {
key = new SecretKeySpec(Base64.getDecoder().decode(keyBase64), "AES");
} else {
// TODO: Is there a way to make this only available in YAML test framework?

Check warning on line 144 in fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/storage/StoreConfig.java

View check run for this annotation

fdb.teamscale.io / Teamscale | Findings

fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/storage/StoreConfig.java#L144 

TODO: Is there a way to make this only available in YAML test framework?
https://fdb.teamscale.io/findings/details/foundationdb-fdb-record-layer?t=FORK_MR%2F3557%2FMMcM%2Fyaml-test-encrypted-records%3AHEAD&id=2A9F8CE307E383D53F0BCFFC18295638
final String keyPassword = options.getOption(Options.Name.ENCRYPTION_PASSWORD);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a better testing strategy is to check in a key to the repo and have the test harness read the file and pass it in?
A concern here is that even if this is clearly marked as "test only" there is still room for abuse or inattentive use of the feature.

if (keyPassword != null) {
SecretKeyFactory kdf;
try {
kdf = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
} catch (NoSuchAlgorithmException ex) {
throw new RelationalException("KDF not found", ErrorCode.UNSUPPORTED_OPERATION, ex);
}
KeySpec ks = new PBEKeySpec(keyPassword.toCharArray(), "YAML-salt".getBytes(StandardCharsets.UTF_8), 1, 128);

Check warning on line 153 in fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/storage/StoreConfig.java

View check run for this annotation

fdb.teamscale.io / Teamscale | Findings

fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/storage/StoreConfig.java#L153 

Make this salt unpredictable
https://fdb.teamscale.io/findings/details/foundationdb-fdb-record-layer?t=FORK_MR%2F3557%2FMMcM%2Fyaml-test-encrypted-records%3AHEAD&id=F03E8B0763F13CD175173B69F8D995A9
try {
key = new SecretKeySpec(kdf.generateSecret(ks).getEncoded(), "AES");
} catch (InvalidKeySpecException ex) {
throw new RelationalException("Key derivation failed", ErrorCode.UNSUPPORTED_OPERATION, ex);
}
} else {
KeyGenerator keyGen;
try {
keyGen = KeyGenerator.getInstance("AES");
} catch (NoSuchAlgorithmException ex) {
throw new RelationalException("Key generator not found", ErrorCode.UNSUPPORTED_OPERATION, ex);
}
keyGen.init(128);
key = keyGen.generateKey();
Copy link
Contributor

@ohadzeliger ohadzeliger Aug 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the end goal of this code path? A one-time key that cannot be reused?

}
}
return TransformedRecordSerializerJCE.newDefaultBuilder()
.setEncryptWhenSerializing(true)
.setEncryptionKey(key)
.setCompressWhenSerializing(true)
.setCompressionLevel(Deflater.DEFAULT_COMPRESSION)
.setWriteValidationRatio(0.0)
.build();
}
}
Loading
Loading