Tools

Storage Migrator

The Storage Migrator is a utility for migrating MicroStream projects to EclipseStore. It is based on OpenRewrite and can migrate source code (package names from one.microstream to org.eclipse.store / org.eclipse.serializer) and type dictionary files of existing storages.

The migrator is executed via Maven on the command line in your MicroStream project folder. The parameters eclipseStoreVersion and typeDictionaryRelativePath control what is migrated:

  • Source code migration is only done when eclipseStoreVersion is set.

  • Type dictionary migration is only done when typeDictionaryRelativePath is set.

Migrate Source Code and Type Dictionary

mvn org.openrewrite.maven:rewrite-maven-plugin:run \
  -Drewrite.activeRecipes=org.eclipse.store.storage.embedded.tools.storage.migrator.ConvertProject \
  -Drewrite.recipeArtifactCoordinates=org.eclipse.store:storage-embedded-tools-storage-migrator:4.2.0 \
  -DeclipseStoreVersion=4.2.0 \
  -Drewrite.plainTextMasks=**/*.ptd \
  -DtypeDictionaryRelativeFilePath=src/main/resources/PersistenceTypeDictionary.ptd

Migrate Source Code Only

mvn org.openrewrite.maven:rewrite-maven-plugin:run \
  -Drewrite.activeRecipes=org.eclipse.store.storage.embedded.tools.storage.migrator.ConvertProject \
  -Drewrite.recipeArtifactCoordinates=org.eclipse.store:storage-embedded-tools-storage-migrator:4.2.0 \
  -DeclipseStoreVersion=4.2.0

Migrate Type Dictionary Only

Copy the type dictionary file (PersistenceTypeDictionary.ptd from the root of your storage folder) into your project, then run:

mvn org.openrewrite.maven:rewrite-maven-plugin:run \
  -Drewrite.activeRecipes=org.eclipse.store.storage.embedded.tools.storage.migrator.ConvertProject \
  -Drewrite.recipeArtifactCoordinates=org.eclipse.store:storage-embedded-tools-storage-migrator:4.2.0 \
  -Drewrite.plainTextMasks=**/*.ptd \
  -DtypeDictionaryRelativeFilePath=src/main/resources/PersistenceTypeDictionary.ptd

After migration, the storage can be opened with EclipseStore.

Standalone JAR

The standalone JAR is not built by default. To build it locally, activate the migrator-standalone Maven profile in the EclipseStore repository:

mvn -Pmigrator-standalone clean package -pl storage/embedded-tools/storage-migrator -am

This produces a fat JAR with all dependencies included:

storage/embedded-tools/storage-migrator/target/storage-embedded-tools-storage-migrator-<version>-jar-with-dependencies.jar

You can then use this JAR as the recipe artifact coordinate in the migration commands above, or publish it to your local Maven repository:

mvn -Pmigrator-standalone install -pl storage/embedded-tools/storage-migrator -am

  • Always create a backup of your code and storage before running the migrator

  • The migration is a one-way operation and cannot be undone

  • The type dictionary migration only updates metadata — your actual data remains unchanged

For more information see the readme file.

Storage Converter

The Storage Converter copies a storage from one storage target to another. This is useful for:

  • Changing the number of storage channels

  • Moving storage from local disk to a cloud storage target or vice-versa

  • Converting the on-disk binary representation of persisted objects

  • Adding, changing, or removing chunk-checksum protection (see Chunk checksums)

Prerequisites

Add the storage converter dependency:

pom.xml
<dependencies>
	<dependency>
		<groupId>org.eclipse.store</groupId>
		<artifactId>storage-embedded-tools-storage-converter</artifactId>
		<version>4.2.0</version>
	</dependency>
</dependencies>

Standalone JAR

Build the standalone JAR by activating the converter-standalone Maven profile:

mvn -Pconverter-standalone clean package

Then run it with an external configuration file for each storage:

java -jar storage-embedded-tools-storage-converter-<version>-standalone.jar sourceConfig.xml targetConfig.xml

Java API

For more control, use the StorageConverter class directly with two StorageConfiguration instances:

StorageConfiguration source = StorageConfiguration.Builder()
	.setStorageFileProvider(
		StorageLiveFileProvider.New(
			NioFileSystem.New().ensureDirectoryPath("path", "to", "source", "storage")
		)
	)
	.createConfiguration();

StorageConfiguration target = StorageConfiguration.Builder()
	.setStorageFileProvider(
		StorageLiveFileProvider.New(
			NioFileSystem.New().ensureDirectoryPath("path", "to", "target", "storage")
		)
	)
	.setChannelCountProvider(StorageChannelCountProvider.New(4))
	.createConfiguration();

new StorageConverter(source, target).start();

To move from local file system to a cloud storage target, configure the target StorageConfiguration with the desired storage target. See Storage Targets for the available options.

  • The source storage must not be in use (not started) during the conversion

  • Always verify the target storage after conversion by starting it and validating your data

  • The converter creates a new independent copy — the source storage is not modified

For more information see the readme file.

Chunk checksums

The converter is chunk-checksum aware. What it writes is governed by the target configuration’s chunk-checksum provider, and what it verifies while reading is governed by the source configuration’s provider — independently. A single conversion can therefore add, change, remove, or re-establish chunk-checksum protection.

Output protection (target configuration)

By default a converted storage is chunk-checksum-protected just like a storage freshly written by the engine: every target data file is written with a file header and covering chunk-checksum records, honoring the target StorageDataFileEvaluator (its file-maximum-size drives file rollover, its coalesce-chunk-target-size drives how often a checksum record is written).

To produce an unprotected target (byte-shaped like a pre-feature storage), configure the target with the "none" provider:

StorageConfiguration target = StorageConfiguration.Builder()
	.setStorageFileProvider(...)
	.setChunkChecksumProvider(StorageChunkChecksumProvider.NewNone()) // no checksums emitted
	.createConfiguration();

A protected target reopens under any policy, including the strict one:

foundation.setConfiguration(
	StorageConfiguration.Builder()
		.setStorageFileProvider(...)
		.setChunkChecksumProvider(
			StorageChunkChecksumProvider.NewSha256Chained(StorageChunkChecksumPolicy.NewStrict()))
		.createConfiguration()
);

Source verification

When the source configuration’s policy verifies, the converter recomputes and checks every chunk-checksum record in the source as it reads, before writing the target. Corruption is caught up front rather than being silently copied through: a verifying policy aborts the conversion on a mismatch, an observe (log) policy reports it and continues. A source without checksums (a legacy storage) simply has nothing to verify.

Changing the checksum (re-keying)

Because the source and target providers are independent, one conversion can:

Source on disk Target provider Effect

none (legacy)

an emitting provider

add checksum protection

protected

NewNone()

remove protection

one algorithm

a different algorithm

change the algorithm

one algorithm

the same algorithm

re-establish full coverage

For example, to switch a chained-SHA-256 store to CRC32C, verify the input with a chained-SHA-256 source provider and emit CRC32C on the target:

StorageConfiguration source = StorageConfiguration.Builder()
	.setStorageFileProvider(...)
	.setChunkChecksumProvider(StorageChunkChecksumProvider.NewSha256Chained()) // verify the chained-SHA-256 input
	.createConfiguration();

StorageConfiguration target = StorageConfiguration.Builder()
	.setStorageFileProvider(...)
	.setChunkChecksumProvider(StorageChunkChecksumProvider.NewCrc32c()) // emit CRC32C
	.createConfiguration();

new StorageConverter(source, target).start();

To add protection to a legacy (unprotected) storage, use NewNone() for the source (nothing to verify) and an emitting provider for the target.

Verify-only

Construct the converter with a single source configuration (no target) to scan and verify a storage’s chunk checksums without writing anything:

new StorageConverter(source).start(); // verifies the source per its policy; writes nothing

On the command line, pass a single configuration file:

java -jar storage-embedded-tools-storage-converter-<version>-standalone.jar sourceConfig.xml

A verifying source policy exits with a non-zero status on the first mismatch; an observe policy reports and completes successfully.

The external configuration files do not (yet) carry a chunk-checksum setting. A conversion driven entirely by configuration files therefore uses the framework-default provider for both source and target — i.e. it is protected by default, but cannot be re-keyed from the command line. To choose a different target algorithm, the "none" provider, or an explicit source/target policy, use the StorageConverter Java API with hand-built StorageConfiguration instances.

Converting binary data

To convert the binary representation of persisted objects BinaryConverter implementations can be specified by the implementations full class name.

To use the supplied the BinaryConverterBitmapLevel2 converter:
java -jar storage-embedded-tools-storage-converter-<version>-standalone.jar src.ini dst.ini -c org.eclipse.store.storage.embedded.tools.storage.converter.BinaryConverterBitmapLevel2

If more than one BinaryConverter shall be applied the -c option, including the converters must be applied in quotation marks:

... "-c binaryConverter1, binaryConverter2"