Troubleshooting

Starting a Storage with Missing Objects

If the storage throws a StorageExceptionConsistency exception stating "No entity found for objectId", the storage is missing an object in the persisted data. In that case, it may be useful to start the storage and ignore that error.

Do not start a storage with missing objects unless you are absolutely certain what you are doing. Doing so may cause the storage garbage collector to delete further objects! And may cause other errors if code tries to use missing objects! Consider enabling the read only storage mode!

To start a storage that misses objects, you need to set a different StorageEntityCollector that ignores that exception:

final EmbeddedStorageFoundation<?> foundation = EmbeddedStorage.Foundation()
    .setStorageEntityCollectorCreator(StorageEntityCollector.Creator.Unchecked());

final EmbeddedStorageManager storage = foundation.start();

Identifying Missing Objects

A missing object cannot be identified directly, but it is possible to locate objects that reference the missing object. See Searching for Missing Objects for how to search for a missing object.

Overwriting an Existing Object by Its ObjectId

Overwriting an object by its ID does not check the object type!

To overwrite an object, you need to create a Storer instance that provides the Storer.store(instance, objectID) API:

Storer storer = storage.createStorer();
storer.store(new String("Overwritten object"), 1000000000000100000L);
storer.commit();

Common Error Messages

"Storage is already running"

This error occurs when trying to start a storage that is already started. Ensure you only call start() once, or call shutdown() before restarting.

// Check if storage is running before starting
if (!storage.isRunning())
{
    storage.start();
}

"StorageExceptionIo" or File Access Errors

File access errors typically indicate:

  • Permission issues — the application does not have read/write access to the storage directory

  • Disk full — the storage target has run out of space

  • Locked storage — another process has a lock file on the storage directory

To resolve lock file issues:

  1. Verify no other process is using the same storage directory

  2. If the previous process crashed without cleaning up, the lock file may be stale

  3. After verifying no other process is active, you can safely delete the lock file

"Class not found" or ClassLoader Errors

If the storage references types that are not on the classpath:

  • Ensure all types that were previously stored are available on the classpath

  • If classes were renamed or moved, configure legacy type mapping

  • If using a custom ClassLoader, see Custom Class Loader

OutOfMemoryError

If the application runs out of memory:

Restoring a Deleted Object with StorageObjectRestorer

If an object was deleted from the storage (e.g. through garbage collection) and a deletion directory is configured, it may be possible to recover the object using StorageObjectRestorer.

StorageObjectRestorer reads and writes the storage files directly. The storage must be shut down before running the restorer. Using it against a running storage will corrupt data.

Prerequisites

  • A deletion directory must have been configured before the object was deleted:

    EmbeddedStorage.Foundation()
    .onConnectionFoundation(f -> f.getStorageFoundation()
        .getFileProvider()
        .setDeletionDirectory(deletionDir));

    Without a deletion directory, superseded data files are permanently deleted and cannot be recovered.

  • You need to know the persistence object id of the object to restore.

Basic Usage

import org.eclipse.store.afs.nio.types.NioFileSystem;
import org.eclipse.store.storage.types.StorageLiveFileProvider;
import org.eclipse.store.storage.util.StorageObjectRestorer;

import java.nio.file.Paths;

// 1. Shut down the storage first!

final NioFileSystem nioFs = NioFileSystem.New();

final StorageLiveFileProvider fileProvider = StorageLiveFileProvider.Builder(nioFs)
    .setDirectory(nioFs.ensureDirectoryPath(Paths.get("/path/to/storage").toString()))
    .setDeletionDirectory(nioFs.ensureDirectoryPath(Paths.get("/path/to/deletions").toString()))
    .createFileProvider();

// channel count must match the value used when the storage was created (default: 1)
final int channelCount = 1;

final StorageObjectRestorer restorer =
    new StorageObjectRestorer.Default(fileProvider, channelCount);

final long objectId = 1000000000000068341L;
final boolean restored = restorer.restoreObject(objectId);

if (restored)
{
    System.out.println("Object " + objectId + " restored – restart storage to use it.");
}
else
{
    System.out.println("Object " + objectId + " not found in deletion directory.");
}

Alternatively, if you have a StorageConfiguration available:

final StorageConfiguration configuration = ...; // your existing configuration
final StorageObjectRestorer restorer = new StorageObjectRestorer.Default(configuration);
restorer.restoreObject(objectId);

How It Works

  1. The restorer scans the deletion directory for data files belonging to the channel that owns the given object id (objectId % channelCount).

  2. It parses every deleted data file and locates the most recent binary entity blob for the requested object id.

  3. The blob is appended as a new data file in the live storage directory of that channel.

  4. A matching entry is written to the channel’s transactions log so the storage engine recognises the new file on the next startup.

Limitations

  • The object can only be restored if its binary blob is still present in the deletion directory. If no deletion directory was configured, or if the deletion directory was cleaned up, recovery is not possible.

  • Only the object itself is restored. If the object references other missing objects, those must be restored separately or handled with StorageEntityCollector.Creator.Unchecked() (see above).

  • The restorer does not verify type compatibility. Ensure the class definition still matches the stored binary layout.

Storage Recovery Steps

If your storage becomes corrupted or is in an inconsistent state:

  1. Stop the application — do not attempt writes to a corrupted storage

  2. Create a backup — copy the entire storage directory before any recovery attempt

  3. Try starting in read-only mode — use the read-only storage mode to inspect the data

  4. Start with unchecked entity collector — if specific objects are missing, use StorageEntityCollector.Creator.Unchecked() as shown above

  5. Restore from backup — if the storage cannot be recovered, restore from a continuous backup or full backup