Data Integrity

Checksums are written into the regular .dat storage files, not into a side file:

The checksum record is written in the same atomic write as the chunk it covers, so a chunk and its checksum can never get out of sync; the file header is written once, as a file’s first entry, when the file is created. The records are encoded as storage "gaps", which means older EclipseStore versions (and the housekeeping/garbage-collection logic) simply skip over them — a checksummed storage remains readable by an engine that does not know about the feature.

Verification happens once, at startup, while the storage initializes and walks its files. It is not repeated for individual lazy or cache loads during normal operation.

The algorithm used for writing is called the primary algorithm. Alongside it, the engine keeps a set of additional algorithms so that a file written by one algorithm can still be verified after the configured primary changes between runs.

A policy decides two things independently: whether to emit checksums on write, and whether to verify them on load. When verifying, each kind of anomaly the load walk can encounter is answered by a reaction:

The anomaly categories are: a checksum mismatch (the stored bytes no longer match their recorded checksum), a chunk-boundary mismatch (a framing desync, where a corrupted entity length rerouted the load walk past a covering record), an unknown record kind (a removed or future algorithm), a missing file header, and uncovered data (data with no covering checksum where coverage was expected).

Rather than assemble these by hand, pick a profile from StorageChunkChecksumPolicy:

The feature is configured through a StorageChunkChecksumProvider, which carries the primary algorithm and the policy. Set it on the StorageConfiguration:

StorageChunkChecksumProvider offers ready-made factories:

The feature is off unless configured; StorageChunkChecksumProvider.NewNone() (or simply New()) expresses that explicitly.

Opening a storage that was written before this feature existed is safe. Pre-feature data files carry no file header, so the engine treats them as legacy: it does not retro-fit checksums onto them, and a lenient policy ignores their missing coverage. Once the feature is enabled, newly created data files are written with checksums, so coverage grows naturally as the storage rolls over to new files. The two states coexist on disk without conflict.

To assert that an already fully migrated storage has complete coverage, switch to NewStrict(). While migrating, NewStrictTolerateLegacy() enforces integrity on new writes while only logging the pre-existing gaps.

A custom checksum algorithm is an implementation of StorageChunkChecksumCalculator.Algorithm. Supply it as the primary (and/or among the additional algorithms) via a factory:

An algorithm is supplied as a factory because each storage channel owns its own stateful instance.

When housekeeping cleans up storage files (see Housekeeping), it relocates the still-live data of a retired file into a new one. With checksums enabled, several small live runs are merged into a single coalesced chunk covered by one checksum, which keeps the checksum overhead of fragmented stores in check.

The soft target size of a coalesced chunk defaults to 1 MB and can be tuned via the additional StorageDataFileEvaluator parameter:

Chunk-checksum verification normally runs only at startup (during the load walk). To re-verify a running storage without restarting it, issue an on-demand integrity check on a StorageConnection (or the StorageManager, which is one):

The check re-walks the live data files of every channel, recomputes each chunk’s checksum and compares it against the stored record. Unlike startup verification it collects every anomaly into the result instead of throwing, so a single call surfaces all corruption found across all channels without halting the storage. The chunk-checksum anomalies route through the configured StorageChunkChecksumPolicy: anomaly types whose reaction is IGNORE are not reported, and a policy that does not verify reports nothing.

In addition, the budgeted variant verifies that the data files it snapshotted stay stable while it works: a sealed (non-head) file is immutable, so if its size changes between resume calls — truncation, growth or rewrite — that is reported as a FILE_SIZE_CHANGED anomaly (always, independent of the policy). Only the head file is exempt, since it legitimately grows.

The check is an issued operation, so it is serialized with stores like any other (see Housekeeping); stores are blocked for its duration. issueFullIntegrityCheck() runs to completion in a single call and sees a consistent, complete set of files as of the moment it runs. Each channel’s head file is verified up to its committed length at that moment (append-only data beyond that point is out of scope and is covered at the next startup or check).

A budgeted variant yields between calls so a long scan need not block stores for its whole duration:

The verification scope is snapshotted at the start of the scan. Because housekeeping may delete or relocate whole data files between budgeted calls, a file dissolved in the meantime is skipped (no false positive) but its relocated data is not re-checked in that run; budgeted coverage is therefore best-effort. For a guaranteed complete, consistent snapshot, use issueFullIntegrityCheck().

The verification pass is side-effect-free: it never alters the running write state (the head file’s chain tip and header are left untouched), so it is safe to run at any time.

The chained SHA-256 algorithm links each chunk’s hash to the previous one, which makes reordering, insertion or deletion of chunks within a file detectable. This linkage is bounded by EclipseStore’s normal housekeeping, and the boundary matters when reasoning about what corruption the chain can detect.

Housekeeping routinely deletes whole storage files: garbage collection retires a file once all of its entities have become unreachable, and file cleanup retires under-utilized files, relocating their still-live data into new files (see Housekeeping). Two consequences follow:

If verification finds a corrupted chunk under a FAIL reaction, the start aborts with a StorageExceptionChunkChecksumMismatch, which reports the affected file, the chunk position and the expected versus actual hash. (The on-demand integrity check never throws; it collects anomalies into its result instead — see On-demand Integrity Verification.)

Off-line repair tooling that writes into a checksum-protected storage but cannot produce a compliant covered file (for example, configured with a mismatching algorithm) fails loudly with a StorageExceptionChunkChecksumUnavailable rather than silently appending unprotected data.

Independent of chunk checksums, EclipseStore can validate at store time that every reference a store writes actually points to an existing entity.

A store’s data may contain references to object ids whose entities are not part of the store itself: the storer found the referenced instance in its object registry (or an unloaded Lazy reference carries a cached id) and trusts that its data already exists in the storage. That trust can be wrong — most commonly when the storage-level garbage collector reclaimed the entity after its last persisted reference was removed, while the Java instance survived in application memory and was later re-attached to the graph. Without validation, such a store silently persists a dangling reference that only surfaces after a later restart as:

With validation enabled, the trusted reference ids travel with the store to the storage engine, and each channel checks them against its entity registry inside the store task, before the data is written — atomically with the store, so the check cannot be raced by garbage collection.

The behavior is controlled by the reference-validation configuration property (see Properties) or StorageConfiguration.Builder#setReferenceValidationPolicy:

To recover from a rejected store, make the missing entities exist again and retry: include the referenced instances explicitly in the same store (e.g. storer.storeAll(parent, child)) or store them with an eager storer first. References registered via Storer#skip variants are an explicit user assertion and are not validated.

Reference validation guards new stores. A dangling reference that is already persisted (pre-existing corruption, crash artifacts, data written by older versions) is first noticed by the storage garbage collector’s marking, when it walks a binary record and hits an object id with no entity — a zombie object id. By default this is only logged as a warning (and reported via StorageEventLogger#logGarbageCollectorEncounteredZombieObjectId); the first hard failure then typically happens at a much later load or restart, after housekeeping may have physically reclaimed related data.

The gc-zombie-oid-handling configuration property (or EmbeddedStorageFoundation#setGCZombieOidHandler(StorageGCZombieOidHandler.Strict())) escalates this:

Treat any zombie warning as a data-integrity alarm: it means a dangling reference exists on disk, whatever its origin.