Changelog

4.2.0

Features

  • Storage restart: an embedded storage can be started again after it has been shut down [278] [701]

  • Data Integrity (opt-in, disabled by default)

    • Chunk checksums on data files, verified on startup and on demand [717] [728]

    • Store-time reference validation with optional self-healing [739] [742] [293] [294] [296] [302]

    • Configurable escalation for zombie object ids found by the garbage collector [743]

  • Power-loss durability: new synchronize() (fsync) barrier on the file system abstraction, applied on file rollover and commit [301] [751]

  • GigaMap

    • Annotation-based index generation for Lucene and JVector, including a compile-time index metamodel [692] [695] [699]

    • Structural index lifecycle: removal of indices and constraints across all index families, redefinition for bitmap indices [694]

    • reindex() to rebuild all indices from the current entity state, plus Lucene back-fill when an index is registered on a populated GigaMap [714] [716] [718] [724]

    • Entity-id based update() / apply(), so Lucene-only and JVector-only maps can be reindexed without a bitmap index [713] [722]

    • External-directory Lucene index commits are now coupled to store() instead of committing eagerly [715] [719]

    • Opt-in null embeddings for the vector index, and BitmapIndices.addWithoutInitialization for lazy registration without back-fill [748] [759]

  • Binary type handler for EqConstList [304] [306]

  • PersistenceShutdownReleasable, an opt-in contract to release resources of persisted instances on storage shutdown [305]

Bugfixes

  • Storage garbage collector: mid-cycle registration race, multi-channel data-loss windows, broken task chain after a failed prepended task, destructive re-execution of partially aborted sweeps, and dangling object ids caused by concurrent data imports [734] [736] [738] [746] [755] [770] [291]

  • Transactions file compaction reworked; the previous implementation could produce a transaction log that failed timestamp validation on the next start and destroyed history on a crash mid-rewrite [760]

  • Backup and rollback hardening: byte-count validation on all backup-relevant copy and write sites, in-flight copies superseded by a truncation, and store rollback consistency [744] [749] [775] [298]

  • Lazy references and storing: lazily skipped instances are pinned until commit, premature links are avoided, loader-built instances are published to the object registry only after completion, used-marked references are exempt from evaluator-driven clearing, and an abandoned batch storer no longer leaks its flush scheduler [289] [290] [292] [299] [307]

  • Storage engine: duplicate entity-cache entries on type id change, wrong hash key when rebuilding the type-in-file table, missing bounds check for variable-length members in BinaryReferenceTraverser, and a task broker reference that prevented auto-shutdown [282] [286] [726] [740] [741] [767]

  • GigaMap index correctness

    • Bitmap indices: Error on update/set/replace, exact-match results of binary composite indices, BinaryIndexerString collisions on keys containing NUL, null-valued Boolean-indexed entities, multi-value indexers dropping pre-existing shared values, notNull() on composite indexers, and remove with short key arrays [685] [686] [687] [688] [689] [690] [691] [756] [776]

    • Use-after-free and double-free in bitmap segment decompression [765] [773]

    • Stale indices after class evolution no longer delete a committed entity, and stale-key removal now fails loudly [762] [771]

    • Anonymous indexer name resolution after reload, and restart idempotence of generated @Unique constraints [693] [702]

  • GigaMap consistency and concurrency

    • Read-lock leaks and deadlocks during iteration; the iteration contract is now enforced [697] [700]

    • Silent lost updates: dirty segments are retained in release() and segments mutated by apply() / update() are pinned [737] [757]

    • Exceptions thrown from user code (indexers, update logic) no longer leave the map inconsistent, and replace() / apply() validate their current argument [705] [758]

    • Lucene reader is no longer reopened on the write path [777]

    • JVector: abandoned vector indices (thread and graph) are reclaimed, and the graph is no longer fully rebuilt on complete() or on shutdown persist [754] [764] [774]

  • Configuration: correct Redis file system URI prefix, Azure accountName typo (deprecated alias retained), and null guards for optional cloud storage credentials [732]

Other changes

  • Documentation on mutex lock behavior, continuous-backup restore consistency and the sweep-retry contract [732] [745]

  • Fixes for the example projects [733]

Migration

The transaction log compaction was reworked [760]. A transactions file compacted by 4.2.0 uses only the already existing entry kinds, so older versions can still read it.

While rewriting the log, however, 4.2.0 first puts the complete compacted content into a checksummed swap file next to the transactions file (<transactions file name>_compaction.swp) and heals an interrupted rewrite from it on the next start. Versions prior to 4.2.0 do not know this file. If a crash interrupts a compaction and the storage is then opened with an older version, the rewrite is not healed and the leftover swap file is ignored.

After a crash during compaction, open the storage with 4.2.0 once before downgrading, and do so in writable mode - a read-only start refuses to perform the repair write.

4.1.0

Features

  • Batch Storer for batched persistence [254] [638]

  • Storage object recovery [634]

  • Enhanced root initialization and retrieval in StorageManager [631] [632]

  • GigaMap

    • Sub-queries across all index types [637]

    • Harden JVector on-disk persistence and add internal architecture doc [657]

    • Make HNSW search beam width configurable [644]

    • Replace finalize() with java.lang.ref.Cleaner [673]

Bugfixes

  • Fix element count calculation issue [252]

  • Fix potential deadlocks in store / storeAll lock ordering [405] [414] [277] [678]

  • Fix flaky deadlock during shutdown in StorageBackupItemQueue [397] [671]

  • Seed live application OIDs into GC mark cycle to prevent zombie OIDs [669]

  • Fix type dictionary provider listener wiring [264]

  • Fix isUnchangedStructure always returning false [262]

  • Fix DynamoDB query and scan pagination to prevent data truncation [400] [680]

  • GigaMap fixes

    • Binary handler corrections (iterators + BitmapEntry metadata) [679]

    • Fix unique-constraint check missing duplicates after entity ejection [664]

    • Fix GigaMap builder for duplicate indexer usage [660]

    • Fix Condition.and() silently dropping the second range bound [653] [655]

    • Prevent leaking non-matching IDs in composite conditions [642]

    • Fix composite key updates and index exception handling [635]

    • Fix race condition in LuceneIndex GraphDirectory during store [628]

Other changes

  • Implemented cleanUp in ConstantSizedBufferProvider [636]

  • Add OffHeapLongHashSet and utility methods [255]

  • Improve clear method in LazyHashMap for better segment unloading [257]

  • Add demo projects [630]

  • JCache: support Cache.unwrap to JCache management MXBeans [672]

4.0.1

Bugfixes

  • Fix race conditions in VectorIndex incremental on-disk mode [620]

  • Resolve method ambiguities in indexer-related classes and interfaces [619]

4.0.0

Features

  • GigaMap

  • Native memory accessors to ensure future compatibility with upcoming Java versions [130] [225]

Bugfixes

Other changes

Migration

Storages created or modified by version 4.0.0 are not backwards compatible with older versions. Once a storage has been opened with version 4.0.0, it most likely cannot be used with prior versions anymore. Make sure to create a backup of your storage before migrating.

Storages created with one of the 4.0.0 beta versions that use the vector index are unfortunately not compatible with the final version due to necessary internal changes. These storages need to be recreated with the final 4.0.0 release.

3.1.0

Features

  • Add new query methods to persistence object registry [210] [212]

  • Add new collect method to binary loader [211]

  • Add Javalin REST service [486] [492]

  • New comparator-based indexer for GigaMap [481] [482]

Bugfixes

  • Fix incorrect loop condition in persistence iterator [216]

  • Refactor UsageMarkable.Default for thread-safe lazy initialization [218]

  • Fix synchronization in storage task broker [514]

Other changes

3.0.1

Bugfixes

  • Fixed type handlers to properly restore comparators in collections [202] [203] [352] [375]

  • Throw correct exception in storage file write [439] [443]

  • GigaMap fixes

    • Fix update of composite index [458] [463]

    • Fix update of multi-value index [444] [446]

    • Fix index method delegation [462] [466]

    • Add Lucene index auto commit [465] [467]

    • Inheritance clarification to ensure Eclipse compiler compatibility [441]

  • Delete unused stuff

    • XThreads#defaultHandleUncaughtThrowable [206]

    • Debug outputs [445]

3.0.0

Features

Bugfixes

  • Fix allocation of string concatenation during garbage collection [187]

  • Fix transaction file entry order [423]

  • Fix synchronization in housekeeping [395]

  • Fix for Google Cloud Firestore config in Spring Boot integration [427] [434]

Migration

  • EclipseStore now is built with Java 17 as target version. Ensure to update your configuration accordingly.

  • The module org.eclipse.serializer:persistence-binary-jdk17 was removed; all contained type handlers are used by default now

2.1.3

Changes

  • Add viewRoots() to PersistenceManager [178]

  • Add referential equality to Equalator [184]

  • Remove unused and deprecated stuff [179] [180]

  • Refactor CacheStore and improve backend code quality [390]

2.1.2

Bugfixes

  • Fix for potential NPE in lock scopes [172]

  • Fix possible NPE in LazyReferenceManager [171]

  • Prevent storage corruption [371] [319] [321]

  • OCI update [369] [374]

2.1.1

Bugfixes

  • Logger settings adjustments [164]

  • Dependency update to resolve vulnerability [360] [361]

  • Register LockAspect as Spring Bean [355] [356]

2.1.0

Features

Bugfixes

2.0.0

Features

  • Additional Type Handlers [72]

  • Enhanced Reloader [135] [136]

  • Various Storer Enhancements [143]

  • Ensure Correct Inventorisation of S3 Directories [250] [270]

Bugfixes

  • Adaptive Housekeeping Idle GC Fix [293]

  • Lock File Manager Fixes [281] [282]

  • LazyHashMap Fix [117] [139]

  • Add Missing Constructor Methods for CSV Converter [283]

  • Fix Intermittent Problem When Opening Database [264]

  • Fix Type Handler Foundation Registration Order Necessity [142] [204]

  • Fix for Allocation of LazyArgs During Cleanup [132] [133]

1.4.0

Features

  • S3 Express One Zone integration [224]

  • S3 Enhancements [233] [242]

  • Type Dictionary Import and Export for Serializer [128]

Bugfixes

  • S3 Fixes [234]

  • NIO Transfer Workaround [238]

1.3.2

Features

  • Spring Boot improvements [206]

  • Ensure Minio compatibility [197]

Bugfixes

  • Fix wrong entity layer service config [123]

  • Follow S3 batch delete limit [213]

1.3.1

Features

Bugfixes

  • Use MethodHandles.Lookup where possible [69]

  • Fix for BinaryStorer reinizialization [111]

1.3.0

Something went wrong with this release, some files are missing. Please use version 1.3.1 instead.

1.2.0

Features

  • Module-infos are back [144]

Bugfixes

  • Fixed memory cleanup [103]

1.1.0

Features

Bugfixes

  • DynamoDB connector fix [118]

1.0.0

This is the first release of EclipseStore, the successor of MicroStream. It is at the same state as the last MicroStream Storage release, 8.1.1. Just living in a different namespace.

Migration

Migration of MicroStream projects can easily be achieved with this tool.

It takes care of the proper rewrite of dependencies and packages.