Changelog

4.3.0

Features

  • GigaMap JVector

    • Segmented on-disk vector index over a memory-mapped vector file [861]

    • Compression formats and in-memory PQ scoring, enablePqCompression now actually produces a compressed index, and update to jvector 4.0.0-rc.9 [825] [834]

    • Reduced boxing and allocation in search, disk graph writing and index bookkeeping [849] [850] [851] [852] [853] [854]

  • GigaMap: store(Persister) is widened to store(PersistenceStoring), so a Storer can be passed [789]

  • New VanillaJS viewer for the REST service [822]

  • Type handlers may supply one legacy handler per persisted layout [320]

Bugfixes

  • Storage

    • Power-loss corruption caused by a housekeeping transfer after a store [778]

    • Channel-barrier waiters are released on a disruption [781]

    • Storage lock-file manager reworked (lifecycle, exclusion, scheduling) [783]

    • Binary import and export failure handling hardened [792]

    • Reindex disables the missing files check [839]

    • S3 connector no longer hands the AWS SDK a single-use content stream provider [836]

  • Serializer

    • Storing and committing after a failed store is rejected until clear() is called [322]

    • LazyHashMap and LazyArrayList segments stay dirty until the store is committed, and storing unloaded LazyArrayList segments into another storage is rejected [321] [323]

    • Crash-safe type dictionary export, coalesced to one export per store commit [313] [315]

    • Direct buffers are kept reachable while in use [786] [311]

    • HashTable field offsets are read from HashTable instead of EqHashTable, and HashMapIdObject.toString() works on an empty map [312] [319]

  • GigaMap index correctness

    • Bitmap indices: total length mismatch on restart after removing entities from a compressed index, intact level2 block when compression fails, and reset of the add cache when a level2 block was rebuilt or freed [821] [866] [867]

    • An index failure during apply() / update() no longer deletes the committed entity [810]

    • Entity ids are derived with long arithmetic and no longer wrap at 2^31 [806]

    • Numeric binary indexers reject null keys with a designed exception instead of a raw NullPointerException [805]

    • The composite index key carrier is cleared before every indexer invocation [798]

    • Condition.and() / or() no longer modify the receiver chain, and a query registered as its own sub-query no longer recurses infinitely [802] [803]

    • The entity locator no longer misses entities in evicted lazy segments [799]

  • GigaMap consistency and concurrency

    • set() into a slot emptied by removeById no longer causes size drift or a ClassCastException, and a segment is released when a removal empties it [793] [797]

    • The equalator no longer decides whether set / replace writes at all [800]

    • Heap eviction no longer deletes from the backing store, and removeAll() purges evicted entries [801]

    • The Iterable passed to addAll and to constraint registration is materialized and traversed exactly once [788] [791]

    • Index registration waits for open readers, like entity mutation does [787]

  • GigaMap reindex and index registration

    • Registration and redefinition build the index before registering it, also for vector indices [812] [841]

    • reindex() re-validates unique constraints, no longer truncates the whole bitmap index group on failure, and is failure-safe for vector and Lucene indices [811] [814] [855] [863]

  • GigaMap JVector

    • Persisting an empty vector index is a no-op instead of a NullPointerException [824]

    • A newly registered vector index is populated exactly once, and deferred builder operations no longer add an ordinal twice after a persist [796] [809]

    • Lock order deadlock in the vector index [848]

    • A deleted node is re-inserted when an add reuses its id [865]

    • The result of a computed vectorizer is copied at the index boundary, so vectorizers may reuse a buffer [862] [863]

  • GigaMap Lucene: query(Query) on an empty map returns an empty result instead of throwing numHits must be > 0 [795]

Deprecations

  • afs-sql and afs-oraclecloud-objectstorage (including the afs-oraclecloud-parent module) are deprecated and will be removed in 5.0.0, there is no replacement

  • The Vaadin based REST client GUI (storage-restclient-app and storage-restclient-app-standalone-assembly) and the console UI of integrations-spring-boot3-console are deprecated and will be removed in 5.0.0, use the bundled Storage Viewer of storage-restservice-javalin instead [822]

Other changes

  • Dependency updates: Spring Boot, Jetty, Jackson, Vaadin and the AFS connector dependencies [815] [816] [817] [820] [858] [875]

  • The SparkJava REST service implementation was removed [819]

  • MMapDirectoryCreator is now final [840]

  • Documentation: storer and BatchStorer behaviour after a failed store, and corrected Lucene index code samples [784] [871]

4.2.0

Features

  • Storage restart: an embedded storage can be started again after it has been shut down [278] [701]

  • Data Integrity (opt-in, disabled by default)

    • Chunk checksums on data files, verified on startup and on demand [717] [728]

    • Store-time reference validation with optional self-healing [739] [742] [293] [294] [296] [302]

    • Configurable escalation for zombie object ids found by the garbage collector [743]

  • Power-loss durability: new synchronize() (fsync) barrier on the file system abstraction, applied on file rollover and commit [301] [751]

  • GigaMap

    • Annotation-based index generation for Lucene and JVector, including a compile-time index metamodel [692] [695] [699]

    • Structural index lifecycle: removal of indices and constraints across all index families, redefinition for bitmap indices [694]

    • reindex() to rebuild all indices from the current entity state, plus Lucene back-fill when an index is registered on a populated GigaMap [714] [716] [718] [724]

    • Entity-id based update() / apply(), so Lucene-only and JVector-only maps can be reindexed without a bitmap index [713] [722]

    • External-directory Lucene index commits are now coupled to store() instead of committing eagerly [715] [719]

    • Opt-in null embeddings for the vector index, and BitmapIndices.addWithoutInitialization for lazy registration without back-fill [748] [759]

  • Binary type handler for EqConstList [304] [306]

  • PersistenceShutdownReleasable, an opt-in contract to release resources of persisted instances on storage shutdown [305]

Bugfixes

  • Storage garbage collector: mid-cycle registration race, multi-channel data-loss windows, broken task chain after a failed prepended task, destructive re-execution of partially aborted sweeps, and dangling object ids caused by concurrent data imports [734] [736] [738] [746] [755] [770] [291]

  • Transactions file compaction reworked; the previous implementation could produce a transaction log that failed timestamp validation on the next start and destroyed history on a crash mid-rewrite [760]

  • Backup and rollback hardening: byte-count validation on all backup-relevant copy and write sites, in-flight copies superseded by a truncation, and store rollback consistency [744] [749] [775] [298]

  • Lazy references and storing: lazily skipped instances are pinned until commit, premature links are avoided, loader-built instances are published to the object registry only after completion, used-marked references are exempt from evaluator-driven clearing, and an abandoned batch storer no longer leaks its flush scheduler [289] [290] [292] [299] [307]

  • Storage engine: duplicate entity-cache entries on type id change, wrong hash key when rebuilding the type-in-file table, missing bounds check for variable-length members in BinaryReferenceTraverser, and a task broker reference that prevented auto-shutdown [282] [286] [726] [740] [741] [767]

  • GigaMap index correctness

    • Bitmap indices: Error on update/set/replace, exact-match results of binary composite indices, BinaryIndexerString collisions on keys containing NUL, null-valued Boolean-indexed entities, multi-value indexers dropping pre-existing shared values, notNull() on composite indexers, and remove with short key arrays [685] [686] [687] [688] [689] [690] [691] [756] [776]

    • Use-after-free and double-free in bitmap segment decompression [765] [773]

    • Stale indices after class evolution no longer delete a committed entity, and stale-key removal now fails loudly [762] [771]

    • Anonymous indexer name resolution after reload, and restart idempotence of generated @Unique constraints [693] [702]

  • GigaMap consistency and concurrency

    • Read-lock leaks and deadlocks during iteration; the iteration contract is now enforced [697] [700]

    • Silent lost updates: dirty segments are retained in release() and segments mutated by apply() / update() are pinned [737] [757]

    • Exceptions thrown from user code (indexers, update logic) no longer leave the map inconsistent, and replace() / apply() validate their current argument [705] [758]

    • Lucene reader is no longer reopened on the write path [777]

    • JVector: abandoned vector indices (thread and graph) are reclaimed, and the graph is no longer fully rebuilt on complete() or on shutdown persist [754] [764] [774]

  • Configuration: correct Redis file system URI prefix, Azure accountName typo (deprecated alias retained), and null guards for optional cloud storage credentials [732]

Other changes

  • Documentation on mutex lock behavior, continuous-backup restore consistency and the sweep-retry contract [732] [745]

  • Fixes for the example projects [733]

Migration

The transaction log compaction was reworked [760]. A transactions file compacted by 4.2.0 uses only the already existing entry kinds, so older versions can still read it.

While rewriting the log, however, 4.2.0 first puts the complete compacted content into a checksummed swap file next to the transactions file (<transactions file name>_compaction.swp) and heals an interrupted rewrite from it on the next start. Versions prior to 4.2.0 do not know this file. If a crash interrupts a compaction and the storage is then opened with an older version, the rewrite is not healed and the leftover swap file is ignored.

After a crash during compaction, open the storage with 4.2.0 once before downgrading, and do so in writable mode - a read-only start refuses to perform the repair write.

4.1.0

Features

  • Batch Storer for batched persistence [254] [638]

  • Storage object recovery [634]

  • Enhanced root initialization and retrieval in StorageManager [631] [632]

  • GigaMap

    • Sub-queries across all index types [637]

    • Harden JVector on-disk persistence and add internal architecture doc [657]

    • Make HNSW search beam width configurable [644]

    • Replace finalize() with java.lang.ref.Cleaner [673]

Bugfixes

  • Fix element count calculation issue [252]

  • Fix potential deadlocks in store / storeAll lock ordering [405] [414] [277] [678]

  • Fix flaky deadlock during shutdown in StorageBackupItemQueue [397] [671]

  • Seed live application OIDs into GC mark cycle to prevent zombie OIDs [669]

  • Fix type dictionary provider listener wiring [264]

  • Fix isUnchangedStructure always returning false [262]

  • Fix DynamoDB query and scan pagination to prevent data truncation [400] [680]

  • GigaMap fixes

    • Binary handler corrections (iterators + BitmapEntry metadata) [679]

    • Fix unique-constraint check missing duplicates after entity ejection [664]

    • Fix GigaMap builder for duplicate indexer usage [660]

    • Fix Condition.and() silently dropping the second range bound [653] [655]

    • Prevent leaking non-matching IDs in composite conditions [642]

    • Fix composite key updates and index exception handling [635]

    • Fix race condition in LuceneIndex GraphDirectory during store [628]

Other changes

  • Implemented cleanUp in ConstantSizedBufferProvider [636]

  • Add OffHeapLongHashSet and utility methods [255]

  • Improve clear method in LazyHashMap for better segment unloading [257]

  • Add demo projects [630]

  • JCache: support Cache.unwrap to JCache management MXBeans [672]

4.0.1

Bugfixes

  • Fix race conditions in VectorIndex incremental on-disk mode [620]

  • Resolve method ambiguities in indexer-related classes and interfaces [619]

4.0.0

Features

  • GigaMap

  • Native memory accessors to ensure future compatibility with upcoming Java versions [130] [225]

Bugfixes

Other changes

Migration

Storages created or modified by version 4.0.0 are not backwards compatible with older versions. Once a storage has been opened with version 4.0.0, it most likely cannot be used with prior versions anymore. Make sure to create a backup of your storage before migrating.

Storages created with one of the 4.0.0 beta versions that use the vector index are unfortunately not compatible with the final version due to necessary internal changes. These storages need to be recreated with the final 4.0.0 release.

3.1.0

Features

  • Add new query methods to persistence object registry [210] [212]

  • Add new collect method to binary loader [211]

  • Add Javalin REST service [486] [492]

  • New comparator-based indexer for GigaMap [481] [482]

Bugfixes

  • Fix incorrect loop condition in persistence iterator [216]

  • Refactor UsageMarkable.Default for thread-safe lazy initialization [218]

  • Fix synchronization in storage task broker [514]

Other changes

3.0.1

Bugfixes

  • Fixed type handlers to properly restore comparators in collections [202] [203] [352] [375]

  • Throw correct exception in storage file write [439] [443]

  • GigaMap fixes

    • Fix update of composite index [458] [463]

    • Fix update of multi-value index [444] [446]

    • Fix index method delegation [462] [466]

    • Add Lucene index auto commit [465] [467]

    • Inheritance clarification to ensure Eclipse compiler compatibility [441]

  • Delete unused stuff

    • XThreads#defaultHandleUncaughtThrowable [206]

    • Debug outputs [445]

3.0.0

Features

Bugfixes

  • Fix allocation of string concatenation during garbage collection [187]

  • Fix transaction file entry order [423]

  • Fix synchronization in housekeeping [395]

  • Fix for Google Cloud Firestore config in Spring Boot integration [427] [434]

Migration

  • EclipseStore now is built with Java 17 as target version. Ensure to update your configuration accordingly.

  • The module org.eclipse.serializer:persistence-binary-jdk17 was removed; all contained type handlers are used by default now

2.1.3

Changes

  • Add viewRoots() to PersistenceManager [178]

  • Add referential equality to Equalator [184]

  • Remove unused and deprecated stuff [179] [180]

  • Refactor CacheStore and improve backend code quality [390]

2.1.2

Bugfixes

  • Fix for potential NPE in lock scopes [172]

  • Fix possible NPE in LazyReferenceManager [171]

  • Prevent storage corruption [371] [319] [321]

  • OCI update [369] [374]

2.1.1

Bugfixes

  • Logger settings adjustments [164]

  • Dependency update to resolve vulnerability [360] [361]

  • Register LockAspect as Spring Bean [355] [356]

2.1.0

Features

Bugfixes

2.0.0

Features

  • Additional Type Handlers [72]

  • Enhanced Reloader [135] [136]

  • Various Storer Enhancements [143]

  • Ensure Correct Inventorisation of S3 Directories [250] [270]

Bugfixes

  • Adaptive Housekeeping Idle GC Fix [293]

  • Lock File Manager Fixes [281] [282]

  • LazyHashMap Fix [117] [139]

  • Add Missing Constructor Methods for CSV Converter [283]

  • Fix Intermittent Problem When Opening Database [264]

  • Fix Type Handler Foundation Registration Order Necessity [142] [204]

  • Fix for Allocation of LazyArgs During Cleanup [132] [133]

1.4.0

Features

  • S3 Express One Zone integration [224]

  • S3 Enhancements [233] [242]

  • Type Dictionary Import and Export for Serializer [128]

Bugfixes

  • S3 Fixes [234]

  • NIO Transfer Workaround [238]

1.3.2

Features

  • Spring Boot improvements [206]

  • Ensure Minio compatibility [197]

Bugfixes

  • Fix wrong entity layer service config [123]

  • Follow S3 batch delete limit [213]

1.3.1

Features

Bugfixes

  • Use MethodHandles.Lookup where possible [69]

  • Fix for BinaryStorer reinizialization [111]

1.3.0

Something went wrong with this release, some files are missing. Please use version 1.3.1 instead.

1.2.0

Features

  • Module-infos are back [144]

Bugfixes

  • Fixed memory cleanup [103]

1.1.0

Features

Bugfixes

  • DynamoDB connector fix [118]

1.0.0

This is the first release of EclipseStore, the successor of MicroStream. It is at the same state as the last MicroStream Storage release, 8.1.1. Just living in a different namespace.

Migration

Migration of MicroStream projects can easily be achieved with this tool.

It takes care of the proper rewrite of dependencies and packages.