Direct Transition outbox commit
- Atomic contents
- Candidate entity observation and zero or more canonical interaction envelopes.
- Replay authority
- The entity outbox retains stable emission identities and rejects conflicting content reuse.
Cohesive.Storage Internals
Commit Boundaries and Durable Evidence
How repository capabilities, Process checkpoints, Relations acquisition, and materialization progress preserve semantic operations across physical storage systems.
Why These Internals Exist
The Cohesive.Storage overview presents the application boundaries: repositories persist entity observations, source readers acquire Relation facts, durable stores retain Process progress, and materialization ports maintain derived views.
Each simple operation sits inside a harder persistence problem. A write may have an ambiguous result. A worker lease may expire. A source may redeliver a change. A target may accept only part of a batch. A restart must distinguish retained progress from work that never committed.
Storage contracts keep the identities, fences, checkpoints, and capability evidence needed to make those cases explicit.
Entity Repository Contract
IEntityRepository persists observations for one logical entity definition. An EntitySnapshot retains the observation, resolved partition key, opaque concurrency token, and optional loaded-field set.
EntityReadOptions keeps four concerns explicit:
- full or projected field selection
- expected logical version
- expected physical concurrency token
- an optional partition key for point-read routing
The logical version belongs to semantic entity state. The EntityConcurrencyToken is an opaque provider fence. A repository may use an ETag, row version, compare-and-swap revision, or another physical mechanism without exposing that representation to the caller.
Batch requests declare None, SamePartition, or AllOrNothing atomicity. EntityBatchCapabilities states native batching, supported atomicity, and maximum item count. A repository with only single-write fallback rejects a stronger request rather than emulating it with a partial sequence.
Typed wrappers use the shared Relations shape-mapping context to map CLR objects and observations. The raw observation remains the durable semantic carrier; object materialization is a projection at the repository edge.
Atomic Transition Boundaries
Storage supports two related commit protocols for entity Transitions:
Process-invoked Transition commit
- Atomic contents
- Candidate entity observation and the exact Process operation receipt, including outcome and handoff envelopes.
- Replay authority
- The entity repository replays the receipt; the Process checkpoint admits it and owns subsequent publication.
The direct path realizes an outbox at the entity commit boundary. The Process path avoids a dual write between authoritative entity state and the durable evidence needed to continue the Process.
EntityTransitionOperationCapabilities distinguishes repositories that can perform the atomic state-and-receipt operation from those that cannot. EntityTransitionProcessOperationAdapter resolves the exact Transition plan, repository, interaction catalog, subject policy, and request-target policy before evaluating and committing the decision.
Process Durable Aggregate
IProcessDurableStore persists a complete ProcessDurableCheckpoint rather than independent status, inbox, outbox, and job tables with unrelated consistency.
One finite Process activation
Load exact checkpoint and definition affinity, then Acquire worker fence, then Advance a bounded activation, then Commit aggregate successor, then Replay or continue
An expected ProcessStorageRevision provides compare-and-swap. A ProcessWorkerFence rejects stale owners. ProcessCommitId plus a deterministic content fingerprint makes an ambiguous exact retry replayable and makes conflicting identity reuse invalid.
Inbox admission advances the same aggregate revision even when no worker is active, closing the race between wait registration and early input. Terminal inbox dispositions are monotonic. Activation and host-operation receipts retain exact occurrence identity so reconstitution does not repeat completed logical work.
ProcessDurableRuntime is the Storage-owned reference driver over this contract. It composes the Process interpreter, lifecycle control, durable request executor, operation adapters, and store without redefining Process semantics.
Canonical Relation Sources
Storage source readers implement physical acquisition for exact graph-qualified shapes. A registration binds:
- one source identity and execution domain
- an
IRelationQuerySourceReader - a target capability profile
- hard batch, buffering, fanout, and concurrency limits
- identity, field, and relationship-reference selectors
The catalog permits one source per graph-qualified shape and rejects duplicate source authority. Registration order does not select a winner.
Readers return completeness and provenance with their rows. Authoritative absence, partial evidence, failure, and inconclusive evidence remain distinct. The canonical Relations evaluator owns semantic interpretation after acquisition.
Current production readers include Cosmos DB and PostgreSQL. Their provider types, persisted bindings, query syntax, and SDK failures remain in adapter packages rather than entering the Storage port.
Relation-Derived Materialization
A MaterializationDocument retains the exact Relation compilation request, compiled-plan fingerprint, and selected output. Loading recompiles that request and fails closed when the plan, selected output, or acquisition contract has drifted.
The Relations requirement graph, dependency manifest, and lineage remain dependency authority. Storage projects them into a fingerprinted impact plan without copying or inventing dependency edges.
The current impact strategies are closed and bounded:
- a non-set root change can map directly to that root
- contributor changes can follow canonical relationships toward roots using complete predicate reads or before-and-after references
- a contributor ledger is exact only when its prior associations and current relationship resolution are complete
- bounded global invalidation is the conservative fallback when explicitly admitted
Set-valued results, missing before-images, unsupported inverse paths, absent capabilities, or exceeded item and byte bounds fail instead of truncating affected work.
Progress, Delivery, and Settlement
Materialization keeps three progress concepts separate:
Source position
- Meaning
- An opaque provider cursor bound to the exact physical plan, source placement, partition, ordering scope, and request fingerprint.
Application checkpoint
- Meaning
- What one materialization definition and target generation durably applied under compare-and-swap and a worker fence.
Source settlement
- Meaning
- The provider acknowledgement performed only after the corresponding application checkpoint is durable.
The invariant order is:
- apply target effects
- commit the application checkpoint
- settle the source
- retain the settlement observation
A crash after application and before settlement may redeliver the same batch. Stable change, delivery, and mutation identities plus the replayed checkpoint make that safe. Settlement cannot make uncommitted target work appear complete.
Pull sources expose caller-owned positions. Managed sources keep provider acknowledgement inside the adapter callback boundary. Both produce the same canonical change pages and deliveries; provider leases, bookmarks, and consumer-group identities remain physical evidence rather than application progress.
Target Generations and Routing
IMaterializationTarget writes fenced, idempotent, version-aware mutations into an isolated generation. A rebuild candidate moves through loading, sealing, validation, promotion, retirement, and cleanup without exposing incomplete data.
Promotion uses an active-pointer compare-and-swap distinct from the generation worker fence. Abandonment can install a tombstone before physical state exists, preventing delayed begin or write operations from reviving a superseded Process attempt.
Backend routing is scoped to an exact placement-slice reference and fingerprint. Membership changes, merges, splits, or cross-slice cleanup require explicit authority; a router never infers that two placement cuts are equivalent.
The current high-level executor supports independent promotion. Progressive and atomic-visibility policies remain declared semantic requirements that need their corresponding durable parent coordination before execution can claim them.
Bounded Lifecycle Control
The Cohesive.Control namespace is currently incubated inside Storage. It models adaptive operating points for concurrency, batching, rates, buffers, and reserved workload capacity without embedding a particular semaphore, timer, queue, or provider SDK.
The AIMD reference regulator consumes explicit observations and time, then returns a recommendation with complete state and evidence. A recommendation becomes effective only at an authorized safe point such as work admission or a batch boundary. Hard limits from semantic, compiler, adapter, and deployment evidence intersect; the controller cannot override them.
Materialization index synchronization binds these loops to exact definition, plan, target, generation, workload, and epoch identities. Pause and continue retain the epoch. Process restart creates a new generation and a fresh control epoch.
Explain and Telemetry
Storage projects execution evidence into the shared explain model. Process explain output adds active waits, operation and attempt lineage, checkpoint state, and input-required diagnostics. Materialization explain adds routing, source progress, backlog, lag, generation health, measured control pressure, and effective operating points.
These projections remain payload-safe and attributable. Runtime observations are marked as measured evidence and cannot silently change the authored definition. Low-cardinality telemetry describes operations and failures without turning provider-specific metrics into semantic authority.
Removed and Current Boundaries
The former IObservationStream, ObservationBatchContext, ObservationRecord, and IChangeStreamRepository contracts have been removed. Change consumers now bind an entity or outbox shape to a materialization change source and retain application progress before acknowledging delivery.
The former query-repository compatibility facade and observation adapters have also been removed. Cohesive.Relations is the sole query authority; Storage provides source readers and physical execution evidence.
Current reference implementations are deliberately explicit about their limits:
InMemoryProcessDurableStoreis a test oracle, not production durability.InMemoryMaterializationSource, progress store, target, and router are conformance fakes.- PostgreSQL currently supplies Relations and materialization source paths, not a general entity repository.
- DurableTask supplies Process status projection, not canonical execution or checkpoint storage.
- Forced Process termination and several higher-level materialization promotion modes still require additional durable coordination surfaces.
These boundaries matter because storage systems provide different mechanisms. Adapter support is admitted only for the guarantees it can currently prove.