Event Versioning and Upcasting in Event-Sourced Systems
Concept
In event-sourced architectures, the application's state is computed by replaying a sequential log of events from an append-only store. Because this event log is immutable and serves as the permanent audit trail, developers cannot perform standard database migrations like executing an ALTER TABLE script. Old events must remain in the store in their original format.
However, business logic and data structures evolve. Fields must be added, renamed, split, or deleted. To reconstruct the state of an aggregate from historic events, the application must resolve the differences between old event schemas and new domain models. This article explains how to address this challenge using Event Upcasting, mapping directly to the principles in Module 12: CQRS & Event Sourcing.
Event Store ──> [ Raw V1 Event (JSON) ]
│
▼
[ Upcaster V1 to V2 ] ──> [ Raw V2 Event (JSON) ]
│
▼
[ Upcaster V2 to V3 ] ──> [ Raw V3 Event (JSON) ]
│
▼
[ Deserialize ]
│
▼
[ Domain Model V3 ]
│
▼
Aggregate Root
The Upcasting Pattern
Upcasting is a programmatic translation technique where old events are upgraded to the current schema in-memory as they are read from the database, before they are deserialized and loaded by the aggregate root.
The database continues to store the historical event exactly as it was written (e.g., version 1). When the application requests the event stream for a specific aggregate:
- The event store client reads the raw bytes (typically JSON or BSON).
- The pipeline intercepts the raw event and inspects its metadata (e.g., version flag).
- The event is routed through a chain of Upcasters (e.g., V1 $\rightarrow$ V2, then V2 $\rightarrow$ V3).
- The final, fully upgraded raw payload is deserialized into the current strongly typed class representation (V3).
- The aggregate processes only the V3 event class, insulating it from historical schema changes.
Constraints
Architects must navigate physical CPU and memory constraints when introducing transformation layers into the read pipeline.
1. In-Memory Processing Latency and Allocation Overhead
Because upcasting occurs dynamically on the read path, every event replayed from the database must pass through the upcaster chain.
- If an aggregate has a long lifecycle (e.g., a customer account with 10,000 events), running upcasters that parse and modify JSON structures dynamically (using
JsonDocumentor string operations) creates significant CPU overhead and garbage collector allocation pressure. - Mitigation: Implement snapshotting. By saving the aggregate's state at regular intervals, the application only needs to replay events that occurred after the last snapshot, bypassing historical upcasting loops.
2. The Chain Dependency Constraint
Upcasters must form a contiguous, non-breaking chain of transformations (V1 $\rightarrow$ V2 $\rightarrow$ V3 $\rightarrow$ V4).
- If a developer deletes the V1 $\rightarrow$ V2 upcaster, assuming V1 events are "too old to care about," any aggregate initialized prior to the schema evolution will fail to load, resulting in system downtime and data corruption errors.
3. Serialization Protocol Immutability
Dynamic upcasting is highly dependent on the chosen serialization format.
- JSON is highly compatible with upcasting because it can be parsed as a dynamic document tree, modified structurally (e.g., renaming a property), and passed down the chain.
- Binary Formats (such as Protobuf or Avro) are much harder to dynamically upcast. Because they rely on compiled schemas and field tags, upcasting requires deserializing the binary payload into an intermediate representation, transforming it, and reserializing it, which degrades performance.
Trade-offs
Choosing how to handle schema changes in event-sourced systems involves balancing write simplicity, read performance, and architecture longevity.
1. Dynamic Upcasting vs. Upcast-on-Write (Migration)
- Dynamic Upcasting (On-Read): Upcasts events on-the-fly during read operations.
- Trade-off: Zero write overhead, and the database remains 100% immutable. However, it increases read latency and means the application code must forever carry legacy upcaster classes.
- Upcast-on-Write (Copy and Migrate): A background process reads old events, upcasts them, and writes the new versions to a new stream, deprecating the old stream.
- Trade-off: Keeps read paths fast and clean. However, it requires complex stream management, doubles storage requirements during migrations, and violates the "append-only immutable ledger" ideal.
2. Weak Schema Conformance vs. Multi-Version Support
- Weak Schema Conformance: Design events to be extremely generic (e.g., using maps/dictionaries of properties).
- Trade-off: Flexible and rarely requires upcasting. However, it removes compiler type-safety, making the code harder to maintain and prone to runtime errors.
- Explicit Multi-Version Classes: Create separate classes in code for each version of an event (e.g.,
UserRegisteredV1,UserRegisteredV2).- Trade-off: Extremely type-safe and clear. However, it results in codebase bloat, as the domain project must retain all historical event classes.
Implementation: Chain-of-Responsibility Upcasting Pipeline in C#
The following C# code demonstrates how to build a dynamic event upcasting pipeline. It parses raw JSON payloads representing historical versions of a UserRegistered event, sequentially routing them through version-specific upcasters to yield the current class structure.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Text.Json.Nodes;
namespace Mpc.EventSourcing.Upcasting
{
// Current event representation expected by the aggregate (Version 3)
public record UserRegistered(
Guid UserId,
string FullName, // Combined FirstName & LastName (V2 change)
string Email,
string PreferredLanguage, // Added with default (V3 change)
DateTime RegisteredAtUtc
);
// DTO for raw event representation in the store
public class RawEvent
{
public string EventType { get; set; }
public int Version { get; set; }
public string PayloadJson { get; set; }
}
// Upcaster interface operating on raw JSON nodes
public interface IEventUpcaster
{
string TargetEventType { get; }
int SourceVersion { get; }
int TargetVersion { get; }
JsonNode Upcast(JsonNode sourcePayload);
}
// Upcaster for Version 1 to Version 2: Combines FirstName & LastName into FullName
public class UserRegisteredV1ToV2Upcaster : IEventUpcaster
{
public string TargetEventType => "UserRegistered";
public int SourceVersion => 1;
public int TargetVersion => 2;
public JsonNode Upcast(JsonNode sourcePayload)
{
var firstName = sourcePayload["FirstName"]?.ToString() ?? string.Empty;
var lastName = sourcePayload["LastName"]?.ToString() ?? string.Empty;
// Remove old keys
var obj = sourcePayload.AsObject();
obj.Remove("FirstName");
obj.Remove("LastName");
// Add new unified key
obj["FullName"] = $"{firstName} {lastName}".Trim();
return obj;
}
}
// Upcaster for Version 2 to Version 3: Adds PreferredLanguage with a default value
public class UserRegisteredV2ToV3Upcaster : IEventUpcaster
{
public string TargetEventType => "UserRegistered";
public int SourceVersion => 2;
public int TargetVersion => 3;
public JsonNode Upcast(JsonNode sourcePayload)
{
var obj = sourcePayload.AsObject();
// Set default value for new field
obj["PreferredLanguage"] = "en-US";
return obj;
}
}
// Orchestrator processing event streams
public class EventUpcastEngine
{
private readonly Dictionary<(string EventType, int SourceVersion), IEventUpcaster> _upcasters;
public EventUpcastEngine(IEnumerable<IEventUpcaster> upcasters)
{
_upcasters = upcasters.ToDictionary(
u => (u.TargetEventType, u.SourceVersion),
u => u
);
}
public object DeserializeAndUpcast(RawEvent rawEvent)
{
var jsonNode = JsonNode.Parse(rawEvent.PayloadJson);
var currentVersion = rawEvent.Version;
// Chain upcasters until no matching upcaster is found
while (_upcasters.TryGetValue((rawEvent.EventType, currentVersion), out var upcaster))
{
jsonNode = upcaster.Upcast(jsonNode);
currentVersion = upcaster.TargetVersion;
}
// Once fully upcasted to version 3, deserialize to current domain class
var options = new JsonSerializerOptions { PropertyNameCaseInsensitive = true };
if (rawEvent.EventType == "UserRegistered" && currentVersion == 3)
{
return JsonSerializer.Deserialize<UserRegistered>(jsonNode.ToJsonString(), options);
}
throw new InvalidOperationException($"Unsupported event type {rawEvent.EventType} at version {currentVersion}");
}
}
}