The Logistics Events KIT is an essential component of the Basic Data Infrastructure (BDI), specifically designed to enable the exchange of logistics events and associated data across the BDI ecosystem. This KIT is vital for organizations that require real-time visibility and coordination within their logistics operations. Building on the secure foundation provided by the TRUST KIT, it utilizes the BDI control plane and Identity, Authentication, and Authorization (IAA) functions to ensure that event data is exchanged securely and efficiently.
The core building block of the Logistics Events KIT is the Pub/Sub service, which allows for the publication and subscription of logistics events. This service facilitates the timely and reliable distribution of event-related information, ensuring that all relevant parties receive critical updates as they happen. By integrating with the BDI control plane, the Logistics Events KIT ensures that all data exchanges comply with established security protocols and governance frameworks set by the TRUST KIT.
Implementing the Logistics Events KIT enables organizations to streamline their logistics processes, improve real-time decision-making, and enhance overall supply chain efficiency. It is a crucial addition for any BDI deployment that involves dynamic and event-driven logistics operations, ensuring that data flows seamlessly and securely throughout the system.
The "Event Demo" (Demo Phase 3) further advanced the "Trusted Goods Release" process by integrating real-time event notifications. This phase demonstrated how event-driven updates, particularly gate-out events, could be leveraged to improve logistics transparency and coordination. Utilizing an event broker (Apache Pulsar) and building on the iSHARE framework, this demo highlighted the importance of timely, secure data sharing in logistics processes.
In Demo Phase 3, the focus was on implementing an event-based notification system to communicate critical logistics events. Key components included:
Setup and Registers:
Implementation of Authorization and Assignment Registers to manage permissions and roles associated with each transport order.
Event Broker:
Use of Apache Pulsar as an event broker to handle event data propagation, allowing secure, role-based access to notifications.
Mockup Systems:
Simulated environments for ERP, Warehouse Management System (WMS), and Transport Management Systems (TMS) to incorporate event notifications and handle delegation to subcontractors.
Messaging and Events:
Focus on gate-out events, notifying the shipper when goods leave the distribution center. Events were managed through a dedicated topic per transport order, enhancing traceability and ensuring only relevant parties accessed event information.
Demonstration and Presentation:
The demo was presented in a series of project meetings, with discussions on implementation insights and future improvements.
Non-Functional Observations and Findings:
Insights from the demo led to suggested improvements in architecture, event handling, and component documentation.
Identified potential challenges, including scalability of the event broker, secure access to event data, and data protection strategies such as publishing only URLs to sensitive information instead of the data itself.
Explored alternative methods, such as webhooks and long polling, to further refine notification mechanisms.
The Event Demo demonstrated the feasibility of using real-time event notifications in logistics. By integrating event brokers and secure data-sharing protocols, this phase emphasized transparency and control in logistics processes, laying the groundwork for scalable, event-driven solutions in the industry.
You can find the document with lessons learnt attached to this article (in Dutch).
Documentation in GitHub https://github.com/Basic-Data-Infrastructure/demo-vertrouwde-goederenafgifte/tree/fase-3
Choreography of Principals and subcontractors: balancing effectivity with security.
In supply chains the chain of business activities starts when a Seller and Buyer agree upon the transaction. This agreement typically includes terms related to transport, insurance, customs, the handover of responsibilities, and payments. The successful execution of this agreement often requires coordination among a large set of actors, including authorities and their subcontractors. This coordination is managed through a "choreography" of actions, where each action is triggered by planned or executed events.
In this context, the Seller and Buyer serve as the Principals, each responsible for their part of the agreement. Typically, each Principal selects preferred contractors to fulfil their portion of the agreement. These main contractors, in turn, become Principals to their own subcontractors, and this chain continues down the line.
The key challenge here is distributing notifications within this dynamic and temporary data exchange network, to ensure effective and efficient coordination of activities, without overburdening the network with traffic.
The BDI framework assumes that commercial relationships between Principals and Subcontractors are established before any actual orders are placed. In this setup, URLs are known, and through the DNS discovery mechanism, the URIs of endpoints for each party are also known. Digital identity and trust are established within the respective associations of each party.
Each Principal is responsible for provisioning a temporary network of subcontractors associated with a specific order.
Principals and their subcontractors communicate through channels (a.k.a. topics in a pub/sub setting). There are two kinds of channels:
Subcontractor specific - often a principal works for different orders with a limited set of partners (preferred suppliers). There is a private channel between the principal and each partner. These channels are bidirectional (both the principal and the subcontractor can post notifications). A subcontractor specific channel exists as long as the relation between principal and subcontractor exist and hence may exceed the lifetime of a specific order.
Order specific - a principal creates a channel specific to the execution of a particular order. Such a channel is unidirectional: the principal is the only party who is allowed to post notifications, all involved subcontractors subscribe to this channel and are therefore notified of each message sent by the principal. Order specific channels exist for the duration of the order only.
The Principal publishes a specific notification to a subcontractor, with metadata on “Request order”. The data accessed by the subcontractor includes the details of the order request. The subcontractor responds with a notification containing metadata on order acceptance. The Principal then accesses the data to confirm the subcontractor's acceptance and any additional details provided.
Upon confirmation, the Principal adds the subcontractor as a subscriber to the common Order Log and posts this event in the Common Order Log. The Common Order Log creates notifications to all relevant parties when an event is posted by (or on behalf of) the Principal.
To secure and streamline interactions within the temporary network, the Principal sends a JWT (JSON Web Token) to the subcontractor. This "Subco" JWT contains:
Order Information: Specific details about the order.
Role of the Subcontractor: Defining the subcontractor's role within the order.
Data Access Rights: Permissions tied to the subcontractor’s role, aligned with an agreed-upon or standardized semantic model.
The subcontractor uses this JWT token when interacting with other subcontractors within the context of the order. This token allows other subcontractors to:
Validate the Subcontractor’s Role: Confirming that the subcontractor is part of the temporary network set up by the Principal.
Verify Access Rights: Ensuring that the subcontractor has the appropriate rights to access data associated with the order.
The Principal repeats this process with all subcontractors until the provisioning of the network is complete.
In real-world operations, it may become necessary to change subcontractors dynamically; removing one subcontractor from the network and replacing them with another.
To facilitate this, the Principal posts a notification of the change to the Order Log. This notification is then distributed to all parties involved in the network. The following steps are taken:
Remove the old subcontractor: The old subcontractor is removed from the subscription lists, cutting off their access to further notifications and data.
Revoke the old JWT Token: The JWT token that was issued to the old subcontractor is revoked, ensuring that they no longer have access to the network's resources.
Add the new subcontractor: The new subcontractor is added to the network, receiving the necessary notifications and access rights.
Create a JWT for the new subcontractor.
One potential pitfall of distributing notifications of events is the exponential increase in the number of interactions (API-calls, identity checks, authorization checks) with the number of participants/subcontractors in the temporary network.
There are three strategies to manage the number and costs of interactions.
A subcontractor communicates most of the notifications to the Principal, one-on-one. The Principal filters notifications and posts only relevant events or notifications from all parties to the Common Order Log.
This limits the information overload while ensuring that all relevant parties are informed of relevant events.
Notifications are sent to a closed network of parties. Some data of an event may be low-risk and can be embedded in the notification within the closed network: for example "ETA delayed by 12 hrs". The straightforward embedding reduces the need to collect the information at the source.
In this strategy, the value of the data is used to simplify the process, increasing efficiency and reducing costs. It is a business decision to balance the loss of value of the data and the gains of increased efficiency.
Common roles have known authorization rules. The results of data selection for these roles may be cached near the BDI-API, reducing the need to evaluate policies in the Authorization Register.
Once all tasks related to the order have been completed, the Principal generates a signed log. This log serves as an official record of all events and actions that took place during the order's execution.
Notification of Log Availability: A notification is sent to all parties involved, informing them that the signed log is available.
Access and Retention: Subcontractors can access this signed log and retain a copy for future reference, whether for compliance purposes or in the event of a dispute.
This structured audit trail ensures transparency, accountability, and the ability to review and verify all actions taken during the course of the order, providing a solid foundation for trust and compliance in the BDI framework.
Once all tasks related to the order have been executed, the Principal initiates the closure of the order by:
Distributing a Closure Notification: Informing all parties that the order is complete.
Removing Subscriptions: Subscriptions to the Order Log are removed after a specified period.
Revoking JWT Tokens: Invalidating the JWT tokens to dissolve the temporary network.
This process ensures the temporary network is securely dismantled, preserving the integrity and confidentiality of the data exchanged during the order's execution.
Event driven coordination requires a means to “go back in time”, inspect earlier events and have an audit trail. One obvious reason is when a subcontractor is changed: the new subcontractor has to be able to quickly come up to speed on the history of events. The second is to have a common reference for either compliance or settling of disputes.
Notifications of events are the digital representation of the result of events in the physical world: for example, ‘expected arrival time issued’, ‘container has been unloaded’. So, a notification of an event is the result of an event, not the event itself, events occur in the physical world, notifications of events occur in the digital world.
Notifications are published on channels (a.k.a. topics), to which all involved parties can subscribe. Following a publication of a notification, all subscribers will receive it.
Take the event that a full container has been loaded onto a container ship in China. All parties involved in the Netherlands, from the port authority, the container terminal, forwarders/processors, Customs, hinterland transporters to the shipper that ordered the goods, want to track the status of the ship and this container from that moment onward. And they create events because of this knowledge: a declaration, reservation of capacity, etc.
An important concept of the BDI is that involved parties can subscribe to channels and their ‘daughters’: they receive all notifications of events that are published to that channel. If required (and permitted) a party can request more data from the source. The concept of notifications of events and subscribing to them is broadly applicable: take the schedule for a building site for example. It is highly effective for all the suppliers and subcontractors if new things or changes to the schedule are automatically identified.
‘Event-driven’ communication is a way of restructuring the logistics of the information exchange between the IT systems of companies. Instead of a data owner sending messages when something of importance needs to be communicated (‘fire and forget’, ‘messaging’), all parties involved receive a signal (‘notification’) from the data owner that something relevant has happened (‘publish event to subscribers’).
That event contains metadata and a link to the source of the data. The receiving party evaluates the metadata and decides whether to follow the link to the source and access the data.
The Event Pub-Sub Service handles the centralized parts of this event-based communication. The actual data exchange happens directly between the parties in a federated manner.
Notifications trigger communication between decoupled services and are common in modern applications built with microservices. In the BDI, a notification corresponds to an event in the physical world.
Events occur in the physical world and are changes of state like a container being unloaded from a ship or a parcel delivered to its destination. Events can also be of a more administrative nature, such as the signing of a contract or the publication of a new expected arrival time.
Notifications may or may not carry data about the event (the item purchased, its price, or a delivery address).
Notifications are published to channels of the pub/sub service and are delivered to all subscribers on the channel.
A party that publishes a notification is called the producer, parties that receive the notification are called consumers.
The part of the infrastructure that is responsible for managing channels and notifications is called the router.
A producer publishes an event to the router, which filters and pushes the events to consumers. Producer services and consumer services are decoupled, which allows them to be scaled, updated, and deployed independently.
This approach has many advantages:
Efficiency:
No polling needed.
Low load on resources.
Effectiveness:
Easy to distribute notifications to many involved parties.
‘Single truth’ data at the source.
Synchronization of activities.
Control:
Distributing notifications with metadata only reveals relatively little information that can be abused.
Data access requires authentication. This increases the control over valuable data.
Authorization rules define what data can be accessed by what role/party.
All access to the data can be logged.
In supply chains the chain of business activities starts when a Seller and Buyer agree upon the transaction. This agreement typically includes terms related to transport, insurance, customs, the handover of responsibilities, and payments. The successful execution of this agreement often requires coordination among a large set of actors, including authorities and their subcontractors. This coordination is managed through a "choreography" of actions, where each action is triggered by planned or executed events.
The choreography describes which channels there are and which parties can subscribe to them. As the design of an appropriate choreography can be challenging and has major impact on the efficiency of the pub/sub service, this subject is discussed in detail in a separate page.
According to EPCIS (ISO/IEC 19987), a notification contains at least the following four aspects: what, where, when, and why and an optionally fifth: how.
What – to which object or entity does this event primarily relate (e.g. pallet, order, truck, wagon, etc.)?
Where – at which location did the event take place (warehouse receipt door, terminal access)?
When – on what date and time did the event take place?
Why – the reason (and in which business activity exactly) that the event took place (goods receipt, freight collection, transport document definitively agreed, etc.).
How – in what state (how) is or was the cargo being transported at the time of the event?
However, in line with BDI federation rules, data on these aspects is not always added directly to the notification. In instead, only a link to the source may be included, and interested parties can get the necessary data at the source.
There are links with the following building blocks:
Semantics
Data model
Data format
Data protocol
Zero Trust Check
The following pattern describes the typical interaction with the Pub-Sub service. First, we enter a configuration phase:
The data owner creates a new notification channel at the service. The channel name is the EventType, in this example, of the data the owner wants to share.
A data consumer asks permission to subscribe to the channel “EventType from Data Owner” using the service. The data consumer will then be notified if the data owner triggers a signal on the event channel he is interested in.
The request for subscription is communicated back to the data owner. The data owner decides if the request should be granted. Fi. based on several queries to the different registers part of the BDI like the Reputation and Qualification registers. Data and trust sovereignty means, in this case, that the owner always has control over who has access to his data.
In this case the data owner grants permission to the data consumer to subscribe to his EventType channel.
All is now setup for the actual event-based communication:
An event occurs at the data owner, and he sends a trigger through the channel “EventType from Data Owner” to all his subscribers.
The trigger is also sent to the data consumer who can use the embedded meta-data in the event trigger to decide if he wants to request the associated data of the event at the data owner.
The data consumer decides to request the data at the owner’s location using the link sent along as part of the trigger.
The owner can now do several checks to see if the data consumer is (still) allowed to access his data. In this case the owner agrees, and the data is sent as response to the query from the consumer.
Granularity of event channels
The granularity of the event channel is an important issue. Depending on the chosen strategy one would either create the need for extra filtering logic at the consumer or at the owner. The specific use case should be leading on how fine-grained the channels should be setup.
Multiple event brokers in a network
It is yet unclear how multiple event brokers, from different suppliers, would work together in a decentralized network. Most available open and commercial solutions do support multiple brokers but typically only the ones from the same supplier.
EPCIS (ISO/IEC 19987:2024) https://www.iso.org/standard/85557.html
