taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

commit ac4d9ab35788e12d6b7bd517c41c149a89d45298
parent 399483707af4911397c26c0622b38ff22e617aa5
Author: Antoine A <>
Date:   Fri,  6 Nov 2026 16:27:40 +0100

DD104 new proposal

Diffstat:
Mdesign-documents/104-wire-gateway-management-API.rst | 367++++++++++++++++++++++++++++++++++++++++++-------------------------------------
1 file changed, 197 insertions(+), 170 deletions(-)

diff --git a/design-documents/104-wire-gateway-management-API.rst b/design-documents/104-wire-gateway-management-API.rst @@ -6,233 +6,261 @@ DD 104: Wire Gateway Management API :DD shepherd: Antoine d'Aligny :Historical contributors: Antoine d'Aligny :First published: 2026-09-18 -:Last substantive change: 2026-09-18 +:Last substantive change: 2026-11-06 Summary ======= -This design document presents the challenges we currently face when handling -transaction failures in wire gateway adapters and proposes a solution for -addressing them. +Extend the Wire Gateway API so administrators can inspect raw transactions +and follow outgoing operations, including failures. Transfers, bounces and +ordinary payments share one operation resource. + +This is a working design document. It describes the new resources and their +relationships, detailed protocol specification will follow later. Motivation ========== -In the current design and implementations, both incoming and outgoing -transactions can fail without administrators being notified or having the -ability to take corrective action. +Some incoming payments never reach Taler history, and some outgoing payments +fail without enough information being available for reconciliation. A bounce +may also be skipped or blocked before any outgoing payment exists. + +Listing operations by creation ID is insufficient for monitoring: older +operations can change status after newer ones. We need current records and a +history of operation statuses. Requirements ============ -* The solution must allow administrators to see failures, understand their - causes, and possibly fix them. -* The solution must be lightweight to implement in wire gateway adapters, as - the goal of this API is to provide a thin wrapper around how each - implementation works. Common logic that can be extracted into another - component should be. +* Expose failures, their reasons and the identifiers needed for reconciliation. +* Include incomplete transactions and decisions not to bounce. +* Share the management model across adapters while keeping their native + execution mechanisms and storage models. +* Reuse existing Wire Gateway conventions, identifiers and stored fields. Proposed Solution ================= -Terminology ------------ - -* `in_tx`: final incoming transaction -* `out_tx`: final outgoing transaction -* `talerable_in_tx`: talerable final incoming transaction (RESERVE | KYC | MAP) -* `talerable_out_tx`: talerable final outgoing transaction (deposit) -* `transfer`: initiated talerable outgoing transaction -* `initiated`: initiated outgoing transaction -* `bounce`: bounced invalid outgoing transaction - -Current API State ------------------ - -We already have endpoints for querying Talerable transactions: -`talerable_in_tx` with `GET /history/incoming` and `talerable_out_tx` with -`GET /history/outgoing`. - -We also already have endpoints for querying `transfer` with `GET /transfers` -and `GET /transfers/$ROW_ID`. +API placement +------------- + +Add read-only endpoints under ``/management`` in the Wire Gateway API. Reuse +its account scope, authentication, types and response conventions. +TODO: should we add a new management API with different auth ? + +All lists use the shared :ref:`row-id-pagination` convention. Operation status +history also supports the existing :ref:`long-polling` convention. Only the new +fields and filters are described below. + +.. list-table:: Proposed endpoints + :header-rows: 1 + :widths: 55 45 + + * - Endpoint + - Purpose + * - ``GET /management/operations`` + - List operations, filter by kind or status. + * - ``GET /management/operations/$OPERATION_ID`` + - Inspect an operation. + * - ``GET /management/operations/status-history`` + - Follow status and message changes, filter by operation ID. + * - ``GET /management/incoming`` + - List raw incoming records, filter by kind. + * - ``GET /management/incoming/$ROW_ID`` + - Inspect an incoming record. + * - ``GET /management/outgoing`` + - List raw outgoing records. + * - ``GET /management/outgoing/$ROW_ID`` + - Inspect an outgoing record. + +Operation lists paginate by ``operation_id``, transaction lists and status +history paginate by their respective ``row_id``. Status-history IDs are +independent of operation IDs. Reuse existing local record IDs where possible. +Links to transfers and Taler history use the existing APIs' IDs. + +Incoming and outgoing endpoints expose current records. Tracking later +completion or reconciliation requires re-reading these records, their change +feeds are deferred. + +Outgoing operations +------------------- + +An operation represents an outgoing intent, even if the adapter implements it +as a native chargeback. A bounce stored as an initiated payment appears once. +Submission retries retain the same operation ID. + +``transfer`` originates from the Wire Gateway transfer API. ``bounce`` returns +an unsuitable incoming payment. ``payment`` is an ordinary payment initiated +through an adapter-specific tool, such as ``libeufin-nexus initiate-payment``. +This management API does not create payments. + +A bank payment observed without a local intent belongs in ``outgoing`` and +has no operation. + +``pending``, ``transient_failure``, ``permanent_failure`` and ``success`` +follow the existing transfer model. Submission or batch acceptance alone +does not mean success: execution must be confirmed. +``late_failure`` records a reversal or failure after success. These management +statuses do not change the existing ``/transfers`` response types. + +.. ts:def:: ManagementOperation + + type ManagementOperation = { + operation_id: SafeUint64; + status: "pending" | "transient_failure" | "permanent_failure" + | "success" | "late_failure"; + status_msg?: string; -What is missing is access to raw (incompled, malformed, etc) transactions, `in_tx` and `out_tx`, as well -as initiated and bounced transactions, `initiated` and `bounce`. + amount?: Amount; + credit_account?: string; + subject?: string; -Generic API Design ------------------- + // Existing link to the reconciled outgoing transaction, when known. + outgoing_transaction_id?: SafeUint64; -Each adapter works differently. In most adapters, `transfer` and `bounce` -produce an `initiated` operation, while other payment systems, such as -Cyclos, provide a proper way to perform a bounce idempotently and have a -specific bounce operation. + // Initiation time + created_at: Timestamp; + details: { [name: string]: string }; + } & ( + | { kind: "transfer"; transfer_id: SafeUint64 } + | { kind: "bounce"; incoming_transaction_id: SafeUint64 } + | { kind: "payment" } + ); -Each adapter also has different internal unique identifiers. For example, -`libeufin_nexus` uses three of them while `depolymerizer-bitcoin` has a -single transaction ID. +``kind`` and its associated ID are top-level fields. Transfers require +``transfer_id``, bounces require ``incoming_transaction_id``, and ordinary +payments have neither. The two source IDs are mutually exclusive. -We need an API that can expose enough detail for manual reconciliation while -remaining generic enough to reuse the same logic across all adapters. +The adapters store one reconciled outgoing transaction per operation. +Additional native references can be exposed in ``details``. Status-change +times belong to status-history entries, no separate current-status timestamp +needs to be maintained. -For `transfer`, we already have the paginated `GET /transfers` endpoint and -the single-entry `GET /transfers/$ROW_ID` endpoint. The problem with this -design is that it is not well suited to following the progress of transfers. -Transfers do not progress or reach finality in the order in which they are -created. +Unknown payment fields may be omitted. Available +failure explanations are exposed in ``status_msg``. An uncertain outcome, +such as a submission timeout, must not be reported as a confirmed rejection. +Status alone does not indicate whether submitting again would be safe. -I propose adding a new status-history endpoint, -`GET /transfers/status-history`. This will allow administrators to understand -the lifecycle of a transaction and track the progress of all transfers in -real time. +Reasons and adapter details +--------------------------- -We also need new endpoints for `incoming` and `outgoing` transactions, -including paginated and single-entry endpoints, so that administrators can -inspect all transactions, including incomplete or invalid ones. This is -necessary to understand and debug why an incoming transaction never reaches -the Talerable history. +``status_msg`` exposes the adapter's existing human-readable explanation. +On incoming records it explains processing or a decision not to bounce, +on operations it explains execution status. Messages are unstructured and +clients should not parse them to determine status. -Finally, we need a way to track both `initiated` and `bounce` transactions -using new endpoints providing paginated listings, status histories and -single-entry views. +``details`` holds adapter-specific information, especially native transaction, +submission and batch identifiers needed for manual reconciliation. Native +reason codes may be included when available, no common reason-code mapping +is required. Generic clients use ``kind`` and operation ``status``. -We could potentially remove all `transfer` endpoints, since transfers are a -subset of `initiated` operatuibs. Whether to do so depends on whether the -transfer-specific API remains useful as a higher-level abstraction. +Incoming and outgoing records +----------------------------- -We could also enforce a single `initiated` operation concept, event if underneath -bounces works differently. As long as doing this it not too messy for the -database layer we should pursue this simplification. +Expose stored transactions even when incomplete or unsuitable for Taler. +Common payment fields retain their existing Wire Gateway meanings, unknown +values are omitted. Bank documents that cannot be parsed into a transaction +remain outside these resources. -.. ts:def:: InTx +.. ts:def:: ManagementInTx - interface InTx { - // Opaque identifier of the returned record. + interface ManagementInTx { row_id: SafeUint64; - - // Unstructured implementation specific fields that must be shown. - // Contains at least the unique identifiers to be used for manual reconciliation. - details: {string: string}; - - // Date of the transaction. - date: Timestamp; - - // Amount received before credit_fee. + date?: Timestamp; amount?: Amount; - - // Unstructured transaction subject subject?: string; - - // Fee paid by the creditor. - // If not null, creditor actually received amount - credit_fee credit_fee?: Amount; - - // Full payto URI to identify the sender of funds. debit_account?: string; - // ID of the bounce operation if bounced - bounce_id?: SafeUint64; - } + kind: "incomplete" | "pending" | "rejected" | "reserve" | "kycauth" | "wad"; + status_msg?: string; -.. ts:def:: OutTx + // Present once a bounce operation has been initiated. + bounce_operation_id?: SafeUint64; + taler_history_id?: SafeUint64; + details: { [name: string]: string }; + } - interface OutTx { - // Opaque identifier of the returned record. - row_id: SafeUint64; +``kind`` is ``incomplete`` when transaction information is missing, ``pending`` +when the transaction is complete but awaiting a mapping, or ``rejected`` when +the payment cannot be accepted for Taler processing. Available explanations +are exposed in ``status_msg``. ``reserve``, ``kycauth`` and ``wad`` identify +accepted payments exposed in Taler history, with ``taler_history_id`` linking +to the corresponding entry. - // Unstructured implementation specific fields that must be shown. - // Contains at least the unique identifiers to be used for manual reconciliation. - details: {string: string}; +``bounce_operation_id`` is present if a bounce has been initiated and absent +otherwise. Before initiation, the incoming record's ``status_msg`` explains +why a return is pending, skipped or blocked. Once initiated, the link remains +regardless of the outcome, execution status and reasons belong to the bounce +operation. - // Date of the transaction. - date: Timestamp; +.. ts:def:: ManagementOutTx - // Amount received before credit_fee. + interface ManagementOutTx { + row_id: SafeUint64; + date?: Timestamp; amount?: Amount; - - // Unstructured transaction subject subject?: string; - - // Fee paid by the debtor. - // If not null, debtor actually paid amount + debit_fee debit_fee?: Amount; - - // Full payto URI to identify the receiver of funds. credit_account?: string; - } - -.. ts:def:: InitiatedTx - - interface InitiatedTx { - // Opaque ID of this operation. - operation_id: SafeUint64; - - // Amount to transfer. - amount: Amount; - // The recipient's account identifier as a full payto:// URI. - credit_account: string; - - // Optional ID of the wire transfer initiation if operation is a transfer - transfer_id?: SafeUint64; - - // Optional ID of the bounced incoming transaction if operation is a bounce - bounced_id?: SafeUint64; - - // Current status - // pending: in progress - // transient_failure: has failed but may succeed later - // permanent_failure: has failed permanently and will never reach finality - // success: has succeeded and reached finality - // late_failure: has failed permanently after reaching the success status - status: "pending" | "transient_failure" | "permanent_failure" | "success" | "late_failure"; - - // Timestamp that indicates when this status was reached. - timestamp: Timestamp; + operation_id?: SafeUint64; + taler_history_id?: SafeUint64; + details: { [name: string]: string }; } -.. ts:def:: InitiatedStatus +Operation status history +------------------------ - interface InitiatedStatus { - // Opaque ID of this operation. - operation_id: SafeUint64; +Record operation creation and changes to ``status`` or ``status_msg`` in a +small append-only history. Each entry stores the resulting status and message, +clients fetch the operation when they need payment or reconciliation details. +Changes to those details alone do not require a history entry. - // Opaque ID of this status, used for pagination. - row_id: SafeUint64; - - // pending: in progress - // transient_failure: has failed but may succeed later - // permanent_failure: has failed permanently and will never reach finality - // success: has succeeded and reached finality - // late_failure: has failed permanently after reaching the success status - status: "pending" | "transient_failure" | "permanent_failure" | "success" | "late_failure"; +.. ts:def:: ManagementStatusChange - // Optional unstructured messages about the status. Can be used to document the reasons for failure or the state of progress. + interface ManagementStatusChange { + row_id: SafeUint64; + operation_id: SafeUint64; + status: "pending" | "transient_failure" | "permanent_failure" + | "success" | "late_failure"; status_msg?: string; - - // Timestamp that indicates when this status was reached. + // When the adapter recorded this status, not the bank execution time. timestamp: Timestamp; } -Management logic ----------------- - -On top of this API, we could build a Management SPA to support administrative operations. -The UI would show recent failures and track whether someone has reviewed them and made a decision on how to handle them. -We could also extend the API with administrative operations, such as manually requesting a retry. +History starts when the feature is enabled. Existing operations remain +available through the current list/detail endpoints without reconstructing +past transitions. -Test Plan -========= +Adapter implementation +---------------------- +Derive operation ``kind`` by joining the existing transfer and bounce tables, +use ``payment`` for other initiated payments. Derive incoming ``kind`` from +existing information: a Taler history entry supplies its accepted type, a +pending mapping record supplies ``pending``, and missing required transaction +information supplies ``incomplete``. A complete record that the adapter has +rejected for Taler processing is ``rejected``. +Magnet and Nexus can reuse initiation IDs as operation IDs, their stored +status/message and their existing outgoing link. Map Nexus ``unsubmitted`` to +``pending`` in the API. -Definition of Done -================== +Cyclos keeps its native chargeback path. Its current ``bounced`` table already +allows a missing ``chargeback_id``. It can hold the bounce intent before the +native call, with status/message fields added for execution tracking and a +small ID mapping to expose both bounces and initiated payments as operations. +Registration and recovery must update the same intent, including when the +chargeback result is initially unknown. -Alternatives -============ +Reuse stored diagnostic messages and identifiers. Where a skipped or blocked +bounce is currently only logged, persist that explanation on the incoming +record. Older records may omit unavailable messages. -Drawbacks -========= +Management logic +---------------- -Discussion / Q&A -================ -\ No newline at end of file +A separate service can consume status history, notify administrators and keep +track of reviewed failures. Adapters remain responsible for execution and +reconciliation. Retry and other corrective commands are outside this proposal.