taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

107-beekeeper.rst (17042B)


      1 DD 107: Beekeeper: Resumable Forms and KYC Media Collection
      2 ###########################################################
      3 
      4 :Design status: Draft
      5 :Implementation status: untested
      6 :DD shepherd: Christian Grothoff
      7 :Historical contributors: Christian Grothoff
      8 :First published: 2026-09-29
      9 :Last substantive change: 2026-11-21
     10 
     11 Summary
     12 =======
     13 
     14 Beekeeper is a new, standalone service that collects structured information
     15 from users through forms defined in JSON.  Forms have several pages and
     16 conditional questions, and users can fill them in over several sessions.
     17 Besides ordinary fields, Beekeeper can collect KYC-grade evidence: HD selfie
     18 videos with a liveness challenge, videos of directed movements, identity
     19 documents (image, MRZ and optionally the ePassport chip), business
     20 documentation (share registers, company registration data) and phone numbers
     21 and postal addresses that are verified by SMS and by letter.  A single JSON
     22 REST API serves a single-page web app and native iOS and Android apps.
     23 Administrators manage forms, invite users, review submissions and send them
     24 back for correction.
     25 
     26 Beekeeper is a standalone product that can also act as an OAuth2 KYC
     27 provider for the Taler exchange, as the existing KYC providers do.
     28 
     29 Motivation
     30 ==========
     31 
     32 Taler's KYC processes rely on external providers or on hand-built
     33 AML/KYC forms.  Operators need a way to collect customer information that
     34 they can host themselves:
     35 
     36 * Requirements change with jurisdictions and regulators.  So the questions
     37   should be data, not code, and changing them should not require an app
     38   release.
     39 * KYC evidence (identity documents, liveness videos, verified addresses) is
     40   highly sensitive.  Sending it to a third party is a privacy and compliance
     41   risk, and many deployments would rather keep it in-house.
     42 * Users rarely finish a long KYC process in one sitting.  They need to leave,
     43   come back from another device and go back to earlier pages, and they need to
     44   be told when a document must be submitted again.
     45 
     46 The same machinery is useful beyond KYC, for example for customer surveys or
     47 movement assessments.  So Beekeeper is designed as a general form service,
     48 with KYC as its most demanding use case.
     49 
     50 Requirements
     51 ============
     52 
     53 * Forms are JSON documents: questions, answer types, validation, a sequence
     54   of pages, and questions that are shown only under conditions.  Uploading a
     55   form rejects inconsistent definitions up front.
     56 * Conditions and validators give the same result on the server, in the
     57   browser and in both native apps.
     58 * Users fill in forms incrementally and resumably.  They see how many steps
     59   remain and can go back.
     60 * Each user has a personal link.  A form can allow recovering a lost link
     61   through a verified contact e-mail address, and can require an e-mail
     62   one-time code in addition to the link.
     63 * Public forms are listed on the start page; if there is only one, a single
     64   "start" button is shown.
     65 * Administrators can upload, inspect, publish, unpublish and retire forms;
     66   create invitation links with or without an e-mail address; assign a form to
     67   a user, or replace a user's form with another one; review, accept, reject
     68   or return submissions; and export them.
     69 * Answers are stored in PostgreSQL as JSON; media are stored as large objects
     70   keyed by their SHA-256 hash.  Administrators can list media sorted by date or
     71   size, view them and delete them in bulk.
     72 * Upload types: the usual HTML fields; HD selfie videos with liveness
     73   detection; videos of directed movements; identity documents with MRZ;
     74   phone numbers verified by SMS; postal addresses verified by letter.  The
     75   user is told when evidence has expired or failed and must be submitted
     76   again.
     77 * The personal data must be protected in line with KYC regulations: encrypted
     78   at rest, with a retention period per form and with administrator access
     79   recorded in an audit log.
     80 
     81 Proposed Solution
     82 =================
     83 
     84 The detailed specifications live in the Beekeeper repository (``docs/``).
     85 This DD summarises the design and the reasons behind it.
     86 
     87 Components
     88 ----------
     89 
     90 Beekeeper is written in Rust, with the crates used by ``taler-rust`` (axum,
     91 hyper, sqlx, tokio, clap, tracing).  It consists of these parts:
     92 
     93 * ``beekeeper-httpd`` serves the REST API, including resumable uploads, and
     94   the static web front ends.  It is the only process that holds the master
     95   key.
     96 * ``beekeeper-analyzer`` workers analyse media.  They take jobs from a queue
     97   in PostgreSQL (``FOR UPDATE SKIP LOCKED``, woken by ``LISTEN/NOTIFY``).
     98   Because they decode untrusted video with FFmpeg, they run sandboxed:
     99   landlock and seccomp, no network access apart from the database socket, and
    100   a database role that can read only its own job.  Analyzers can run on
    101   separate GPU machines.
    102 * ``beekeeper-admin`` is a command-line tool for database setup, the first
    103   administrator account, key rotation, garbage collection and retention.
    104 * Messages are delivered by helper programs for e-mail, SMS and letters.
    105   They use the same calling convention as challenger's ``challenger-send-*``
    106   scripts, so the existing provider integrations can be reused.
    107 
    108 The SPA and the admin UI live in the Beekeeper repository itself (one
    109 package, two bundles), using established libraries for resumable uploads
    110 (tus-js-client), the proof of work (ALTCHA) and live camera guidance
    111 (MediaPipe tasks-vision).  The native apps are written in Swift and Kotlin,
    112 mainly because reading the ePassport chip over NFC needs NFCPassportReader on
    113 iOS and JMRTD on Android.  They live in the same repository (``apps/``), so
    114 that an API or expression change lands in one commit across server, SPA and
    115 apps.  Each has a platform-independent core (evaluator, form model, API and
    116 tus client) that is tested on Linux; the Android app uses Jetpack Compose and
    117 CameraX and no Google Play services, so that it can be built for F-Droid.
    118 
    119 The server builds on ``taler-common`` from ``taler-rust`` (configuration,
    120 database migrations, logging) and on a vendored copy of ``taler-api``, whose
    121 changes are kept as a series of patches meant for upstream.  Its error codes
    122 are registered in GANA, in a registry of their own
    123 (``beekeeper-error-codes``, values 100000 to 110000), and returned in the
    124 usual ``ErrorDetail`` format.
    125 
    126 The analyzer uses Debian's builds of ONNX Runtime, FFmpeg and Tesseract.  The
    127 models are distributed in a separate ``beekeeper-models`` repository with
    128 their provenance and conversion scripts.
    129 
    130 Form language
    131 -------------
    132 
    133 A form is a JSON document with pages, and each page has a list of typed
    134 questions: ``info``, ``text``, ``number``, ``boolean``, ``choice``,
    135 ``date``, ``email``, ``phone``, ``postal_address``, ``file``, ``video`` and
    136 ``identity_document``.  Labels can be given per language.  Each question can
    137 have ``required`` and ``visible_if`` conditions and a list of validators.
    138 Options per form cover manual or automatic acceptance, the e-mail OTP,
    139 whether users may go back, the retention period, and when evidence must be
    140 resubmitted.
    141 
    142 Conditions use JSONLogic *syntax* with a small set of operators (comparison,
    143 arithmetic, string operations, a portable regular-expression subset, date
    144 arithmetic and ``verified``).  Their *semantics* are defined strictly rather
    145 than by "whatever JavaScript does":
    146 
    147 * Equality is strict and deep.
    148 * Type errors are errors, not coercions.
    149 * String lengths and comparisons are defined over Unicode scalar values.
    150 * ``\d``, ``\w`` and ``\s`` mean their ASCII classes.
    151 
    152 Only this makes four independent implementations agree.  A shared set of
    153 conformance test vectors, currently 215, keeps them in step.  A form is
    154 rejected if an expression refers to a later question, so visibility is
    155 computed in one pass in document order, and the progress indicator stays
    156 stable while the user fills in later pages.
    157 
    158 Users and authentication
    159 ------------------------
    160 
    161 Each user has a personal link.  Its secret sits in the URL fragment, so it
    162 never reaches server logs or ``Referer`` headers, and the client exchanges it
    163 for a session.
    164 
    165 * **One-time code**: a form may require an e-mail one-time code on top of
    166   the link.
    167 * **Recovery**: if the user has a verified contact address, they can request
    168   a new link by e-mail.  The old link stays valid until the new one is first
    169   used, so a third party who requests a recovery cannot lock the user out.
    170 * **Replaced forms**: when a user's form is replaced with another, answers
    171   are carried over by question id and type.  A user can have several open
    172   assignments, in an order the administrator chooses.
    173 
    174 Administrators have database accounts with the roles reviewer, manager,
    175 superuser and service, and log in with a password and TOTP.  Every
    176 administrator action is written to an audit log.
    177 
    178 Verification
    179 ------------
    180 
    181 E-mail addresses and phone numbers are verified with codes while the user is
    182 on the page.  A postal address is verified by a letter with a code.  The form
    183 can be submitted while the letter is in transit.  If the code is not entered
    184 before the deadline, or a later media analysis fails, the assignment goes back
    185 to the user in the ``returned`` state, with the affected pages marked.
    186 
    187 Media, liveness and identity documents
    188 --------------------------------------
    189 
    190 Uploads use the tus 1.0 protocol (with the creation and checksum
    191 extensions), which lets large videos survive unreliable mobile connections.
    192 A user can refer only to blobs that they uploaded themselves; knowing a
    193 SHA-256 hash is not enough.
    194 
    195 For a liveness video, the server issues a random sequence of actions (for
    196 example "turn head left, blink, smile") shortly before recording, and the
    197 video must show them in the right order and time windows.  The analyzer runs
    198 open models through ONNX Runtime (``ort``):
    199 
    200 * YuNet for face detection;
    201 * the MediaPipe face and hand landmark models for eye, mouth and hand actions;
    202 * head pose from landmarks;
    203 * RTMPose or BlazePose for arm and body movements;
    204 * MiniFASNet for passive anti-spoofing;
    205 * SFace to compare the face with the document photo.
    206 
    207 The MRZ is read with Tesseract using the OCR-B training data, and its check
    208 digits are verified.  For NFC passports, the server checks the chip's
    209 security object against the CSCA master list with RustCrypto ``cms`` and
    210 ``x509-cert``.
    211 
    212 Each check yields a score.  Clear passes and clear failures are decided
    213 automatically; everything in between goes to a human reviewer, who sees the
    214 per-action timings and key frames.  Vendor services can be added later
    215 through a trait that takes a blob and a challenge and returns the same result
    216 structure.  InsightFace is not used because its models are licensed for
    217 non-commercial use only.
    218 
    219 Encryption at rest
    220 ------------------
    221 
    222 Envelope encryption protects the data:
    223 
    224 * **Keys**: a master key file wraps a random data key per user and per blob.
    225 * **Answers and addresses** are encrypted with XChaCha20-Poly1305.  The
    226   row's identifiers are bound in as associated data, so a ciphertext cannot
    227   be moved to another row.
    228 * **Blobs** are encrypted in 64 KiB chunks (STREAM), so ranged reads and
    229   resumed uploads work.
    230 * **Deletion** removes the data key first ("crypto-shredding"), which makes
    231   the data unreadable in backups too.
    232 * **Left unencrypted** are identifiers, states, timestamps, sizes and hashes,
    233   so that the admin UI can list and sort.  Contact addresses are found through
    234   an HMAC of the normalised address.
    235 * **The analyzer** never sees the master key.  Each job carries the blob key
    236   wrapped for that analyzer's X25519 key.
    237 
    238 Output
    239 ------
    240 
    241 Administrators get submissions through the admin UI, as a JSON or ZIP export,
    242 and through a change feed with long polling.  There are deliberately no
    243 webhooks: consumers pull, so there is no outbound delivery to retry and no
    244 receiver to authenticate.
    245 
    246 Taler integration
    247 -----------------
    248 
    249 Beekeeper implements the protocol of the exchange's ``oauth2`` KYC logic in
    250 the same shape as challenger: ``POST /oauth2/setup/$CLIENT_ID`` (with the
    251 client secret; the measure's ``initial_address`` pre-fills answers of the
    252 same name), ``GET /oauth2/authorize/$NONCE``, ``POST /oauth2/token`` and
    253 ``GET /oauth2/info``.  A client (registered with ``beekeeper-admin
    254 oauth-client add``) names the forms each process assigns; the process is one
    255 Beekeeper user, so administrators can assign further forms or return answers
    256 as usual.
    257 
    258 As the exchange takes the answer of ``/oauth2/info`` as final, the user is
    259 handed back only once every assignment of the process is accepted,
    260 automatically or by a reviewer; until then the web app shows that the answers
    261 are being checked, and opening the exchange's KYC link again resumes the
    262 process.  A rejected or expired assignment hands the user back with
    263 ``error=access_denied``.
    264 
    265 ``/oauth2/info`` returns the visible answers, the verified questions and the
    266 analysis outcomes.  Files are inlined as the exchange's ``{FILENAME,
    267 MIME_TYPE, CONTENTS}`` objects for the question types the client chose and up
    268 to its size limit, and given as metadata (SHA-256, type, size) otherwise.
    269 The converter ``taler-exchange-kyc-beekeeper-converter`` maps question ids to
    270 attribute names by upper-casing them (``full_name`` gives ``FULL_NAME``).
    271 
    272 Test Plan
    273 =========
    274 
    275 * **Conformance**: the expression vectors run in the Rust test suite and, as
    276   they are written, in the TypeScript, Swift and Kotlin test suites.
    277 * **Static checks**: unit tests, and every example form in the repository
    278   must pass them.
    279 * **Server**: integration tests against a real PostgreSQL database for page
    280   saves, resumption, submission, carry-over, returns, recovery and OTP.
    281   Also: token revocation, and that users cannot access other users' blobs.
    282 * **Uploads**: tus conformance tests, including interrupted and resumed
    283   uploads and checksum mismatches.
    284 * **Verification**: the delivery helpers are replaced by scripts that record
    285   the message, so tests can read the code.
    286 * **Analyzer**: a corpus of recorded videos (genuine; photo, replay and mask
    287   attacks; wrong action order) with expected outcomes.  Thresholds are tuned
    288   on it, and regressions are measured against it.
    289 * **Sandbox**: tests that the analyzer cannot open network sockets or files
    290   outside its model directory.
    291 
    292 Definition of Done
    293 ==================
    294 
    295 * Forms can be uploaded, checked, published and retired; users can be invited
    296   and can fill in, resume, go back and submit on the SPA and on both native
    297   apps.
    298 * E-mail, SMS and letter verification work through challenger-compatible
    299   helper programs.
    300 * Liveness, directed movement, MRZ and face-match analysis run in the
    301   sandboxed analyzer, with a human review queue.
    302 * All personal data is encrypted at rest; retention and crypto-shredding are
    303   implemented; administrator actions are audited.
    304 * Export and the change feed are available and documented in the API
    305   reference.
    306 * The exchange can use Beekeeper as an OAuth2 KYC provider; the round trip
    307   through the exchange's ``oauth2`` plugin (``taler-exchange-kyc-tester``)
    308   yields the expected attributes.
    309 
    310 Alternatives
    311 ============
    312 
    313 * **A commercial KYC vendor for liveness** would give certified
    314   presentation-attack detection (ISO/IEC 30107-3), but would send the most
    315   sensitive data to a third party and bind deployments to its pricing.  It
    316   remains possible through the analysis trait.
    317 * **Extending challenger** would reuse its address verification, but
    318   challenger verifies one address per process, and multi-page forms, media and
    319   review do not fit its model.  Beekeeper reuses its helper programs instead.
    320 * **An existing form standard** (JSON Schema with a UI schema, SurveyJS,
    321   FHIR Questionnaire) would bring existing tooling.  None of them covers the
    322   media and verification types, and their condition languages lack
    323   semantics that can be reproduced exactly in four languages.
    324 * **Plain JSONLogic** has libraries for all four platforms, but these
    325   libraries disagree on coercion, equality and string handling.  Conditions
    326   that behave differently on the server and in the app would be a source of
    327   bugs and of rejected submissions.
    328 * **A TypeScript or Kotlin backend** would share code with a client.  Rust
    329   fits the existing ``taler-rust`` code base and its deployment, and is
    330   memory-safe near FFmpeg and the ONNX models.
    331 * **Webhooks** would push results to consumers, but they need retries,
    332   signing and a receiver.  Long polling gives similar latency without these.
    333 
    334 Drawbacks
    335 =========
    336 
    337 * **Own pipeline code**: without MediaPipe's runtime and OpenCV, we have to
    338   write the model pre- and post-processing ourselves.  This is most of the
    339   analyzer work.
    340 * **Four evaluators**: the condition language has to be implemented four
    341   times.  The conformance suite reduces this risk but does not remove it.
    342 * **Consequences of encryption**: the database cannot search inside answers,
    343   and losing the master key loses all data.
    344 
    345 Discussion / Q&A
    346 ================
    347 
    348 * Open: the expected scale (number of users and media volume), which decides
    349   whether large objects in the database remain adequate for media storage.