README.md (11423B)
1 # taler-packaging ng 2 3 This directory contains the improved implementation of container-based 4 packaging for GNU Taler and associated packages. 5 6 The major improvement is that a component can be built *without* having to 7 rebuild every single package. 8 9 Instead, all dependencies are managed via apt. Each package is built in a 10 fresh environment, with build dependencies pulled in via apt. Previously built 11 packages are available via a file-based apt source. 12 13 Build-time dependencies are automatically installed, making sure that 14 missing build-time dependencies would be detected. 15 16 The packaging logic is also the same for Debian and Ubuntu. 17 18 ## Prerequisites 19 20 You need "podman". On Debian bookworm, you will get shitty performance unless you: 21 22 ``` 23 # apt install containers-storage 24 ``` 25 26 Then check with 27 28 ``` 29 # podman info --debug 30 ``` 31 32 that you get "graphDriverName: overlay" 33 34 If not, try 35 36 ``` 37 # podman system reset 38 ``` 39 40 (deletes all containers, enables reset of the storage layer). 41 42 43 ## Prerequisites for Cross-Builds 44 45 To build for other architecture, you need to install the following packages: 46 ``` 47 apt-get install qemu-user-static binfmt-support 48 ``` 49 50 ## Structure 51 52 * `packages/$DISTRO-$DISTRO_VERNAME`: Output folder for debian packages. 53 Also contains a `Packages.xz` metadata file generated by `dpkg-scanpackages` 54 so that this folder can be directly consumed as a trusted package source. 55 56 * `packages.toml`: Repository, package, version, builder, and dependency configuration. 57 * `buildscripts/*`: Build scripts used during the package build steps. A 58 `debian-overlay` builder copies tracked Debian packaging onto an upstream 59 source tree before building it. 60 * `distros/*`: Files for building for a specific distro. 61 * `server-side/`: Server repository topology and initialization notes. 62 63 64 ## Versions 65 66 `packages.toml` is the single source of truth. A repository entry defines its 67 URL and default builder. A package entry selects a repository and tag, and may 68 also define `debian_path`, `dependencies`, `enabled`, `auto_upgrade`, or 69 override `builder`. Missing paths and dependency lists default to empty, 70 packages default to enabled and eligible for automatic upgrades, and builders 71 default to `generic`. Set `auto_upgrade = false` when separately maintained 72 packaging metadata must be reviewed together with each source update. 73 74 ```toml 75 [repositories."typescript-core"] 76 url = "git://git.taler.net/taler-typescript-core.git" 77 builder = "pnpm-workspace" 78 79 [packages."taler-wallet-cli"] 80 repository = "typescript-core" 81 tag = "v1.6.33-dev.3" 82 debian_path = "packages/taler-wallet-cli" 83 dependencies = [] 84 enabled = true 85 ``` 86 87 The `pnpm-workspace` builder groups packages only when their effective 88 repository URL, tag, and builder are identical. It clones and bootstraps once, 89 installs and builds the union of the selected workspace dependency closures, 90 and then produces each Debian package separately. Packages from one repository 91 may use different tags; they are put into different build groups. 92 93 During the individual Debian builds, this builder sets 94 `TALER_PACKAGING_PREBUILT=1`. Workspace package Makefiles use it to skip their 95 normal dependency/build step and install the artifacts prepared by the grouped 96 build instead. Standalone `make install` behavior is unchanged. 97 98 Configuration is parsed with Python's standard `tomllib`. `taler-pkg upgrade` 99 rewrites it in canonical order using a schema-specific serializer and validates 100 the result before atomically replacing the file. Manual formatting and custom 101 comments are therefore not preserved across upgrades. 102 103 Supported tag syntax (see `buildscripts/generic`): 104 105 * `v$maj.$min.$patch`, e.g. `v1.2.3`: release version. 106 * `v$maj.$min.$patch-dev.$n`, e.g. `v1.2.3-dev.4`: dev version. 107 * `deb-v$maj.$min.$patch-$revision`, e.g. `deb-v1.2.3-1`: release version with a 108 debian revision, for rebuilds that only change `debian/`. 109 110 Debian revisions of dev versions are *not* supported. Other tag formats occur 111 in some of the repositories (`debian-1.2.3`, `v1.2.3-debian-1`, `v1.2.3a`, ...) 112 and are ignored by `taler-pkg`. 113 114 Tags are ordered by major, minor and patch first. Within one release, dev 115 versions sort before the release and debian revisions after it: 116 117 ``` 118 v1.2.3-dev.1 < v1.2.3-dev.2 < v1.2.3 < deb-v1.2.3-1 < v1.2.4-dev.1 119 ``` 120 121 ### From git tag to debian version 122 123 The shared build library derives the debian version of a package from the tag in 124 two steps. `get_tag_debver` strips the `v` / `deb-v` prefix and rewrites 125 `-dev.$n` into `~dev$n`, which sorts before the corresponding release. 126 `make_codename_version` then appends the distro codename. If the version has no 127 debian revision at that point, `-0` is inserted first: a plain `1.2.3+trixie` 128 would be *newer* than `1.2.3`, and `1.2.3~trixie` would be older than any real 129 `1.2.3-$revision`, so both would break version requirements in dependencies. 130 131 Examples, for codename `trixie`: 132 133 ``` 134 v1.2.3 => 1.2.3-0+trixie 135 v1.2.3-dev.4 => 1.2.3~dev4-0+trixie 136 v1.2.3-dev.10 => 1.2.3~dev10-0+trixie 137 deb-v1.2.3 => 1.2.3-0+trixie 138 deb-v1.2.3-1 => 1.2.3-1+trixie 139 ``` 140 141 The resulting versions sort the same way as the tags they come from, so bumping 142 a package tag always produces a package that apt considers newer: 143 144 ``` 145 1.2.3~dev1-0+trixie < 1.2.3~dev10-0+trixie < 1.2.3-0+trixie < 1.2.3-1+trixie 146 ``` 147 148 149 ## Recipes 150 151 ### Building for a distribution 152 153 ``` 154 ./taler-pkg build $DISTRO 155 ``` 156 157 Supported publishing distributions are `debian-trixie` and `ubuntu-noble`. 158 159 ### Initializing the package repositories 160 161 ``` 162 ./taler-pkg init 163 ``` 164 165 This idempotently initializes both supported distributions on 166 `taler-packaging@taler.net`. Testing is initially published from a snapshot of 167 its mutable aptly repo. Stable starts with an empty snapshot and is 168 subsequently changed only by promotion. Existing repositories and publications 169 are validated; incompatible existing state is reported rather than replaced. 170 The `apt/debian` and `apt/ubuntu` prefixes use aptly's default endpoint, so 171 they appear below `$rootDir/public`. 172 173 Repository signing can prompt for the signing-key passphrase when a publication 174 is created or updated. 175 176 ### Publishing and promoting packages 177 178 ``` 179 # Preview and publish locally built packages to testing. 180 ./taler-pkg publish --dry debian-trixie 181 ./taler-pkg publish debian-trixie 182 183 # List the packages in the current stable or testing publication. 184 ./taler-pkg show-published debian-trixie 185 ./taler-pkg show-published debian-trixie-stable 186 ./taler-pkg show-published debian-trixie-testing 187 188 # Show only the latest version of each package per architecture. 189 ./taler-pkg show-published --latest debian-trixie-testing 190 191 # Compare published testing with stable, then promote that exact snapshot. 192 ./taler-pkg promote --dry debian-trixie 193 ./taler-pkg promote debian-trixie 194 ``` 195 196 Publishing imports only packages newer than those already in testing or in any 197 publication sharing its prefix (`apt/debian` or `apt/ubuntu`) and `main` 198 component, including stable. Versions are compared per package name and 199 architecture. Equal versions are skipped even when the local build has 200 different contents; publishing a changed build requires a newer version or 201 Debian revision. `--dry` performs the same checks without changing the server. 202 An incomplete server inventory aborts publishing before any files are uploaded. 203 204 After importing uploads, publishing snapshots the complete testing repo and 205 switches the testing publication to that snapshot. This also happens when no 206 new files need uploading, so pending imports can be published. Promotion 207 switches stable to the snapshot currently published in testing; imports that 208 have not been successfully published cannot reach stable through promotion. 209 210 `show-published` queries the selected publication's snapshot through aptly, 211 lists its enabled architectures plus architecture-independent packages, and 212 prints sorted, unique `name_version_architecture` lines. A base distro selects 213 stable; the `-stable` and `-testing` suffixes select either channel explicitly. 214 Pass `--latest` to show only the newest version of each package per architecture, 215 using Debian version ordering within the selected distro and channel. 216 Architecture-independent packages (`all`) are grouped separately. 217 The same suffixes work for `ubuntu-noble`. Pending testing imports are not 218 included. These suffixes are specific to `show-published`; building, publishing, 219 and promoting continue to use the base distro. 220 221 Packages found only in stable are skipped without copying them into testing. 222 Since promotion replaces stable with the complete testing snapshot, those 223 packages can disappear from stable on the next promotion. The testing repo 224 retains all imported versions. Testing snapshots are also kept, including 225 unpublished snapshots from failed attempts, so that an older release can be 226 selected manually if a rollback is needed. Conflicts already imported into 227 testing require a separate repair; skipping local uploads does not remove them. 228 229 After a successful publish, including one with nothing new to upload, all 230 top-level regular `.deb` and `.ddeb` files in the remote upload directory 231 `/home/taler-packaging/DISTRO` are removed. Other files and subdirectories are 232 preserved. Uploaded files remain there if uploading, importing, or publishing 233 fails. A cleanup failure is reported separately with a nonzero exit status, 234 even though publication succeeded. Run only one publisher per upload 235 directory at a time. 236 237 ### Bumping component versions 238 239 ``` 240 # Show current vs. latest version 241 ./taler-pkg show-latest 242 243 # Show which package tags would change 244 ./taler-pkg upgrade --dry 245 246 # Bump all components to the latest release tag 247 ./taler-pkg upgrade 248 249 # Bump only some components 250 ./taler-pkg upgrade gnunet taler-exchange 251 252 # Also consider dev tags 253 ./taler-pkg upgrade --dev 254 255 # Set a version manually 256 # Edit the package's tag in packages.toml 257 ``` 258 259 `upgrade` considers every package in `packages.toml`, including disabled ones, 260 queries each distinct repository URL once, and writes back the newest tag for 261 each selected package. Without `--dev`, only release and `deb-v` tags are 262 considered, with `--dev` also dev tags. 263 264 Components are only bumped forwards. A package already pointing to a version 265 newer than the newest candidate is left alone, which is what happens to packages 266 pinned to a dev tag when running without `--dev`. Unsupported tag syntax is 267 also left alone. 268 269 Packages with `auto_upgrade = false` are skipped by a default upgrade, but can 270 still be upgraded when named explicitly. `libmicrohttpd` uses this setting: to 271 update it, first refresh and review its tracked Debian overlay from the 272 corresponding Debian packaging tag, then name it explicitly or edit its tag in 273 `packages.toml`. See `README-libmicrohttpd.md` for the version requirement and 274 packaging rationale. 275 276 Packages sharing a repository are updated independently. Passing package names 277 to `upgrade` can intentionally leave packages from one repository on different 278 tags. 279 280 `upgrade` only changes `packages.toml`; run `./taler-pkg build $DISTRO` 281 afterwards. 282 283 ### Forcing a rebuild 284 285 ``` 286 rm packages/$distro/$component@$arch.built.tag 287 ``` 288 289 ### Adding a new distro 290 291 A new distro needs just needs a Dockerfile in ``distros/Dockerfile.$distro``. 292 293 294 ## Future Improvements 295 296 * caching (gradle, npm, ...) between builds 297 * git checkouts on host, allowing fully offline builds 298 * more automation for common tasks 299 * more distros