1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
|
# Copyright (C) 2020, 2021 Maxime Devos
# SPDX-License-Identifier: FSFAP
# Copying and distribution of this file, with or without modification,
# are permitted in any medium without royalty provided the copyright
# notice and this notice are preserved. This file is offered as-is,
# without any warranty.
* TODO Version 0.1 [66%]
The message queue code seems to work, there is a configuration
parser, there is an API to the NSE client, so we almost have
enough functionality for a demonstration ...
** DONE NSE client
See examples/nse-web.scm
*** TODO A GUI (use guile-gnome)
I made a web interface instead. (examples/nse-web.scm)
*** DONE Load configuration from a file
** DONE Documentation [6/6]
*** DONE Concurrency basics
*** DONE Configuration section
Incomplete but good enough.
*** DONE Network structures (incomplete)
*** DONE Communication with services
Incomplete but decent?
*** DONE Estimation of size of network
*** DONE Installation instructions
It became build instructions instead.
** TODO release
* (Guix) Substitutes over GNUnet
Source files:
+ gnu/gnunet/scripts/download-store.scm
+ gnu/gnunet/scripts/publish-store.scm
For previous attempts, see the guix mailing list
and guile-gnunet.
** (Informative) How to use GNUnet FS without networking
(At cost of not finding and sending content from / to
other peers, of course)
--no-grafts: its only for testing, so not spending any
time with grafting is appreciated.
#+BEGIN_SRC shell
gnunetdir=`guix build gnunet --no-grafts`
OLDPATH=$PATH
PATH=$gnunetdir/bin:$gnunetdir/lib/gnunet/libexec:$PATH
export PATH; hash
gnunet-service-fs& gnunet-service-datastore&
# Now have some fun without having to worry about
# any potential security bugs!
#+END_SRC
** DONE Publishing and fetching directories over GNUnet
Example usage of publish-store:
#+BEGIN_SRC shell
guile -L . -e '(@ (gnu gnunet scripts publish-store) main)' -- -i `guix build hello`
#+END_SRC
Output:
#+BEGIN_SRC fundamental
Published at gnunet://fs/chk/DD0C9CM0EG1WPV5MQMWJ1B6QPVPR4R0DEZH8EW2BPASV9B9Q13P2TXWWX59RRBYGSS29TJEJE8WBNS97WQBDWHB7PGE0P21FMNYT83G.YVWEM5SWBHFQR1YFG8TAJD5J3284XDEK144BFY3B2F3BRD678T3HZPDTP7WB18A04N3HT4HC0SZZ6H71XQRVVZNZX2ZGJXXSJXD9KW0.24798 in gnunet-nix-archive-json/0 format
#+END_SRC
Example usage of download-store:
#+BEGIN_SRC shell
guile -L . -e '(@ (gnu gnunet scripts download-store) main)' -- -i $THE_GNUNET_URI -o t
# Output: plenty of progress messages
ls -R t
# Success!
#+END_SRC
** TODO Doing it *efficiently* [1/7]
Some proposals (some forthe implementation, some of the directory format):
* [ ] inline tiny files
* [ ] parallel downloads
* [ ] compress files
Potential problem: if some peers use different compressors,
then the files will be different --> more duplication on the network,
less performant.
* [ ] maybe use a binary format instead of JSON
* [ ] factor out store names.
Sometimes, changing a build dependency changes very little,
perhaps only the embedded store names (e.g. when grafting). Maybe
zero out the store references in the files, and mark the location
anv value in the JSON container. This way, less data should be
duplicated over the network (which should lead to better performance!).
* [X] indexing (letting gnunet-service-fs refer to the store items by
file name instead of inserting the store items in a database).
This is done by default in the implementation for the
gnunet-nix-archive-json/0 format. The former point ‘factor out store
names’ will undo this, however (which could be resolved with
a Hurd translator or FUSE filesystem).
* [ ] contact 'gnunet-service-fs' directly, instead of calling a
binary for each file.
** TODO Let Guix use a GNUnet substituter
Make the substituters ‘pluggable’ somehow. Maybe allow
to use the user's substituter, instead of the system's
subsituter? (Practical for testing and experimentation.)
Allow substitutes over http, ipfs, GNUnet, torrents (?).
Insert some text about open coding here.
*** TODO support nar format [1/2]
For easy use from Guix publish & substitute.
- [X] for downloading to
- [ ] for publishing from
(probably not required, but nice for symmetry)
*** TODO make Guix publish extensible [2/3]
- [X] write proposal
- [X] submit proposal
https://issues.guix.gnu.org/46214
XXX not really happy about it ...
- [ ] incorporate feedback for ‘narinfo generation hooks’,
& proposal makes it upstream
*** TODO make Guix substitute extensible [3/4]
- [X] write proposal
- [X] make sure it actually works
(test with publish-store and download-store)
(currently rather crude)
The HTTP substituter has been rewritten in terms
of the proposal (and "make check" succeeds). However,
it hasn't been tested with a GNUnet or IPFS substituter.
- [X] submit proposal
https://issues.guix.gnu.org/46800
- [ ] incorporate feedback & proposal makes it upstream
Ludovic Courtes (12 Mar 2021):
I don’t think my concerns are really addressed :-), but
at the same time I think we need a playground for these things
so we can actually grownew substitute methods like those you’ve
been looking at. Hmmm tricky!
*** TODO add gnunet extension to guix publish and guix substitute [0/5]
- [ ] clean up source code somewhat
- [ ] release
- [ ] submit package definitions
- [ ] submit extra service definitions
- [ ] incorporate feedback & code makes it upstream
** TODO (external) GNUnet service for Guix System
Put it in a container to easy people's minds (including
my own).
** TODO more TODO
Maybe substitutes in GNS or something? Let's first
address the former points.
|
