This document is a summary of the changes made to GNUnet for version 0.9.x (from 0.8.x) and what this major redesign tries to address. First of all, the redesign does not (intentionally) change anything fundamental about the application-level protocols or how files are encoded and shared. However, it is not protocol-compatible due to other changes that do not relate to the essence of the application protocols. This choice was made since productive development and readable code were considered more important than compatibility at this point. The redesign tries to address the following major problem groups describing isssues that apply more or less to all GNUnet versions prior to 0.9.x: PROBLEM GROUP 1 (scalability): * The code was modular, but bugs were not. Memory corruption in one plugin could cause crashes in others and it was not always easy to identify the culprit. This approach fundamentally does not scale (in the sense of GNUnet being a framework and a GNUnet server running hundreds of different application protocols -- and the result still being debuggable, secure and stable). * The code was heavily multi-threaded resulting in complex locking operations. GNUnet 0.8.x had over 70 different mutexes and almost 1000 lines of lock/unlock operations. It is challenging for even good programmers to program or maintain good multi-threaded code with this complexity. The excessive locking essentially prevents GNUnet 0.8 from actually doing much in parallel on multicores. * Despite efforts like Freeway, it was virtually impossible to contribute code to GNUnet 0.8 that was not writen in C/C++. * Changes to the configuration almost always required restarts of gnunetd; the existence of change-notifications does not really change that (how many users are even aware of SIGHUP, and how few options worked with that -- and at what expense in code complexity!). * Valgrinding could only be done for the entire gnunetd process. Given that gnunetd does quite a bit of CPU-intensive crypto, this could not be done for a system under heavy (or even moderate) load. * Stack overflows with threads, while rare under Linux these days, result in really nasty and hard-to-find crashes. * structs of function pointers in service APIs were needlessly adding complexity, especially since in most cases there was no actual polymorphism SOLUTION: * Use multiple, lously-coupled processes and one big select loop in each (supported by a powerful util library to eliminate code duplication for each process). * Eliminate all threads, manage the processes with a master-process (gnunet-arm, for automatic restart manager) which also ensures that configuration changes trigger the necessary restarts. * Use continuations (with timeouts) as a way to unify cron-jobs and other event-based code (such as waiting on network IO). => Using multiple processes ensures that memory corruption stays localized. => Using multiple processes will make it easy to contribute services written in other language(s). => Individual services can now be subjected to valgrind => Process priorities can be used to schedule the CPU better Note that we can not just use one process with a big select loop because we have blocking operations (and the blocking is outside of our control, thanks to MySQL, sqlite, gethostbyaddr, etc.). So in order to perform reasonably well, we need some construct for parallel execution. RULE: If your service contains blocking functions, it MUST be a process by itself. If your service is sufficiently complex, you MAY choose to make it a separate process. * Eliminate structs with function pointers for service APIs; instead, provide a library (still ending in _service.h) API that transmits the requests nicely to the respective process (easier to use, no need to "request" service in the first place; API can cause process to be started/stopped via ARM if necessary). PROBLEM GROUP 2 (UTIL-APIs causing bugs): * The existing logging functions were awkward to use and their expressive power was never really used for much. * While we had some rules for naming functions, there were still plenty of inconsistencies. * Specification of default values in configuration could result in inconsistencies between defaults in config.scm and defaults used by the program; also, different defaults might have been specified for the same option in different parts of the program. * The TIME API did not distinguish between absolute and relative time, requiring users to know which type of value some variable contained and to manually convert properly. Combined with the possibility of integer overflows this is a major source of bugs. * The TIME API for seconds has a theoretical problem with a 32-bit overflow on some platforms which is only partially fixed by the old code with some hackery. SOLUTION: * Logging was radically simplified. * Functions are now more conistently named. * Configuration has no more defaults; instead, we load a global default configuration file before the user-specific configuration (which can be used to override defaults); the global default configuration file will be generated from config.scm. * Time now distinguishes between struct GNUNET_TIME_Absolute and struct GNUNET_TIME_Relative. We use structs so that the compiler won't coerce for us (forcing the use of specific conversion functions which have checks for overflows, etc.). Naturally the need to use these functions makes the code a bit more verbose, but that's a good thing given the potential for bugs. * There is no more TIME API function to do anything with 32-bit seconds * There is now a bandwidth API to handle non-trivial bandwidth utilization calculations PROBLEM GROUP 3 (statistics): * Databases and others needed to store capacity values similar to what stats was already doing, but across process lifetimes ("state"-API was a partial solution for that, but using it was clunky) * Only gnunetd could use statistics, but other processes in the GNUnet system might have had good uses for it as well SOLUTION: * New statistics library and service that offer an API to inspect and modify statistics * Statistics are distinguished by service name in addition to the name of the value * Statistics can be marked as persistent, in which case they are written to disk when the statistics service shuts down. => One solution for existing stats uses, application stats, database stats and versioning information! PROBLEM GROUP 4 (Testing): * The existing structure of the code with modules stored in places far away from the test code resulted in tools like lcov not giving good results. * The codebase had evolved into a complex, deeply nested hierarchy often with directories that then only contained a single file. Some of these files had the same name making it hard to find the source corresponding to a crash based on the reported filename/line information. * Non-trivial portions of the code lacked good testcases, and it was not always obvious which parts of the code were not well-tested. SOLUTION: * Code that should be tested together is now in the same directory. * The hierarchy is now essentially flat, each major service having on directory under src/; naming conventions help to make sure that files have globally-unique names * All code added to the new repository must come with testcases with reasonable coverage. PROBLEM GROUP 5 (core/transports): * The new DV service requires session key exchange between DV-neighbours, but the existing session key code can not be used to achieve this. * The core requires certain services (such as identity, pingpong, fragmentation, transport, traffic, session) which makes it meaningless to have these as modules (especially since there is really only one way to implement these) * HELLO's are larger than necessary since we need one for each transport (and hence often have to pick a subset of our HELLOs to transmit) * Fragmentation is done at the core level but only required for a few transports; future versions of these transports might want to be aware of fragments and do things like retransmission * Autoconfiguration is hard since we have no good way to detect (and then use securely) our external IP address * It is currently not possible for multiple transports between the same pair of peers to be used concurrently in the same direction(s) * We're using lots of cron-based jobs to periodically try (and fail) to build and transmit SOLUTION: * Rewrite core to integrate most of these services into one "core" service. * Redesign HELLO to contain the addresses for all enabled transports in one message (avoiding having to transmit the public key and signature many, many times) * With discovery being part of the transport service, it is now also possible to "learn" our external IP address from other peers (we just add plausible addresses to the list; other peers will discard those addresses that don't work for them!) * New DV will consist of a "transport" and a high-level service (to handle encrypted DV control- and data-messages). * Move expiration from one field per HELLO to one per address * Require signature in PONG, not in HELLO (and confirm on address at a time) * Move fragmentation into helper library linked against by UDP (and others that might need it) * Link-to-link advertising of our HELLO is transport responsibility; global advertising/bootstrap remains responsibility of higher layers * Change APIs to be event-based (transports pull for transmission data instead of core pushing and failing) PROBLEM GROUP 6 (FS-APIs): * As with gnunetd, the FS-APIs are heavily threaded, resulting in hard-to-understand code (slightly better than gnunetd, but not much). * GTK in particular does not like this, resulting in complicated code to switch to the GTK event thread when needed (which may still be causing problems on Gnome, not sure). * If GUIs die (or are not properly shutdown), state of current transactions is lost (FSUI only saves to disk on shutdown) * FILENAME metadata is killed by ECRS/FSUI to avoid exposing HOME, but what if the user set it manually? * The DHT was a generic data structure with no support for ECRS-style block validation SOLUTION: * Eliminate threads from FS-APIs * Incrementally store FS-state always also on disk using many small files instead of one big file * Have API to manipulate sharing tree before upload; have auto-construction modify FILENAME but allow user-modifications afterwards * DHT API was extended with a BLOCK API for content validation by block type; validators for FS and DHT block types were written; BLOCK API is also used by gap routing code. PROBLEM GROUP 7 (User experience): * Searches often do not return a sufficient / significant number of results * Sharing a directory with thousands of similar files (image/jpeg) creates thousands of search results for the mime-type keyword (problem with DB performance, network transmission, caching, end-user display, etc.) * Users that wanted to share important content had no way to tell the system to replicate it more; replication was also inefficient (this desired feature was sometimes called "power" publishing or content pushing) SOLUTION: * Have option to canonicalize keywords (see suggestion on mailinglist end of June 2009: keep consonants and sort those alphabetically); not fully implemented yet * When sharing directories, extract keywords first and then push keywords that are common in all files up to the directory level; when processing an AND-ed query and a directory is found to match the result, do an inspection on the metadata of the files in the directory to possibly produce further results (requires downloading of the directory in the background); needs more testing * A desired replication level can now be specified and is tracked in the datastore; migration prefers content with a high replication level (which decreases as replicase are created) => datastore format changed; we also took out a size field that was redundant, so the overall overhead remains the same * Peers with a full disk (or disabled migration) can now notify other peers that they are not interested in migration right now; as a result, less bandwidth is wasted pushing content to these peers (and replication counters are not generally decreased based on copies that are just discarded; naturally, there is still no guarantee that the replicas will stay available) SUMMARY: * Features eliminated from util: - threading (goal: good riddance!) - complex logging features [ectx-passing, target-kinds] (goal: good riddance!) - complex configuration features [defaults, notifications] (goal: good riddance!) - network traffic monitors (goal: eliminate) - IPC semaphores (goal: d-bus? / eliminate?) - second timers * New features in util: - scheduler - service and program boot-strap code - bandwidth and time APIs - buffered IO API - HKDF implementation (crypto) - load calculation API - bandwidth calculation API * Major changes in util: - more expressive server (replaces selector) - DNS lookup replaced by async service