diff options
author | ng0@n0.is <ng0@n0.is> | 2019-08-21 14:32:29 +0000 |
---|---|---|
committer | ng0@n0.is <ng0@n0.is> | 2019-08-21 14:32:29 +0000 |
commit | 297fc34bed8ca8260b0d94aea2aadda9720739f5 (patch) | |
tree | b748a746737a902bb735793ba180fd40f247b22a | |
parent | 80dd88765f69f27f2f06433525518b7355b7d9f7 (diff) | |
download | libmicrohttpd-gsoc2019-gsoc-final.tar.gz libmicrohttpd-gsoc2019-gsoc-final.zip |
self-contain the report, soon to be sent to libmicrohttpd mailinglist.gsoc-final
-rw-r--r-- | report | 394 |
1 files changed, 394 insertions, 0 deletions
@@ -0,0 +1,394 @@ | |||
1 | Hi, | ||
2 | |||
3 | here's my summary for the work I've done during | ||
4 | Google Summer of Code 2019 for libmicrohttpd. | ||
5 | (https://summerofcode.withgoogle.com/projects/#4926907403993088) | ||
6 | |||
7 | As the communication with my mentor happened almost exclusively | ||
8 | via private Email and Mumble Voicecalls, the gist of it is | ||
9 | to optimize syscalls with focus on setsockopt optimization on | ||
10 | various platforms (Debian Linux, FreeBSD 12-RELEASE-p3, NetBSD-CURRENT, | ||
11 | cygwin on Windows 10). OSX/macOS is listed but was disregarded as | ||
12 | the process would've meant losing too much time as I was simply | ||
13 | not familiar with the process of setting up this in my test environment. | ||
14 | |||
15 | The summary and analysis of the state of syscalls can be found at | ||
16 | https://d.n0.is/pub/libmicrohttpd/gsoc2019/syscalls.html and | ||
17 | checked out via mercurial: | ||
18 | hg clone https://c.n0.is/libmicrohttpd/gsoc2019 | ||
19 | |||
20 | The tests I wrote to analyse the sycalls focused on 4 scenarios: | ||
21 | 1. continuous response generation (ensure no constant setsockopt() calls | ||
22 | during transmission) | ||
23 | 2. tiny response generation (fits in one packet, including header, | ||
24 | ensure one packet and sending without corking) | ||
25 | 3. modest response generation (header first, then body, ensure last part | ||
26 | of body is sent without corking) | ||
27 | 4. response generation using sendfile() (making sure that after last write | ||
28 | operation the packet is sent without corking, no unnecessary setsockopts) | ||
29 | |||
30 | 1) initial syscall assessment | ||
31 | |||
32 | Initially, before changing the code, it can be observed that we make too | ||
33 | many syscalls of setsockopt() on Linux (Debian) and other platforms. | ||
34 | The setsockopt() calls on a platform which supports MSG_MORE (Linux) are | ||
35 | unnecessary high, and on FreeBSD TCP_NOPUSH is not used. | ||
36 | The way the syscalls are called is unnecessary expensive. | ||
37 | |||
38 | * Linux (Debian) | ||
39 | - Continuous Response Generation test shows a pattern of setsockopt, sendto, | ||
40 | setsockopt, setsockopt, setsockopt, setsockopt, setsockopt, setsockopt. | ||
41 | After this the body of the message is send, and we see 2 more setsockopt() | ||
42 | calls at the end. | ||
43 | (9 setsockopt calls) | ||
44 | |||
45 | - Tiny Response Generation test starts with 1 setsockopt() call after | ||
46 | receiving the GET request (TCP_CORK). | ||
47 | After sending the header, we see 6 setsockopt() calls. | ||
48 | Again after sending the body see 2 final setsockopt() calls. | ||
49 | (9 setsockopt calls) | ||
50 | |||
51 | - Modest Response Generation test starts with 1 setsockopt() call after | ||
52 | receiving the GET request. We send the header. 6 calls to setsockopt() | ||
53 | follow. The body is send, and now we get 2 more setsockopt() calls | ||
54 | at the end. | ||
55 | (9 setsockopt calls) | ||
56 | |||
57 | - Response Generation using sendfile() test starts with 1 call to | ||
58 | setsockopt() after the GET request is received, immediately before | ||
59 | the header is send. Then we see 6 calls to setsockopt(). | ||
60 | sendfile() is called, setsockopt() is called with TCP_NODELAY. | ||
61 | The filedescriptor is closed. setsockopt() is called with | ||
62 | TCP_NODELAY. | ||
63 | (9 setsockopt calls) | ||
64 | |||
65 | * FreeBSD | ||
66 | - Continuous Response Generation test shows that setsockopt() is called | ||
67 | after receiving the GET request, 1 call immediately before sending the | ||
68 | header, 1 call immediately after sending the header (TCP_NODELAY). | ||
69 | The body is send, and no further setsockopt() call is made. | ||
70 | (3 setsockopt calls) | ||
71 | |||
72 | - Tiny Response Generation test has a comparable pattern: after receiving | ||
73 | the GET request, 1 call is made to setsockopt() immediately before calling | ||
74 | sendto() on the header and 1 call is made to setsockopt() immediately | ||
75 | after sending the header. No further setsockopt() calls are made. | ||
76 | (2 setsockopt calls) | ||
77 | |||
78 | - Modest Response Generation test shows a comparable pattern: after | ||
79 | receiving the GET request, 1 call is made to setsockopt() immediately | ||
80 | before calling sendto() on the header and 1 call is made to setsockopt() | ||
81 | immediately after sending the header. | ||
82 | No further setsockopt() calls are made. | ||
83 | (2 setsockopt calls) | ||
84 | |||
85 | - Response Generation using sendfile() result is comparable to the 3 tests | ||
86 | run before it. | ||
87 | (2 setsockopt calls) | ||
88 | |||
89 | * NetBSD | ||
90 | - Continuous Response Generation test shows 1 call to setsockopt() | ||
91 | before sending the header, and 1 call to setsockopt() after | ||
92 | sending the header. | ||
93 | (2 setsockopt calls) | ||
94 | |||
95 | - Tiny Response Generation test shows that after receiving the | ||
96 | GET request, we see 1 call to setsockopt, before sending | ||
97 | the header. Immediately after the header see 2 calls to | ||
98 | setsockopt(). After the body is send we see 1 more call to | ||
99 | setsockopt(). | ||
100 | (4 setsockopt calls) | ||
101 | |||
102 | - Modest Response Generation test shows that after receiving the | ||
103 | GET request, we see 1 setsockopt() call which is followed by | ||
104 | sending the header. This is followed by 2 calls to setsockopt(). | ||
105 | After this, the body is send, followed by another call to setsockopt(). | ||
106 | (4 setsockopt calls) | ||
107 | |||
108 | - Response Generation using sendfile() shows that after receiving the | ||
109 | GET request, the file with the content "a" is created. | ||
110 | We see 1 call to setsockopt() just before the header is send. | ||
111 | Immediately after this we see 1 more setsockopt call. | ||
112 | After a call to pread() we see another setsockopt call | ||
113 | followed by a sendto of "a". | ||
114 | (3 setsockopt calls) | ||
115 | |||
116 | * cygwin x64 (under Windows 10) | ||
117 | - Continuous Response Generation test shows 1 call of main to | ||
118 | setsockopt() in the beginning. | ||
119 | After this, MHD-single calls setsockopt 2 times | ||
120 | (setsockopt, followed by cygwin_send, followed by another | ||
121 | setsockopt call). The pattern in cygwin is comparable to the | ||
122 | ktruss log in NetBSD, where we see a setsockopt, sendto, | ||
123 | setsockopt at the beginning of the log. | ||
124 | (3 setsockopt calls) | ||
125 | |||
126 | - Tiny Response Generation test shows 1 call to setsockopt. | ||
127 | This is followed by a send of the size 99 (the header). Then follows | ||
128 | another setsockopt, followed by another call to send (the body). | ||
129 | (2 setsockopt calls) | ||
130 | |||
131 | - Modest Response Generation test shows 2 calls to setsockopt() related | ||
132 | to the socket we use, each one of them before the respective send() is | ||
133 | called. | ||
134 | (2 setsockopt calls) | ||
135 | |||
136 | - Response Generation using sendfile() shows 1 call to setsockopt(), | ||
137 | followed by a send() with size 99 (the header). | ||
138 | This is followed by 1 call to setsockopt(), and one more call to send() | ||
139 | with size 1 (the body). | ||
140 | (2 setsockopt calls) | ||
141 | |||
142 | 2) approach taken, description of the changes made, | ||
143 | |||
144 | I looked into the system specific tweaks we could start with. This included | ||
145 | looking and reading into the documentation and source changelogs of the | ||
146 | targeted Operating Systems and their TCP/IP stack implementation. | ||
147 | It was concluded that MSG_MORE should be prefered on Linux if it exists, | ||
148 | and get priority over any other existing tcp socket flags. | ||
149 | TCP_NODELAY is considered for toggling Naggle's algorithm on or off. | ||
150 | On FreeBSD I had to read into additions to the stack in the last 10+ years, | ||
151 | and TCP_NOPUSH was found to be the closest to the Linux specific TCP_CORK. | ||
152 | Both TCP_NOPUSH and TCP_CORK deal with not sending out partial frames if | ||
153 | set, an optimization to consider. | ||
154 | Through incremental testing, rewriting and reasoning about the best way | ||
155 | the systems should be adressed, I got to the endresult. | ||
156 | |||
157 | Two helper functions were written (pre_cork_setsockopt(), | ||
158 | post_cork_setsockopt()) to handle setsockopt() calls. | ||
159 | pre_cork_setsockopt() is called before any send()/sendmsg() calls. | ||
160 | post_cork_setsockopt() is called after any send()/sendmsg() calls. If the | ||
161 | MSG_MORE option is supported on this platform, we do nothing in those | ||
162 | functions. | ||
163 | If the sk_cork_on boolean is already the state we want to set, we | ||
164 | do nothing. | ||
165 | An additional helper function was added to toggle Naggle's Algorithm, | ||
166 | MHD_socket_set_nodelay_(). | ||
167 | In the pre_cork_setsockopt() function, if TCP_CORK/TCP_NOPUSH/MSG_MORE | ||
168 | do not exist on a platform, we set Naggle to on/off, | ||
169 | in all other cases we simply set Naggle to always off. This is achieved | ||
170 | with a third helper function, MHD_socket_set_nodelay_(). | ||
171 | |||
172 | Two function were written to handle send() calls. | ||
173 | MHD_send_on_connection_() sends a given buffer on connection and | ||
174 | remembers the current state of the socket options. | ||
175 | setsockopt() is only called when absolutely necessary. | ||
176 | If the connection is using TLS, GnuTLS handles the corking. | ||
177 | In all other cases (ie plaintext transmission) we call | ||
178 | pre_cork_setsockopt(), followed by the send(), and post_cork_setsockopt(). | ||
179 | |||
180 | MHD_send_on_connection2_() sends the header followed by the buffer on the | ||
181 | connection given. | ||
182 | When sendmsg() is available, sendmsg() is used to send | ||
183 | both (header and buffer) at once and returns the number of bytes sent | ||
184 | from both buffers. | ||
185 | Otherwise if writev() is available, writev() is used to send | ||
186 | both (header and buffer) at once and returns the number of bytes sent | ||
187 | from both buffers or -1 on error. | ||
188 | If sendmsg() or writev() are unavailable, only the header is send via | ||
189 | MHD_send_on_connection_(). | ||
190 | If we have writev or sendmsg, we call pre_cork_setsockopt with 'false' | ||
191 | for the want_cork argument. | ||
192 | If we succeeded in sending the full buffer, we call post_cork_setsockopt() | ||
193 | again with 'false' for the want_cork argument, as we want to make sure | ||
194 | that the OS flushes at the end. | ||
195 | |||
196 | The previously existing function for sendfile() wrapping in connection.c was | ||
197 | moved and renamed to MHD_send_sendfile_(). | ||
198 | |||
199 | The functions were then tested and replaced all previous setsockopt calls in | ||
200 | connection.c. In connection.c, the following functions were removed: | ||
201 | - socket_start_normal_buffering | ||
202 | - socket_start_no_buffering | ||
203 | |||
204 | and calls of both of them in connection.c were removed, in addition to | ||
205 | removing calls of socket_start_no_buffering_flush() and | ||
206 | socket_start_extra_buffering(). | ||
207 | |||
208 | 3) resulting syscall behavior | ||
209 | |||
210 | Overall we see an improvement to the number of times setsockopt() is | ||
211 | called as well as to the time it is called. | ||
212 | On Linux (Debian) setsockopt() for the measurement we did is no longer | ||
213 | called, as expected. The only setsockopt() call logged is provided as | ||
214 | reference. The MSG_MORE flag related code is working. | ||
215 | |||
216 | Compared to the first tests on FreeBSD in most testcases we now get no | ||
217 | setsockopt() calls after the header. | ||
218 | |||
219 | On NetBSD the comparison before/after shows that for the sendfile() test the | ||
220 | order of the setsockopt call changed and we have 1 call less. For the tiny | ||
221 | response generation test we have 2 calls less to setsocktop() than before, | ||
222 | the same applies for modest response generation. | ||
223 | |||
224 | On cygwin the visible improvements include for tiny response generation | ||
225 | only 1 call to setsockopt() instead of 2 setsockopt() calls. | ||
226 | |||
227 | * Linux (Debian) | ||
228 | - continuous response generation: | ||
229 | setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 | ||
230 | before header is send. | ||
231 | sendto() with MSG_NOSIGNAL|MSG_MORE for header. | ||
232 | sendto() with MSG_NOSIGNAL for the body. | ||
233 | No other setsockopt() calls. | ||
234 | - tiny response generation: | ||
235 | setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 | ||
236 | before the header is send, no other setsockopt() calls. | ||
237 | - modest response generation | ||
238 | setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 | ||
239 | before the header is send, no further setsockopt() calls. | ||
240 | - response generation using sendfile() | ||
241 | setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 | ||
242 | before the header is send, no further setsockopt() calls. | ||
243 | |||
244 | * FreeBSD | ||
245 | - continuous response generation: | ||
246 | setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0) | ||
247 | setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0) | ||
248 | Before header is send: | ||
249 | setsockopt(4,IPPROTO_TCP,TCP_FASTOPEN_MIN_COOKIE_LEN,0x7fffdfffde5c,4) = 0 (0x0) | ||
250 | header: | ||
251 | sendto(4,"HTTP/1.1 200 OK\r\nConnection: K"...,108,MSG_NOSIGNAL,NULL,0) = 108 (0x6c) | ||
252 | followed by a pattern of: | ||
253 | sendto(4,"1\r\nb\r\n",6,MSG_NOSIGNAL,NULL,0) = 6 (0x6) | ||
254 | poll({ 3/POLLIN 4/POLLPRI|POLLOUT|POLLRDBAND },2,-1) = 1 (0x1) | ||
255 | Ends with another setsockopt call: | ||
256 | setsockopt(4,IPPROTO_TCP,TCP_FASTOPEN_MIN_COOKIE_LEN,0x7fffdfffde5c,4) = 0 (0x0) | ||
257 | - tiny response generation: | ||
258 | setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0) | ||
259 | setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0) | ||
260 | before the header is send. | ||
261 | No further setsockopt() calls. | ||
262 | - modest response generation: | ||
263 | setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0) | ||
264 | setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0) | ||
265 | before the header is send. | ||
266 | No further setsockopt() calls. | ||
267 | - response generation using sendfile(): | ||
268 | setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0) | ||
269 | setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0) | ||
270 | before the header is send. | ||
271 | setsockopt(4,IPPROTO_TCP,TCP_FASTOPEN_MIN_COOKIE_LEN,0x7fffdfffde5c,4) = 0 (0x) | ||
272 | before the HTTP/1.1 200 OK is send. | ||
273 | |||
274 | * NetBSD | ||
275 | - continuous response generation: | ||
276 | before receiving the GET request: | ||
277 | setsockopt(0xa, 0xffff, 0x800, 0x7e25e041a1bc, 0x4) = 0 | ||
278 | After receiving the GET request: | ||
279 | setsockopt(0xa, 0x6, 0x1, 0x7e25dcbffec8, 0x4) = 0 | ||
280 | sendto(0xa, 0x7e25e067f800, 0x6c, 0x400, 0, 0) = 108 | ||
281 | "HTTP/1.1 200 OK\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunk" | ||
282 | sendto(0xa, 0x7e25e067f807, 0x6, 0x400, 0, 0) = 6 | ||
283 | "1\r\nb\r\n" | ||
284 | setsockopt(0xa, 0x6, 0x1, 0x7e25dcbffecc, 0x4) = 0 | ||
285 | poll(0x7e25e067a000, 0x2, 0xffffffff) = 1 | ||
286 | sendto(0xa, 0x7e25e067f807, 0x6, 0x400, 0, 0) = 6 | ||
287 | "1\r\nb\r\n" | ||
288 | poll(0x7e25e067a000, 0x2, 0xffffffff) = 1 | ||
289 | sendto(0xa, 0x7e25e067f807, 0x6, 0x400, 0, 0) = 6 | ||
290 | "1\r\nb\r\n" | ||
291 | |||
292 | and so forth. At the end one last setsockopt() call: | ||
293 | setsockopt(0xa, 0x6, 0x1, 0x7e25dcbffec8, 0x4) = 0 | ||
294 | - tiny response generation: | ||
295 | before GET request: | ||
296 | setsockopt(0xa, 0xffff, 0x800, 0x7986b2e1a1bc, 0x4) = 0 | ||
297 | No other setsockopt() call. | ||
298 | - modest response generation: | ||
299 | before receiving the GET request: | ||
300 | setsockopt(0xa, 0xffff, 0x800, 0x7efa1c01a1bc, 0x4) = 0 | ||
301 | No other setsockopt() calls. | ||
302 | - response generation using sendfile(): | ||
303 | before receiving the GET request: | ||
304 | setsockopt(0xa, 0xffff, 0x800, 0x723db1e1a1bc, 0x4) = 0 | ||
305 | After receiving the GET request: | ||
306 | setsockopt(0xa, 0x6, 0x1, 0x723dae5ffec8, 0x4) = 0 | ||
307 | sendto(0xa, 0x723db21052c0, 0x63, 0x400, 0, 0) = 99 | ||
308 | "HTTP/1.1 200 OK\r\nConnection: Keep-Alive\r\nContent-Length: 1\r\nDat" | ||
309 | pread(0xb, 0x723db20fb0a0, 0x1, 0, 0) = 1 | ||
310 | "a" | ||
311 | sendto(0xa, 0x723db20fb0a0, 0x1, 0x400, 0, 0) = 1 | ||
312 | "a" | ||
313 | setsockopt(0xa, 0x6, 0x1, 0x723dae5ffecc, 0x4) = 0 | ||
314 | close(0xb) = 0 | ||
315 | |||
316 | * cygwin x64 (under Windows 10) | ||
317 | - continuous response generation: | ||
318 | [MHD-single] inc 1643 __set_winsock_errno: setsockopt:1689 | ||
319 | - winsock error 10042 -> errno 109 | ||
320 | [MHD-single] inc 1643 cygwin_setsockopt: -1 = | ||
321 | setsockopt(6, 6, 0x4, 0xFFDFCB5C, 4), errno 109 | ||
322 | [MHD-single] inc 1643 cygwin_send: 108 = send(6, | ||
323 | 0x6000790A0, 108, 0x20) | ||
324 | [MHD-single] inc 1643 cygwin_send: 6 = send(6, | ||
325 | 0x6000790A7, 6, 0x20) | ||
326 | [MHD-single] inc 1643 __set_errno: void | ||
327 | __set_winsock_errno(const char*, int):203 setting errno 109 | ||
328 | [MHD-single] inc 1643 __set_winsock_errno: setsockopt:1689 | ||
329 | - winsock error 10042 -> errno 109 | ||
330 | [MHD-single] inc 1643 cygwin_setsockopt: -1 = | ||
331 | setsockopt(6, 6, 0x4, 0xFFDFCB58, 4), errno 109 | ||
332 | |||
333 | and at the end: | ||
334 | [MHD-single] inc 1643 __set_winsock_errno: | ||
335 | setsockopt:1689 - winsock error 10042 -> errno 109 | ||
336 | [MHD-single] inc 1643 cygwin_setsockopt: -1 = | ||
337 | setsockopt(6, 6, 0x4, 0xFFDFCB5C, 4), errno 109 | ||
338 | - tiny response generation: | ||
339 | [MHD-single] trg 1653 fhandler_socket_inet::setsockopt: | ||
340 | setsockopt optval=1 | ||
341 | [MHD-single] trg 1653 cygwin_setsockopt: 0 = setsockopt(6, | ||
342 | 6, 0x1, 0xFFDFCBFC, 4) | ||
343 | - modest response generation: | ||
344 | [main] mrg 1649 fhandler_socket_inet::setsockopt: | ||
345 | setsockopt optval=1 | ||
346 | [main] mrg 1649 cygwin_setsockopt: 0 = setsockopt(5, | ||
347 | 65535, 0x4, 0xFFFFCA10, 4) | ||
348 | [MHD-single] mrg 1649 fhandler_socket_inet::setsockopt: | ||
349 | setsockopt optval=1 | ||
350 | [MHD-single] mrg 1649 cygwin_setsockopt: 0 = setsockopt(6, | ||
351 | 6, 0x1, 0xFFDFCBFC, 4) | ||
352 | - response generation using sendfile(): | ||
353 | [MHD-single] response_generation_sendfile 1657 | ||
354 | __set_winsock_errno: setsockopt:1689 - winsock error 10042 -> errno 109 | ||
355 | [MHD-single] response_generation_sendfile 1657 | ||
356 | cygwin_setsockopt: -1 = setsockopt(6, 6, 0x4, 0xFFDFCB5C, 4), errno 109 | ||
357 | [MHD-single] response_generation_sendfile 1657 | ||
358 | cygwin_send: 99 = send(6, 0x6000790C0, 99, 0x20) | ||
359 | [MHD-single] response_generation_sendfile 1657 | ||
360 | fhandler_disk_file::prw_open: 0x0 = NtOpenFile (0x264, 0x80100000, | ||
361 | \??\C:\cygwin64\home\ng0\src\gsoc2019\a.txt, io, 0x7, 0x4020) | ||
362 | [MHD-single] response_generation_sendfile 1657 | ||
363 | fhandler_disk_file::pread: 1 = pread(0x60007D148, 1, 0, 0x0) | ||
364 | [MHD-single] response_generation_sendfile 1657 pread: 1 = | ||
365 | pread(7, 0x60007D148, 1, 0) | ||
366 | [MHD-single] response_generation_sendfile 1657 | ||
367 | cygwin_send: 1 = send(6, 0x60007D148, 1, 0x20) | ||
368 | [MHD-single] response_generation_sendfile 1657 | ||
369 | __set_errno: void __set_winsock_errno(const char*, int):203 setting | ||
370 | errno 109 | ||
371 | [MHD-single] response_generation_sendfile 1657 | ||
372 | __set_winsock_errno: setsockopt:1689 - winsock error 10042 -> errno 109 | ||
373 | [MHD-single] response_generation_sendfile 1657 | ||
374 | cygwin_setsockopt: -1 = setsockopt(6, 6, 0x4, 0xFFDFCB58, 4), errno 109 | ||
375 | [MHD-single] response_generation_sendfile 1657 close: | ||
376 | close(7) | ||
377 | |||
378 | 4) future work to be done | ||
379 | |||
380 | In post_cork_setsockopt() and pre_cork_setsockopt() more of the possible | ||
381 | errno cases to catch must be handled. | ||
382 | |||
383 | In connection.c there is one last line which can be removed once the code in | ||
384 | socket_start_no_buffering_flush() has been dealt with (FreeBSD specific, | ||
385 | discussion about it did not conclude in my project time). | ||
386 | |||
387 | 5) additional notes | ||
388 | |||
389 | Even though some of the flags used are new'ish, | ||
390 | Steven's "Unix Network Programming" and the TCP/IP Illustrated volumes | ||
391 | were a great help for this work in addition to the documented source code | ||
392 | of the Operating Systems (where available). | ||
393 | Thanks to my fellow NetBSD developers who I could ask about system | ||
394 | and network specific features in NetBSD when the code wasn't enough. | ||