aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorng0@n0.is <ng0@n0.is>2019-08-21 14:32:29 +0000
committerng0@n0.is <ng0@n0.is>2019-08-21 14:32:29 +0000
commit297fc34bed8ca8260b0d94aea2aadda9720739f5 (patch)
treeb748a746737a902bb735793ba180fd40f247b22a
parent80dd88765f69f27f2f06433525518b7355b7d9f7 (diff)
downloadlibmicrohttpd-gsoc2019-gsoc-final.tar.gz
libmicrohttpd-gsoc2019-gsoc-final.zip
self-contain the report, soon to be sent to libmicrohttpd mailinglist.gsoc-final
-rw-r--r--report394
1 files changed, 394 insertions, 0 deletions
diff --git a/report b/report
new file mode 100644
index 0000000..5660ed4
--- /dev/null
+++ b/report
@@ -0,0 +1,394 @@
1Hi,
2
3here's my summary for the work I've done during
4Google Summer of Code 2019 for libmicrohttpd.
5(https://summerofcode.withgoogle.com/projects/#4926907403993088)
6
7As the communication with my mentor happened almost exclusively
8via private Email and Mumble Voicecalls, the gist of it is
9to optimize syscalls with focus on setsockopt optimization on
10various platforms (Debian Linux, FreeBSD 12-RELEASE-p3, NetBSD-CURRENT,
11cygwin on Windows 10). OSX/macOS is listed but was disregarded as
12the process would've meant losing too much time as I was simply
13not familiar with the process of setting up this in my test environment.
14
15The summary and analysis of the state of syscalls can be found at
16https://d.n0.is/pub/libmicrohttpd/gsoc2019/syscalls.html and
17checked out via mercurial:
18hg clone https://c.n0.is/libmicrohttpd/gsoc2019
19
20The tests I wrote to analyse the sycalls focused on 4 scenarios:
211. continuous response generation (ensure no constant setsockopt() calls
22 during transmission)
232. tiny response generation (fits in one packet, including header,
24 ensure one packet and sending without corking)
253. modest response generation (header first, then body, ensure last part
26 of body is sent without corking)
274. response generation using sendfile() (making sure that after last write
28 operation the packet is sent without corking, no unnecessary setsockopts)
29
301) initial syscall assessment
31
32Initially, before changing the code, it can be observed that we make too
33many syscalls of setsockopt() on Linux (Debian) and other platforms.
34The setsockopt() calls on a platform which supports MSG_MORE (Linux) are
35unnecessary high, and on FreeBSD TCP_NOPUSH is not used.
36The way the syscalls are called is unnecessary expensive.
37
38* Linux (Debian)
39- Continuous Response Generation test shows a pattern of setsockopt, sendto,
40 setsockopt, setsockopt, setsockopt, setsockopt, setsockopt, setsockopt.
41 After this the body of the message is send, and we see 2 more setsockopt()
42 calls at the end.
43 (9 setsockopt calls)
44
45- Tiny Response Generation test starts with 1 setsockopt() call after
46 receiving the GET request (TCP_CORK).
47 After sending the header, we see 6 setsockopt() calls.
48 Again after sending the body see 2 final setsockopt() calls.
49 (9 setsockopt calls)
50
51- Modest Response Generation test starts with 1 setsockopt() call after
52 receiving the GET request. We send the header. 6 calls to setsockopt()
53 follow. The body is send, and now we get 2 more setsockopt() calls
54 at the end.
55 (9 setsockopt calls)
56
57- Response Generation using sendfile() test starts with 1 call to
58 setsockopt() after the GET request is received, immediately before
59 the header is send. Then we see 6 calls to setsockopt().
60 sendfile() is called, setsockopt() is called with TCP_NODELAY.
61 The filedescriptor is closed. setsockopt() is called with
62 TCP_NODELAY.
63 (9 setsockopt calls)
64
65* FreeBSD
66- Continuous Response Generation test shows that setsockopt() is called
67 after receiving the GET request, 1 call immediately before sending the
68 header, 1 call immediately after sending the header (TCP_NODELAY).
69 The body is send, and no further setsockopt() call is made.
70 (3 setsockopt calls)
71
72- Tiny Response Generation test has a comparable pattern: after receiving
73 the GET request, 1 call is made to setsockopt() immediately before calling
74 sendto() on the header and 1 call is made to setsockopt() immediately
75 after sending the header. No further setsockopt() calls are made.
76 (2 setsockopt calls)
77
78- Modest Response Generation test shows a comparable pattern: after
79 receiving the GET request, 1 call is made to setsockopt() immediately
80 before calling sendto() on the header and 1 call is made to setsockopt()
81 immediately after sending the header.
82 No further setsockopt() calls are made.
83 (2 setsockopt calls)
84
85- Response Generation using sendfile() result is comparable to the 3 tests
86 run before it.
87 (2 setsockopt calls)
88
89* NetBSD
90- Continuous Response Generation test shows 1 call to setsockopt()
91 before sending the header, and 1 call to setsockopt() after
92 sending the header.
93 (2 setsockopt calls)
94
95- Tiny Response Generation test shows that after receiving the
96 GET request, we see 1 call to setsockopt, before sending
97 the header. Immediately after the header see 2 calls to
98 setsockopt(). After the body is send we see 1 more call to
99 setsockopt().
100 (4 setsockopt calls)
101
102- Modest Response Generation test shows that after receiving the
103 GET request, we see 1 setsockopt() call which is followed by
104 sending the header. This is followed by 2 calls to setsockopt().
105 After this, the body is send, followed by another call to setsockopt().
106 (4 setsockopt calls)
107
108- Response Generation using sendfile() shows that after receiving the
109 GET request, the file with the content "a" is created.
110 We see 1 call to setsockopt() just before the header is send.
111 Immediately after this we see 1 more setsockopt call.
112 After a call to pread() we see another setsockopt call
113 followed by a sendto of "a".
114 (3 setsockopt calls)
115
116* cygwin x64 (under Windows 10)
117- Continuous Response Generation test shows 1 call of main to
118 setsockopt() in the beginning.
119 After this, MHD-single calls setsockopt 2 times
120 (setsockopt, followed by cygwin_send, followed by another
121 setsockopt call). The pattern in cygwin is comparable to the
122 ktruss log in NetBSD, where we see a setsockopt, sendto,
123 setsockopt at the beginning of the log.
124 (3 setsockopt calls)
125
126- Tiny Response Generation test shows 1 call to setsockopt.
127 This is followed by a send of the size 99 (the header). Then follows
128 another setsockopt, followed by another call to send (the body).
129 (2 setsockopt calls)
130
131- Modest Response Generation test shows 2 calls to setsockopt() related
132 to the socket we use, each one of them before the respective send() is
133 called.
134 (2 setsockopt calls)
135
136- Response Generation using sendfile() shows 1 call to setsockopt(),
137 followed by a send() with size 99 (the header).
138 This is followed by 1 call to setsockopt(), and one more call to send()
139 with size 1 (the body).
140 (2 setsockopt calls)
141
1422) approach taken, description of the changes made,
143
144I looked into the system specific tweaks we could start with. This included
145looking and reading into the documentation and source changelogs of the
146targeted Operating Systems and their TCP/IP stack implementation.
147It was concluded that MSG_MORE should be prefered on Linux if it exists,
148and get priority over any other existing tcp socket flags.
149TCP_NODELAY is considered for toggling Naggle's algorithm on or off.
150On FreeBSD I had to read into additions to the stack in the last 10+ years,
151and TCP_NOPUSH was found to be the closest to the Linux specific TCP_CORK.
152Both TCP_NOPUSH and TCP_CORK deal with not sending out partial frames if
153set, an optimization to consider.
154Through incremental testing, rewriting and reasoning about the best way
155the systems should be adressed, I got to the endresult.
156
157Two helper functions were written (pre_cork_setsockopt(),
158post_cork_setsockopt()) to handle setsockopt() calls.
159pre_cork_setsockopt() is called before any send()/sendmsg() calls.
160post_cork_setsockopt() is called after any send()/sendmsg() calls. If the
161MSG_MORE option is supported on this platform, we do nothing in those
162functions.
163If the sk_cork_on boolean is already the state we want to set, we
164do nothing.
165An additional helper function was added to toggle Naggle's Algorithm,
166MHD_socket_set_nodelay_().
167In the pre_cork_setsockopt() function, if TCP_CORK/TCP_NOPUSH/MSG_MORE
168do not exist on a platform, we set Naggle to on/off,
169in all other cases we simply set Naggle to always off. This is achieved
170with a third helper function, MHD_socket_set_nodelay_().
171
172Two function were written to handle send() calls.
173MHD_send_on_connection_() sends a given buffer on connection and
174remembers the current state of the socket options.
175setsockopt() is only called when absolutely necessary.
176If the connection is using TLS, GnuTLS handles the corking.
177In all other cases (ie plaintext transmission) we call
178pre_cork_setsockopt(), followed by the send(), and post_cork_setsockopt().
179
180MHD_send_on_connection2_() sends the header followed by the buffer on the
181connection given.
182When sendmsg() is available, sendmsg() is used to send
183both (header and buffer) at once and returns the number of bytes sent
184from both buffers.
185Otherwise if writev() is available, writev() is used to send
186both (header and buffer) at once and returns the number of bytes sent
187from both buffers or -1 on error.
188If sendmsg() or writev() are unavailable, only the header is send via
189MHD_send_on_connection_().
190If we have writev or sendmsg, we call pre_cork_setsockopt with 'false'
191for the want_cork argument.
192If we succeeded in sending the full buffer, we call post_cork_setsockopt()
193again with 'false' for the want_cork argument, as we want to make sure
194that the OS flushes at the end.
195
196The previously existing function for sendfile() wrapping in connection.c was
197moved and renamed to MHD_send_sendfile_().
198
199The functions were then tested and replaced all previous setsockopt calls in
200connection.c. In connection.c, the following functions were removed:
201- socket_start_normal_buffering
202- socket_start_no_buffering
203
204and calls of both of them in connection.c were removed, in addition to
205removing calls of socket_start_no_buffering_flush() and
206socket_start_extra_buffering().
207
2083) resulting syscall behavior
209
210Overall we see an improvement to the number of times setsockopt() is
211called as well as to the time it is called.
212On Linux (Debian) setsockopt() for the measurement we did is no longer
213called, as expected. The only setsockopt() call logged is provided as
214reference. The MSG_MORE flag related code is working.
215
216Compared to the first tests on FreeBSD in most testcases we now get no
217setsockopt() calls after the header.
218
219On NetBSD the comparison before/after shows that for the sendfile() test the
220order of the setsockopt call changed and we have 1 call less. For the tiny
221response generation test we have 2 calls less to setsocktop() than before,
222the same applies for modest response generation.
223
224On cygwin the visible improvements include for tiny response generation
225only 1 call to setsockopt() instead of 2 setsockopt() calls.
226
227* Linux (Debian)
228- continuous response generation:
229 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
230 before header is send.
231 sendto() with MSG_NOSIGNAL|MSG_MORE for header.
232 sendto() with MSG_NOSIGNAL for the body.
233 No other setsockopt() calls.
234- tiny response generation:
235 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
236 before the header is send, no other setsockopt() calls.
237- modest response generation
238 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
239 before the header is send, no further setsockopt() calls.
240- response generation using sendfile()
241 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
242 before the header is send, no further setsockopt() calls.
243
244* FreeBSD
245- continuous response generation:
246 setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0)
247 setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0)
248 Before header is send:
249 setsockopt(4,IPPROTO_TCP,TCP_FASTOPEN_MIN_COOKIE_LEN,0x7fffdfffde5c,4) = 0 (0x0)
250 header:
251 sendto(4,"HTTP/1.1 200 OK\r\nConnection: K"...,108,MSG_NOSIGNAL,NULL,0) = 108 (0x6c)
252 followed by a pattern of:
253 sendto(4,"1\r\nb\r\n",6,MSG_NOSIGNAL,NULL,0) = 6 (0x6)
254 poll({ 3/POLLIN 4/POLLPRI|POLLOUT|POLLRDBAND },2,-1) = 1 (0x1)
255 Ends with another setsockopt call:
256 setsockopt(4,IPPROTO_TCP,TCP_FASTOPEN_MIN_COOKIE_LEN,0x7fffdfffde5c,4) = 0 (0x0)
257- tiny response generation:
258 setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0)
259 setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0)
260 before the header is send.
261 No further setsockopt() calls.
262- modest response generation:
263 setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0)
264 setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0)
265 before the header is send.
266 No further setsockopt() calls.
267- response generation using sendfile():
268 setsockopt(4,IPPROTO_TCP,TCP_NODELAY,0x7fffdfffdecc,4) = 0 (0x0)
269 setsockopt(4,SOL_SOCKET,SO_NOSIGPIPE,0x800249f94,4) = 0 (0x0)
270 before the header is send.
271 setsockopt(4,IPPROTO_TCP,TCP_FASTOPEN_MIN_COOKIE_LEN,0x7fffdfffde5c,4) = 0 (0x)
272 before the HTTP/1.1 200 OK is send.
273
274* NetBSD
275- continuous response generation:
276 before receiving the GET request:
277 setsockopt(0xa, 0xffff, 0x800, 0x7e25e041a1bc, 0x4) = 0
278 After receiving the GET request:
279 setsockopt(0xa, 0x6, 0x1, 0x7e25dcbffec8, 0x4) = 0
280 sendto(0xa, 0x7e25e067f800, 0x6c, 0x400, 0, 0) = 108
281 "HTTP/1.1 200 OK\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunk"
282 sendto(0xa, 0x7e25e067f807, 0x6, 0x400, 0, 0) = 6
283 "1\r\nb\r\n"
284 setsockopt(0xa, 0x6, 0x1, 0x7e25dcbffecc, 0x4) = 0
285 poll(0x7e25e067a000, 0x2, 0xffffffff) = 1
286 sendto(0xa, 0x7e25e067f807, 0x6, 0x400, 0, 0) = 6
287 "1\r\nb\r\n"
288 poll(0x7e25e067a000, 0x2, 0xffffffff) = 1
289 sendto(0xa, 0x7e25e067f807, 0x6, 0x400, 0, 0) = 6
290 "1\r\nb\r\n"
291
292 and so forth. At the end one last setsockopt() call:
293 setsockopt(0xa, 0x6, 0x1, 0x7e25dcbffec8, 0x4) = 0
294- tiny response generation:
295 before GET request:
296 setsockopt(0xa, 0xffff, 0x800, 0x7986b2e1a1bc, 0x4) = 0
297 No other setsockopt() call.
298- modest response generation:
299 before receiving the GET request:
300 setsockopt(0xa, 0xffff, 0x800, 0x7efa1c01a1bc, 0x4) = 0
301 No other setsockopt() calls.
302- response generation using sendfile():
303 before receiving the GET request:
304 setsockopt(0xa, 0xffff, 0x800, 0x723db1e1a1bc, 0x4) = 0
305 After receiving the GET request:
306 setsockopt(0xa, 0x6, 0x1, 0x723dae5ffec8, 0x4) = 0
307 sendto(0xa, 0x723db21052c0, 0x63, 0x400, 0, 0) = 99
308 "HTTP/1.1 200 OK\r\nConnection: Keep-Alive\r\nContent-Length: 1\r\nDat"
309 pread(0xb, 0x723db20fb0a0, 0x1, 0, 0) = 1
310 "a"
311 sendto(0xa, 0x723db20fb0a0, 0x1, 0x400, 0, 0) = 1
312 "a"
313 setsockopt(0xa, 0x6, 0x1, 0x723dae5ffecc, 0x4) = 0
314 close(0xb) = 0
315
316* cygwin x64 (under Windows 10)
317- continuous response generation:
318 [MHD-single] inc 1643 __set_winsock_errno: setsockopt:1689
319 - winsock error 10042 -> errno 109
320 [MHD-single] inc 1643 cygwin_setsockopt: -1 =
321 setsockopt(6, 6, 0x4, 0xFFDFCB5C, 4), errno 109
322 [MHD-single] inc 1643 cygwin_send: 108 = send(6,
323 0x6000790A0, 108, 0x20)
324 [MHD-single] inc 1643 cygwin_send: 6 = send(6,
325 0x6000790A7, 6, 0x20)
326 [MHD-single] inc 1643 __set_errno: void
327 __set_winsock_errno(const char*, int):203 setting errno 109
328 [MHD-single] inc 1643 __set_winsock_errno: setsockopt:1689
329 - winsock error 10042 -> errno 109
330 [MHD-single] inc 1643 cygwin_setsockopt: -1 =
331 setsockopt(6, 6, 0x4, 0xFFDFCB58, 4), errno 109
332
333 and at the end:
334 [MHD-single] inc 1643 __set_winsock_errno:
335 setsockopt:1689 - winsock error 10042 -> errno 109
336 [MHD-single] inc 1643 cygwin_setsockopt: -1 =
337 setsockopt(6, 6, 0x4, 0xFFDFCB5C, 4), errno 109
338- tiny response generation:
339 [MHD-single] trg 1653 fhandler_socket_inet::setsockopt:
340 setsockopt optval=1
341 [MHD-single] trg 1653 cygwin_setsockopt: 0 = setsockopt(6,
342 6, 0x1, 0xFFDFCBFC, 4)
343- modest response generation:
344 [main] mrg 1649 fhandler_socket_inet::setsockopt:
345 setsockopt optval=1
346 [main] mrg 1649 cygwin_setsockopt: 0 = setsockopt(5,
347 65535, 0x4, 0xFFFFCA10, 4)
348 [MHD-single] mrg 1649 fhandler_socket_inet::setsockopt:
349 setsockopt optval=1
350 [MHD-single] mrg 1649 cygwin_setsockopt: 0 = setsockopt(6,
351 6, 0x1, 0xFFDFCBFC, 4)
352- response generation using sendfile():
353 [MHD-single] response_generation_sendfile 1657
354 __set_winsock_errno: setsockopt:1689 - winsock error 10042 -> errno 109
355 [MHD-single] response_generation_sendfile 1657
356 cygwin_setsockopt: -1 = setsockopt(6, 6, 0x4, 0xFFDFCB5C, 4), errno 109
357 [MHD-single] response_generation_sendfile 1657
358 cygwin_send: 99 = send(6, 0x6000790C0, 99, 0x20)
359 [MHD-single] response_generation_sendfile 1657
360 fhandler_disk_file::prw_open: 0x0 = NtOpenFile (0x264, 0x80100000,
361 \??\C:\cygwin64\home\ng0\src\gsoc2019\a.txt, io, 0x7, 0x4020)
362 [MHD-single] response_generation_sendfile 1657
363 fhandler_disk_file::pread: 1 = pread(0x60007D148, 1, 0, 0x0)
364 [MHD-single] response_generation_sendfile 1657 pread: 1 =
365 pread(7, 0x60007D148, 1, 0)
366 [MHD-single] response_generation_sendfile 1657
367 cygwin_send: 1 = send(6, 0x60007D148, 1, 0x20)
368 [MHD-single] response_generation_sendfile 1657
369 __set_errno: void __set_winsock_errno(const char*, int):203 setting
370 errno 109
371 [MHD-single] response_generation_sendfile 1657
372 __set_winsock_errno: setsockopt:1689 - winsock error 10042 -> errno 109
373 [MHD-single] response_generation_sendfile 1657
374 cygwin_setsockopt: -1 = setsockopt(6, 6, 0x4, 0xFFDFCB58, 4), errno 109
375 [MHD-single] response_generation_sendfile 1657 close:
376 close(7)
377
3784) future work to be done
379
380In post_cork_setsockopt() and pre_cork_setsockopt() more of the possible
381errno cases to catch must be handled.
382
383In connection.c there is one last line which can be removed once the code in
384socket_start_no_buffering_flush() has been dealt with (FreeBSD specific,
385discussion about it did not conclude in my project time).
386
3875) additional notes
388
389Even though some of the flags used are new'ish,
390Steven's "Unix Network Programming" and the TCP/IP Illustrated volumes
391were a great help for this work in addition to the documented source code
392of the Operating Systems (where available).
393Thanks to my fellow NetBSD developers who I could ask about system
394and network specific features in NetBSD when the code wasn't enough.