smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Locked
smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
 
|
Hi All,
I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened: * sendfile() first returns 35 (EAGAIN): # dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' dtrace: description 'syscall::sendfile:return ' matched 1 probe CPU ID FUNCTION:NAME 23 64274 sendfile:return libc.so.7`__sys_sendfile+0xa libsmbd-base-samba4.so`vfswrap_sendfile+0x127 libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46 libtevent.so.0.9.31`poll_event_loop_poll+0x75c errno = 35 * then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN: # dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' 16 63748 writev:return libc.so.7`_writev+0xa libsys-rw-samba4.so`sys_writev+0x21 libsys-rw-samba4.so`write_data_iov+0x88 libsys-rw-samba4.so`write_data+0x39 libsmbd-base-samba4.so`fake_sendfile+0x16e libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 errno = 35 * and this is the stack trace of smbd: # echo bt | lldb -p 1810 (lldb) process attach --pid 1810 Process 1810 stopped Executable module set to "/tmw-nas-3p/samba/sbin/smbd". Architecture set to: x86_64--freebsd11.1. (lldb) bt * thread #1 * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10 frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634 frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101 frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49 frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94 frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329 frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283 frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078 frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593 frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104 frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174 frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716 frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697 frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011 frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159 frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164 frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120 frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387 frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109 frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641 frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950 frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988 frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605 frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662 frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721 frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678 frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863 frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140 frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024 frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605 frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662 frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721 frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678 frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863 frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391 frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050 frame #36: 0x0000000001027cf0 smbd`_start + 384 In my smb.conf, I have socket options set as follows: socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5 By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug? Thanks, --Youzhong |
RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
|
|
Sorry, I meant:
if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened. From: Youzhong Yang Sent: Friday, November 03, 2017 4:27 PM To: [hidden email] Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf Hi All, I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened: * sendfile() first returns 35 (EAGAIN): # dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' dtrace: description 'syscall::sendfile:return ' matched 1 probe CPU ID FUNCTION:NAME 23 64274 sendfile:return libc.so.7`__sys_sendfile+0xa libsmbd-base-samba4.so`vfswrap_sendfile+0x127 libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46 libtevent.so.0.9.31`poll_event_loop_poll+0x75c errno = 35 * then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN: # dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' 16 63748 writev:return libc.so.7`_writev+0xa libsys-rw-samba4.so`sys_writev+0x21 libsys-rw-samba4.so`write_data_iov+0x88 libsys-rw-samba4.so`write_data+0x39 libsmbd-base-samba4.so`fake_sendfile+0x16e libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 errno = 35 * and this is the stack trace of smbd: # echo bt | lldb -p 1810 (lldb) process attach --pid 1810 Process 1810 stopped Executable module set to "/tmw-nas-3p/samba/sbin/smbd". Architecture set to: x86_64--freebsd11.1. (lldb) bt * thread #1 * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10 frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634 frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101 frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49 frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94 frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329 frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283 frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078 frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593 frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104 frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174 frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716 frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697 frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011 frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159 frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164 frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120 frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387 frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109 frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641 frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950 frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988 frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605 frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662 frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721 frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678 frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863 frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140 frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024 frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605 frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662 frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721 frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678 frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863 frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391 frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050 frame #36: 0x0000000001027cf0 smbd`_start + 384 In my smb.conf, I have socket options set as follows: socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5 By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug? Thanks, --Youzhong |
RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
|
|
Don't know if it's relevant, the manpage of sendfile() says:
HISTORY The sendfile() system call first appeared in FreeBSD 3.0. This manual page first appeared in FreeBSD 3.1. In FreeBSD 10 support for sending shared memory descriptors had been introduced. In FreeBSD 11 a non- blocking implementation had been introduced. From: Youzhong Yang Sent: Friday, November 03, 2017 4:28 PM To: [hidden email] Subject: RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf Sorry, I meant: if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened. From: Youzhong Yang Sent: Friday, November 03, 2017 4:27 PM To: [hidden email]<mailto:[hidden email]> Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf Hi All, I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened: * sendfile() first returns 35 (EAGAIN): # dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' dtrace: description 'syscall::sendfile:return ' matched 1 probe CPU ID FUNCTION:NAME 23 64274 sendfile:return libc.so.7`__sys_sendfile+0xa libsmbd-base-samba4.so`vfswrap_sendfile+0x127 libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46 libtevent.so.0.9.31`poll_event_loop_poll+0x75c errno = 35 * then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN: # dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' 16 63748 writev:return libc.so.7`_writev+0xa libsys-rw-samba4.so`sys_writev+0x21 libsys-rw-samba4.so`write_data_iov+0x88 libsys-rw-samba4.so`write_data+0x39 libsmbd-base-samba4.so`fake_sendfile+0x16e libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 errno = 35 * and this is the stack trace of smbd: # echo bt | lldb -p 1810 (lldb) process attach --pid 1810 Process 1810 stopped Executable module set to "/tmw-nas-3p/samba/sbin/smbd". Architecture set to: x86_64--freebsd11.1. (lldb) bt * thread #1 * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10 frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634 frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101 frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49 frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94 frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329 frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283 frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078 frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593 frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104 frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174 frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716 frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697 frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011 frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159 frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164 frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120 frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387 frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109 frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641 frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950 frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988 frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605 frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662 frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721 frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678 frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863 frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140 frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024 frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605 frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662 frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721 frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678 frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863 frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391 frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050 frame #36: 0x0000000001027cf0 smbd`_start + 384 In my smb.conf, I have socket options set as follows: socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5 By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug? Thanks, --Youzhong |
RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
|
|
In reply to this post by Samba - samba-technical mailing list
Hi Timur,
That’s too bad. Yes, I am using zfs on freebsd. This one explains why: https://wiki.freebsd.org/DevSummit/20170907/ZFS “ZFS currently does not play nicely with sendfile(2). The idea with sendfile(2) is to share pages between the buffer cache and the mbuf, to prevent copying. The call results in a range of bytes from an open file descriptor being faulted into the buffer cache, and then those pages being shared all the way through to the network stack until the bytes are transmitted. With ZFS, since it does not use the OS buffer cache, data is copied from the ZFS ARC to the buffer cache, where it is then shared with the mbuf etc. Solaris has a solution to solve this issue, and it might make sense to adapt our sendfile mechanism to work in a similar way. [Solaris PSARC: Copy Reduction Interface](https://www.mail-archive.com/opensolaris-arc@.../msg17203.html) This creates two new VFS operations, reqzcbuf which asks the filesystem to prepare a UIO for zero copy operations. In the case of ZFS, this involves allocating a new array to track the ARC headers of each buffer that is loaned out from the ZFS ARC.” But it’s really bad that using sendfile() causes Samba to hang. Thanks, -Youzhong From: Timur I. Bakeyev [mailto:[hidden email]] Sent: Sunday, November 05, 2017 7:23 PM To: Youzhong Yang <[hidden email]> Cc: [hidden email] Subject: Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf Hi, Youzhong! You didn't specify what file system do you use. Please, keep in mind that ZFS doesn't support 'sendfile' properly. With regards, Timur Bakeyev On Fri, Nov 3, 2017 at 9:28 PM, Youzhong Yang via samba-technical <[hidden email]<mailto:[hidden email]>> wrote: Sorry, I meant: if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened. From: Youzhong Yang Sent: Friday, November 03, 2017 4:27 PM To: [hidden email]<mailto:[hidden email]> Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf Hi All, I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened: * sendfile() first returns 35 (EAGAIN): # dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' dtrace: description 'syscall::sendfile:return ' matched 1 probe CPU ID FUNCTION:NAME 23 64274 sendfile:return libc.so.7`__sys_sendfile+0xa libsmbd-base-samba4.so`vfswrap_sendfile+0x127 libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46 libtevent.so.0.9.31`poll_event_loop_poll+0x75c errno = 35 * then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN: # dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}' 16 63748 writev:return libc.so.7`_writev+0xa libsys-rw-samba4.so`sys_writev+0x21 libsys-rw-samba4.so`write_data_iov+0x88 libsys-rw-samba4.so`write_data+0x39 libsmbd-base-samba4.so`fake_sendfile+0x16e libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a libtalloc.so.2.1.9`_tc_free_internal+0x152 libtalloc.so.2.1.9`_tc_free_children_internal+0xac libtalloc.so.2.1.9`_tc_free_internal+0x331 libtalloc.so.2.1.9`_talloc_free_internal+0xb2 libtalloc.so.2.1.9`_talloc_free+0x114 libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 errno = 35 * and this is the stack trace of smbd: # echo bt | lldb -p 1810 (lldb) process attach --pid 1810 Process 1810 stopped Executable module set to "/tmw-nas-3p/samba/sbin/smbd". Architecture set to: x86_64--freebsd11.1. (lldb) bt * thread #1 * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10 frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634 frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101 frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49 frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94 frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329 frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283 frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078 frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593 frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104 frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174 frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716 frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697 frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011 frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159 frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164 frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120 frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387 frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109 frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641 frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950 frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988 frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605 frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662 frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721 frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678 frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863 frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140 frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024 frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605 frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662 frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721 frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678 frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863 frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391 frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050 frame #36: 0x0000000001027cf0 smbd`_start + 384 In my smb.conf, I have socket options set as follows: socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5 By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug? Thanks, --Youzhong |
Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
|
|
Yes, I'm aware of the inefficiency of sendfile() on ZFS, but hanging of
Samba is something new. I'd suggest you to file a bug on bugzilla.samba.org regarding that. With best regards, Timur Bakeyev. On Mon, Nov 6, 2017 at 1:51 AM, Youzhong Yang <[hidden email]> wrote: > Hi Timur, > > > > That’s too bad. Yes, I am using zfs on freebsd. > > > > This one explains why: > > > > https://wiki.freebsd.org/DevSummit/20170907/ZFS > > > > “ZFS currently does not play nicely with sendfile(2). The idea with > sendfile(2) is to share pages between the buffer cache and the mbuf, to > prevent copying. The call results in a range of bytes from an open file > descriptor being faulted into the buffer cache, and then those pages being > shared all the way through to the network stack until the bytes are > transmitted. > > With ZFS, since it does not use the OS buffer cache, data is copied from > the ZFS ARC to the buffer cache, where it is then shared with the mbuf > etc. Solaris has a solution to solve this issue, and it might make sense to > adapt our sendfile mechanism to work in a similar way. [Solaris PSARC: Copy > Reduction Interface](https://www.mail-archive.com/opensolaris-arc@ > mail.opensolaris.org/msg17203.html) This creates two new VFS operations, > reqzcbuf which asks the filesystem to prepare a UIO for zero copy > operations. In the case of ZFS, this involves allocating a new array to > track the ARC headers of each buffer that is loaned out from the ZFS ARC.” > > But it’s really bad that using sendfile() causes Samba to hang. > > Thanks, > > -Youzhong > > *From:* Timur I. Bakeyev [mailto:[hidden email]] > *Sent:* Sunday, November 05, 2017 7:23 PM > *To:* Youzhong Yang <[hidden email]> > *Cc:* [hidden email] > *Subject:* Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' > in smb.conf > > > > Hi, Youzhong! > > You didn't specify what file system do you use. Please, keep in mind that > ZFS doesn't support 'sendfile' properly. > > > > With regards, > > Timur Bakeyev > > > > On Fri, Nov 3, 2017 at 9:28 PM, Youzhong Yang via samba-technical < > [hidden email]> wrote: > > Sorry, I meant: > > if 'use sendfile' is set to no, everything works well. But if it is set to > yes, here is what happened. > > From: Youzhong Yang > Sent: Friday, November 03, 2017 4:27 PM > To: [hidden email] > Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in > smb.conf > > > Hi All, > > I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if > 'use sendfile' is set to yes, everything works well. But if it is set to > no, here is what happened: > > > * sendfile() first returns 35 (EAGAIN): > > # dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); > printf("errno = %d", errno);}' > dtrace: description 'syscall::sendfile:return ' matched 1 probe > CPU ID FUNCTION:NAME > 23 64274 sendfile:return > libc.so.7`__sys_sendfile+0xa > libsmbd-base-samba4.so`vfswrap_sendfile+0x127 > libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d > libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e > libtalloc.so.2.1.9`_tc_free_internal+0x152 > libtalloc.so.2.1.9`_tc_free_children_internal+0xac > libtalloc.so.2.1.9`_tc_free_internal+0x331 > libtalloc.so.2.1.9`_talloc_free_internal+0xb2 > libtalloc.so.2.1.9`_talloc_free+0x114 > libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a > libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 > libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b > libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f > libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c > libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f > libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef > libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 > libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 > libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46 > libtevent.so.0.9.31`poll_event_loop_poll+0x75c > errno = 35 > > > * then smbd keeps calling writev(), which always returns -1 with errno > being EAGAIN: > > # dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno > = %d", errno);}' > 16 63748 writev:return > libc.so.7`_writev+0xa > libsys-rw-samba4.so`sys_writev+0x21 > libsys-rw-samba4.so`write_data_iov+0x88 > libsys-rw-samba4.so`write_data+0x39 > libsmbd-base-samba4.so`fake_sendfile+0x16e > libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a > libtalloc.so.2.1.9`_tc_free_internal+0x152 > libtalloc.so.2.1.9`_tc_free_children_internal+0xac > libtalloc.so.2.1.9`_tc_free_internal+0x331 > libtalloc.so.2.1.9`_talloc_free_internal+0xb2 > libtalloc.so.2.1.9`_talloc_free+0x114 > libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a > libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886 > libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b > libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f > libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c > libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f > libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef > libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03 > libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2 > errno = 35 > > > * and this is the stack trace of smbd: > > # echo bt | lldb -p 1810 > (lldb) process attach --pid 1810 > Process 1810 stopped > > Executable module set to "/tmw-nas-3p/samba/sbin/smbd". > Architecture set to: x86_64--freebsd11.1. > (lldb) bt > * thread #1 > * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10 > frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, > iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634 > frame #2: libsys-rw-samba4.so`sys_writev(fd=40, > iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101 > frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, > orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49 > frame #4: libsys-rw-samba4.so`write_data(fd=40, > buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94 > frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, > fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329 > frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) > at smb2_read.c:283 > frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, > location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078 > frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, > ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at > talloc.c:1593 > frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, > location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104 > frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, > location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174 > frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, > location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716 > frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) > at smb2_server.c:3697 > frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) > at smb2_server.c:3011 > frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, > status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, > location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159 > frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) > at smb2_read.c:164 > frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, > location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120 > frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, > subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387 > frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) > at smb2_read.c:109 > frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) > at smb2_server.c:2641 > frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, > fde_flags=1) at smb2_server.c:3950 > frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, > fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at > smb2_server.c:3988 > frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, > tvalp=0x00007fffffffe068) at tevent_poll.c:605 > frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, > location="../source3/smbd/process.c:4140") at tevent_poll.c:662 > frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, > location="../source3/smbd/process.c:4140") at tevent.c:721 > frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, > location="../source3/smbd/process.c:4140") at tevent_poll.c:678 > frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, > location="../source3/smbd/process.c:4140") at tevent.c:863 > frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, > msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140 > frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, > fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at > server.c:1024 > frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, > tvalp=0x00007fffffffe548) at tevent_poll.c:605 > frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, > location="../source3/smbd/server.c:1391") at tevent_poll.c:662 > frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, > location="../source3/smbd/server.c:1391") at tevent.c:721 > frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, > location="../source3/smbd/server.c:1391") at tevent_poll.c:678 > frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, > location="../source3/smbd/server.c:1391") at tevent.c:863 > frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, > parent=0x0000000812c52680) at server.c:1391 > frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050 > frame #36: 0x0000000001027cf0 smbd`_start + 384 > > In my smb.conf, I have socket options set as follows: > > socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 > SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5 > > By the way, I am running Samba 4.6.8. Has anyone experienced the same > issue? Is it a FreeBSD kernel bug? > > Thanks, > > --Youzhong > > > |
Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
|
|
In reply to this post by Samba - samba-technical mailing list
On 2017-11-03 at 20:26 +0000 Youzhong Yang via samba-technical sent off:
> I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened: is https://github.com/samba-team/samba/pull/76 helping you maybe? Björn |
RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
|
|
I tried your patch, yes, it works, no more spinning.
Thanks! Is there any chance this patch will get integrated soon? -----Original Message----- From: Björn JACKE [mailto:[hidden email]] Sent: Friday, November 10, 2017 8:36 PM To: Youzhong Yang <[hidden email]> Cc: [hidden email] Subject: Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf On 2017-11-03 at 20:26 +0000 Youzhong Yang via samba-technical sent off: > I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened: is https://github.com/samba-team/samba/pull/76 helping you maybe? Björn |
Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
|
|
Hi,
On 2017-11-11 at 16:18 +0000 Youzhong Yang via samba-technical sent off: > I tried your patch, yes, it works, no more spinning. it's not my patch, I just remembered having see that pull request some time ago. > Thanks! Is there any chance this patch will get integrated soon? you should open a bug report in bugzilla.samba.org describing your problem and point to that patch set and try to get the attention of a developer to review and apply the patch set. Björn |
«
Return to Samba - samba-technical
|
1 view|%1 views
| Free forum by Nabble | Edit this page |