smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
Hi All,

I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:


  *   sendfile() first returns 35 (EAGAIN):

# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU     ID                    FUNCTION:NAME
23  64274                  sendfile:return
              libc.so.7`__sys_sendfile+0xa
              libsmbd-base-samba4.so`vfswrap_sendfile+0x127
              libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
              libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
              libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35


  *   then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:

# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16  63748                    writev:return
              libc.so.7`_writev+0xa
              libsys-rw-samba4.so`sys_writev+0x21
              libsys-rw-samba4.so`write_data_iov+0x88
              libsys-rw-samba4.so`write_data+0x39
              libsmbd-base-samba4.so`fake_sendfile+0x16e
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35


  *   and this is the stack trace of smbd:

# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped

Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
  * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
    frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
    frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
    frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
    frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
    frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
    frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
    frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
    frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
    frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
    frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
    frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
    frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
    frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
    frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
    frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
    frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
    frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
    frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
    frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
    frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
    frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
    frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
    frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
    frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
    frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
    frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
    frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
    frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
    frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
    frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
    frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
    frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
    frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
    frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
    frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
    frame #36: 0x0000000001027cf0 smbd`_start + 384

In my smb.conf, I have socket options set as follows:

socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5

By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?

Thanks,

--Youzhong

Reply | Threaded
Open this post in threaded view
|

RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
Sorry, I meant:

if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened.

From: Youzhong Yang
Sent: Friday, November 03, 2017 4:27 PM
To: [hidden email]
Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Hi All,

I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:


  *   sendfile() first returns 35 (EAGAIN):

# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU     ID                    FUNCTION:NAME
23  64274                  sendfile:return
              libc.so.7`__sys_sendfile+0xa
              libsmbd-base-samba4.so`vfswrap_sendfile+0x127
              libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
              libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
              libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35


  *   then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:

# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16  63748                    writev:return
              libc.so.7`_writev+0xa
              libsys-rw-samba4.so`sys_writev+0x21
              libsys-rw-samba4.so`write_data_iov+0x88
              libsys-rw-samba4.so`write_data+0x39
              libsmbd-base-samba4.so`fake_sendfile+0x16e
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35


  *   and this is the stack trace of smbd:

# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped

Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
  * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
    frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
    frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
    frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
    frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
    frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
    frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
    frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
    frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
    frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
    frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
    frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
    frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
    frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
    frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
    frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
    frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
    frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
    frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
    frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
    frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
    frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
    frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
    frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
    frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
    frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
    frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
    frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
    frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
    frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
    frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
    frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
    frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
    frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
    frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
    frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
    frame #36: 0x0000000001027cf0 smbd`_start + 384

In my smb.conf, I have socket options set as follows:

socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5

By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?

Thanks,

--Youzhong

Reply | Threaded
Open this post in threaded view
|

RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
Don't know if it's relevant, the manpage of sendfile() says:

HISTORY
     The sendfile() system call first appeared in FreeBSD 3.0.  This manual
     page first appeared in FreeBSD 3.1.  In FreeBSD 10 support for sending
     shared memory descriptors had been introduced.  In FreeBSD 11 a non-
     blocking implementation had been introduced.

From: Youzhong Yang
Sent: Friday, November 03, 2017 4:28 PM
To: [hidden email]
Subject: RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Sorry, I meant:

if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened.

From: Youzhong Yang
Sent: Friday, November 03, 2017 4:27 PM
To: [hidden email]<mailto:[hidden email]>
Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Hi All,

I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:


  *   sendfile() first returns 35 (EAGAIN):

# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU     ID                    FUNCTION:NAME
23  64274                  sendfile:return
              libc.so.7`__sys_sendfile+0xa
              libsmbd-base-samba4.so`vfswrap_sendfile+0x127
              libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
              libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
              libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35


  *   then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:

# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16  63748                    writev:return
              libc.so.7`_writev+0xa
              libsys-rw-samba4.so`sys_writev+0x21
              libsys-rw-samba4.so`write_data_iov+0x88
              libsys-rw-samba4.so`write_data+0x39
              libsmbd-base-samba4.so`fake_sendfile+0x16e
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35


  *   and this is the stack trace of smbd:

# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped

Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
  * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
    frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
    frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
    frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
    frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
    frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
    frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
    frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
    frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
    frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
    frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
    frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
    frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
    frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
    frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
    frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
    frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
    frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
    frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
    frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
    frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
    frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
    frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
    frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
    frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
    frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
    frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
    frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
    frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
    frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
    frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
    frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
    frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
    frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
    frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
    frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
    frame #36: 0x0000000001027cf0 smbd`_start + 384

In my smb.conf, I have socket options set as follows:

socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5

By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?

Thanks,

--Youzhong

Reply | Threaded
Open this post in threaded view
|

RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
Hi Timur,

That’s too bad. Yes, I am using zfs on freebsd.

This one explains why:

https://wiki.freebsd.org/DevSummit/20170907/ZFS


“ZFS currently does not play nicely with sendfile(2). The idea with sendfile(2) is to share pages between the buffer cache and the mbuf, to prevent copying. The call results in a range of bytes from an open file descriptor being faulted into the buffer cache, and then those pages being shared all the way through to the network stack until the bytes are transmitted.

With ZFS, since it does not use the OS buffer cache, data is copied from the ZFS ARC to the buffer cache, where it is then shared with the mbuf etc. Solaris has a solution to solve this issue, and it might make sense to adapt our sendfile mechanism to work in a similar way. [Solaris PSARC: Copy Reduction Interface](https://www.mail-archive.com/opensolaris-arc@.../msg17203.html) This creates two new VFS operations, reqzcbuf which asks the filesystem to prepare a UIO for zero copy operations. In the case of ZFS, this involves allocating a new array to track the ARC headers of each buffer that is loaned out from the ZFS ARC.”

But it’s really bad that using sendfile() causes Samba to hang.

Thanks,

-Youzhong
From: Timur I. Bakeyev [mailto:[hidden email]]
Sent: Sunday, November 05, 2017 7:23 PM
To: Youzhong Yang <[hidden email]>
Cc: [hidden email]
Subject: Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Hi, Youzhong!

You didn't specify what file system do you use. Please, keep in mind that ZFS doesn't support 'sendfile' properly.

With regards,
Timur Bakeyev

On Fri, Nov 3, 2017 at 9:28 PM, Youzhong Yang via samba-technical <[hidden email]<mailto:[hidden email]>> wrote:
Sorry, I meant:

if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened.

From: Youzhong Yang
Sent: Friday, November 03, 2017 4:27 PM
To: [hidden email]<mailto:[hidden email]>
Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Hi All,

I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:


  *   sendfile() first returns 35 (EAGAIN):

# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU     ID                    FUNCTION:NAME
23  64274                  sendfile:return
              libc.so.7`__sys_sendfile+0xa
              libsmbd-base-samba4.so`vfswrap_sendfile+0x127
              libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
              libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
              libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35


  *   then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:

# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16  63748                    writev:return
              libc.so.7`_writev+0xa
              libsys-rw-samba4.so`sys_writev+0x21
              libsys-rw-samba4.so`write_data_iov+0x88
              libsys-rw-samba4.so`write_data+0x39
              libsmbd-base-samba4.so`fake_sendfile+0x16e
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35


  *   and this is the stack trace of smbd:

# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped

Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
  * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
    frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
    frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
    frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
    frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
    frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
    frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
    frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
    frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
    frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
    frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
    frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
    frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
    frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
    frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
    frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
    frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
    frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
    frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
    frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
    frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
    frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
    frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
    frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
    frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
    frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
    frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
    frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
    frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
    frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
    frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
    frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
    frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
    frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
    frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
    frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
    frame #36: 0x0000000001027cf0 smbd`_start + 384

In my smb.conf, I have socket options set as follows:

socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5

By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?

Thanks,

--Youzhong

Reply | Threaded
Open this post in threaded view
|

Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
Yes, I'm aware of the inefficiency of sendfile() on ZFS, but hanging of
Samba is something new. I'd suggest you to file a bug on bugzilla.samba.org
regarding that.

With best regards,
Timur Bakeyev.

On Mon, Nov 6, 2017 at 1:51 AM, Youzhong Yang <[hidden email]>
wrote:

> Hi Timur,
>
>
>
> That’s too bad. Yes, I am using zfs on freebsd.
>
>
>
> This one explains why:
>
>
>
> https://wiki.freebsd.org/DevSummit/20170907/ZFS
>
>
>
> “ZFS currently does not play nicely with sendfile(2). The idea with
> sendfile(2) is to share pages between the buffer cache and the mbuf, to
> prevent copying. The call results in a range of bytes from an open file
> descriptor being faulted into the buffer cache, and then those pages being
> shared all the way through to the network stack until the bytes are
> transmitted.
>
> With ZFS, since it does not use the OS buffer cache, data is copied from
> the ZFS ARC to the buffer cache, where it is then shared with the mbuf
> etc. Solaris has a solution to solve this issue, and it might make sense to
> adapt our sendfile mechanism to work in a similar way. [Solaris PSARC: Copy
> Reduction Interface](https://www.mail-archive.com/opensolaris-arc@
> mail.opensolaris.org/msg17203.html) This creates two new VFS operations,
> reqzcbuf which asks the filesystem to prepare a UIO for zero copy
> operations. In the case of ZFS, this involves allocating a new array to
> track the ARC headers of each buffer that is loaned out from the ZFS ARC.”
>
> But it’s really bad that using sendfile() causes Samba to hang.
>
> Thanks,
>
> -Youzhong
>
> *From:* Timur I. Bakeyev [mailto:[hidden email]]
> *Sent:* Sunday, November 05, 2017 7:23 PM
> *To:* Youzhong Yang <[hidden email]>
> *Cc:* [hidden email]
> *Subject:* Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes'
> in smb.conf
>
>
>
> Hi, Youzhong!
>
> You didn't specify what file system do you use. Please, keep in mind that
> ZFS doesn't support 'sendfile' properly.
>
>
>
> With regards,
>
> Timur Bakeyev
>
>
>
> On Fri, Nov 3, 2017 at 9:28 PM, Youzhong Yang via samba-technical <
> [hidden email]> wrote:
>
> Sorry, I meant:
>
> if 'use sendfile' is set to no, everything works well. But if it is set to
> yes, here is what happened.
>
> From: Youzhong Yang
> Sent: Friday, November 03, 2017 4:27 PM
> To: [hidden email]
> Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in
> smb.conf
>
>
> Hi All,
>
> I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if
> 'use sendfile' is set to yes, everything works well. But if it is set to
> no, here is what happened:
>
>
>   *   sendfile() first returns 35 (EAGAIN):
>
> # dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack();
> printf("errno = %d", errno);}'
> dtrace: description 'syscall::sendfile:return ' matched 1 probe
> CPU     ID                    FUNCTION:NAME
> 23  64274                  sendfile:return
>               libc.so.7`__sys_sendfile+0xa
>               libsmbd-base-samba4.so`vfswrap_sendfile+0x127
>               libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
>               libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
>               libtalloc.so.2.1.9`_tc_free_internal+0x152
>               libtalloc.so.2.1.9`_tc_free_children_internal+0xac
>               libtalloc.so.2.1.9`_tc_free_internal+0x331
>               libtalloc.so.2.1.9`_talloc_free_internal+0xb2
>               libtalloc.so.2.1.9`_talloc_free+0x114
>               libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
>               libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
>               libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
>               libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
>               libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
>               libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
>               libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
>               libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
>               libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
>               libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
>               libtevent.so.0.9.31`poll_event_loop_poll+0x75c
> errno = 35
>
>
>   *   then smbd keeps calling writev(), which always returns -1 with errno
> being EAGAIN:
>
> # dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno
> = %d", errno);}'
> 16  63748                    writev:return
>               libc.so.7`_writev+0xa
>               libsys-rw-samba4.so`sys_writev+0x21
>               libsys-rw-samba4.so`write_data_iov+0x88
>               libsys-rw-samba4.so`write_data+0x39
>               libsmbd-base-samba4.so`fake_sendfile+0x16e
>               libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
>               libtalloc.so.2.1.9`_tc_free_internal+0x152
>               libtalloc.so.2.1.9`_tc_free_children_internal+0xac
>               libtalloc.so.2.1.9`_tc_free_internal+0x331
>               libtalloc.so.2.1.9`_talloc_free_internal+0xb2
>               libtalloc.so.2.1.9`_talloc_free+0x114
>               libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
>               libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
>               libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
>               libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
>               libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
>               libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
>               libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
>               libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
>               libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
> errno = 35
>
>
>   *   and this is the stack trace of smbd:
>
> # echo bt | lldb -p 1810
> (lldb) process attach --pid 1810
> Process 1810 stopped
>
> Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
> Architecture set to: x86_64--freebsd11.1.
> (lldb) bt
> * thread #1
>   * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
>     frame #1: libthr.so.3`__thr_writev(fd=<unavailable>,
> iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
>     frame #2: libsys-rw-samba4.so`sys_writev(fd=40,
> iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
>     frame #3: libsys-rw-samba4.so`write_data_iov(fd=40,
> orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
>     frame #4: libsys-rw-samba4.so`write_data(fd=40,
> buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
>     frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60,
> fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
>     frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660)
> at smb2_read.c:283
>     frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
>     frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080,
> ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at
> talloc.c:1593
>     frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
>     frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
>     frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
>     frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60)
> at smb2_server.c:3697
>     frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0)
> at smb2_server.c:3011
>     frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0,
> status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680,
> location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
>     frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000)
> at smb2_read.c:164
>     frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0,
> location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
>     frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0,
> subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
>     frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0)
> at smb2_read.c:109
>     frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0)
> at smb2_server.c:2641
>     frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60,
> fde_flags=1) at smb2_server.c:3950
>     frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0,
> fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at
> smb2_server.c:3988
>     frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0,
> tvalp=0x00007fffffffe068) at tevent_poll.c:605
>     frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent_poll.c:662
>     frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent.c:721
>     frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent_poll.c:678
>     frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent.c:863
>     frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0,
> msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
>     frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0,
> fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at
> server.c:1024
>     frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0,
> tvalp=0x00007fffffffe548) at tevent_poll.c:605
>     frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent_poll.c:662
>     frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent.c:721
>     frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent_poll.c:678
>     frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent.c:863
>     frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0,
> parent=0x0000000812c52680) at server.c:1391
>     frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
>     frame #36: 0x0000000001027cf0 smbd`_start + 384
>
> In my smb.conf, I have socket options set as follows:
>
> socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576
> SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5
>
> By the way, I am running Samba 4.6.8. Has anyone experienced the same
> issue? Is it a FreeBSD kernel bug?
>
> Thanks,
>
> --Youzhong
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On 2017-11-03 at 20:26 +0000 Youzhong Yang via samba-technical sent off:
> I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:

is https://github.com/samba-team/samba/pull/76 helping you maybe?

Björn

Reply | Threaded
Open this post in threaded view
|

RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
I tried your patch, yes, it works, no more spinning.

Thanks! Is there any chance this patch will get integrated soon?

-----Original Message-----
From: Björn JACKE [mailto:[hidden email]]
Sent: Friday, November 10, 2017 8:36 PM
To: Youzhong Yang <[hidden email]>
Cc: [hidden email]
Subject: Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

On 2017-11-03 at 20:26 +0000 Youzhong Yang via samba-technical sent off:
> I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:

is https://github.com/samba-team/samba/pull/76 helping you maybe?

Björn

Reply | Threaded
Open this post in threaded view
|

Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

Samba - samba-technical mailing list
Hi,
On 2017-11-11 at 16:18 +0000 Youzhong Yang via samba-technical sent off:
> I tried your patch, yes, it works, no more spinning.

it's not my patch, I just remembered having see that pull request some time ago.

> Thanks! Is there any chance this patch will get integrated soon?

you should open a bug report in bugzilla.samba.org describing your problem and
point to that patch set and try to get the attention of a developer to review
and apply the patch set.

Björn