Zero-copy patch

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Zero-copy patch

Jacky Lam-2
Dear all,

    Does any one have the kernel patch for zero-copy from skb to IO buffer/cache? The performance of Samba using splice is slowing down because of busy to copy the buffer from socket.

    Thanks.

Jacky

________________________________
IMPORTANT CONFIDENTIALITY NOTICE
This message and any attached documents contain information from ViXS Systems, Inc. and are confidential and privileged and further subject to any confidentiality agreement between the parties. The information is intended to be viewed only by the individual(s) or entity(ies) to whom the message is addressed. If you are not the intended recipient, be aware that reading, disclosing, copying, distributing or using the contents of this transmission is prohibited. Please notify us immediately if you have received this transmission in error, and delete this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Volker Lendecke
On Mon, Jan 31, 2011 at 08:42:35PM -0500, Jacky Lam wrote:
> Does any one have the kernel patch for zero-copy from skb
> to IO buffer/cache? The performance of Samba using splice
> is slowing down because of busy to copy the buffer from
> socket.

Wait -- If I used 2 splice calls, one from TCP socket to a
pipe and the second from the pipe to a on-disk file it still
does a copy internally?

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Volker Lendecke
On Tue, Feb 01, 2011 at 12:17:35PM +0100, Volker Lendecke wrote:
> > Does any one have the kernel patch for zero-copy from skb
> > to IO buffer/cache? The performance of Samba using splice
> > is slowing down because of busy to copy the buffer from
> > socket.
>
> Wait -- If I used 2 splice calls, one from TCP socket to a
> pipe and the second from the pipe to a on-disk file it still
> does a copy internally?

Hmmm. Found the following comment in relatively recent Linux
kernel source:

 *      - Destination page already exists in the address space and there
 *        are users of it. For that case we have no other option that
 *        copying the data. Tough luck.
 *      - Destination page already exists in the address space, but there
 *        are no users of it. Make sure it's uptodate, then drop it. Fall
 *        through to last case.
 *      - Destination page does not exist, we can add the pipe page to
 *        the page cache and avoid the copy.

So it seems that for normal samba file server use it should
avoid the copy.

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2
I am curious about that as well. But I have double check most of the time spending while writing to samba server is that two splice call. And the kernel profile is showing that __copy_user() is using 25% CPU time during that period of time.

I don't know if it is platform dependent problem.....have you try to turn on splice and do a kernel profiling?
________________________________________
From: Volker Lendecke [[hidden email]]
Sent: Tuesday, February 01, 2011 6:30 AM
To: Jacky Lam
Cc: [hidden email]
Subject: Re: Zero-copy patch

On Tue, Feb 01, 2011 at 12:17:35PM +0100, Volker Lendecke wrote:
> > Does any one have the kernel patch for zero-copy from skb
> > to IO buffer/cache? The performance of Samba using splice
> > is slowing down because of busy to copy the buffer from
> > socket.
>
> Wait -- If I used 2 splice calls, one from TCP socket to a
> pipe and the second from the pipe to a on-disk file it still
> does a copy internally?

Hmmm. Found the following comment in relatively recent Linux
kernel source:

 *      - Destination page already exists in the address space and there
 *        are users of it. For that case we have no other option that
 *        copying the data. Tough luck.
 *      - Destination page already exists in the address space, but there
 *        are no users of it. Make sure it's uptodate, then drop it. Fall
 *        through to last case.
 *      - Destination page does not exist, we can add the pipe page to
 *        the page cache and avoid the copy.

So it seems that for normal samba file server use it should
avoid the copy.

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen

IMPORTANT CONFIDENTIALITY NOTICE
This message and any attached documents contain information from ViXS Systems, Inc. and are confidential and privileged and further subject to any confidentiality agreement between the parties. The information is intended to be viewed only by the individual(s) or entity(ies) to whom the message is addressed. If you are not the intended recipient, be aware that reading, disclosing, copying, distributing or using the contents of this transmission is prohibited. Please notify us immediately if you have received this transmission in error, and delete this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Volker Lendecke
On Tue, Feb 01, 2011 at 09:03:06AM -0500, Jacky Lam wrote:
> I am curious about that as well. But I have double check
> most of the time spending while writing to samba server is
> that two splice call. And the kernel profile is showing
> that __copy_user() is using 25% CPU time during that
> period of time.
>
> I don't know if it is platform dependent problem.....have
> you try to turn on splice and do a kernel profiling?

No, I haven't. This is purely from code inspection. I
haven't worked on the splice/recvfile implementation for
quite a while.

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Jeremy Allison
In reply to this post by Volker Lendecke
On Tue, Feb 01, 2011 at 12:30:43PM +0100, Volker Lendecke wrote:

> On Tue, Feb 01, 2011 at 12:17:35PM +0100, Volker Lendecke wrote:
> > > Does any one have the kernel patch for zero-copy from skb
> > > to IO buffer/cache? The performance of Samba using splice
> > > is slowing down because of busy to copy the buffer from
> > > socket.
> >
> > Wait -- If I used 2 splice calls, one from TCP socket to a
> > pipe and the second from the pipe to a on-disk file it still
> > does a copy internally?
>
> Hmmm. Found the following comment in relatively recent Linux
> kernel source:
>
>  *      - Destination page already exists in the address space and there
>  *        are users of it. For that case we have no other option that
>  *        copying the data. Tough luck.
>  *      - Destination page already exists in the address space, but there
>  *        are no users of it. Make sure it's uptodate, then drop it. Fall
>  *        through to last case.
>  *      - Destination page does not exist, we can add the pipe page to
>  *        the page cache and avoid the copy.
>
> So it seems that for normal samba file server use it should
> avoid the copy.

Yes, it seems so, but when I tested it in practice it made
things worse. I don't know why :-(.

If Jacky can figure that out it would be a great help to
everyone.

Jeremy.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Jeremy Allison
In reply to this post by Jacky Lam-2
On Tue, Feb 01, 2011 at 09:03:06AM -0500, Jacky Lam wrote:
> I am curious about that as well. But I have double check most of the time spending while writing to samba server is that two splice call. And the kernel profile is showing that __copy_user() is using 25% CPU time during that period of time.
>
> I don't know if it is platform dependent problem.....have you try to turn on splice and do a kernel profiling?

Please do and let us know why it isn't working right.
I can help bug the kernel devs to look at it.

FYI for everyone else, I've sent Jacky the kernel
patch to test off-list.

Jeremy.
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2
Thanks for the patch. It is Lunar new year for China now. I will back to work next week and let you know the result.

Jacky

-----Original Message-----
From: Jeremy Allison [mailto:[hidden email]]
Sent: Wednesday, February 02, 2011 1:58 AM
To: Jacky Lam
Cc: [hidden email]; [hidden email]
Subject: Re: Zero-copy patch

On Tue, Feb 01, 2011 at 09:03:06AM -0500, Jacky Lam wrote:
> I am curious about that as well. But I have double check most of the time spending while writing to samba server is that two splice call. And the kernel profile is showing that __copy_user() is using 25% CPU time during that period of time.
>
> I don't know if it is platform dependent problem.....have you try to turn on splice and do a kernel profiling?

Please do and let us know why it isn't working right.
I can help bug the kernel devs to look at it.

FYI for everyone else, I've sent Jacky the kernel
patch to test off-list.

Jeremy.

IMPORTANT CONFIDENTIALITY NOTICE
This message and any attached documents contain information from ViXS Systems, Inc. and are confidential and privileged and further subject to any confidentiality agreement between the parties. The information is intended to be viewed only by the individual(s) or entity(ies) to whom the message is addressed. If you are not the intended recipient, be aware that reading, disclosing, copying, distributing or using the contents of this transmission is prohibited. Please notify us immediately if you have received this transmission in error, and delete this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2
In reply to this post by Jeremy Allison
Jeremy,

        After tracing the kernel code, I come up with a disappointing conclusion.

        The heavy memory copy are coming from two places in kernel:

        1. net/core/skbuff.c: linear_to_page()
        2. fs/splice.c: pipe_to_file()

        The first one is actually introduced in 2.6.28.6/2.6.29 to fix a data corruption bug when splice from socket to socket. I quote the kernel change log at the end of mail. This one seems don't bother Samba, so I roll back the fix and get a 10% improvement.

        The second one do memcopy when
                *       Destination page already exists in the address space and there
                *       are users of it. For that case we have no other option that
                *       copying the data. Tough luck.
        But I am curious who the users are (mistake?). So, I just comment out the memory copy and expect data corruption. I want to check what can I achieve if all these things are probably fixed.

        Finally, I get the throughput more or less the same as using read()/write(). It doesn't worthwhile to bother all those trouble at all. Maybe the data flow of current implementation of socket splice are inefficient enough indeed.

        If there is not any other idea, I think I would rather spend time on how to make the patches you send me to work.

        Thanks.

BR,
Jacky

================================================================================
commit 813fa24255a5de93ef3fc4c2efff3ee31a2545b6
Author: Jarek Poplawski <[hidden email]>
Date:   Mon Jan 19 17:03:56 2009 -0800

    net: Fix data corruption when splicing from sockets.

    [ Upstream commit 8b9d3728977760f6bd1317c4420890f73695354e ]

    The trick in socket splicing where we try to convert the skb->data
    into a page based reference using virt_to_page() does not work so
    well.

    The idea is to pass the virt_to_page() reference via the pipe
    buffer, and refcount the buffer using a SKB reference.

    But if we are splicing from a socket to a socket (via sendpage)
    this doesn't work.

    The from side processing will grab the page (and SKB) references.
    The sendpage() calls will grab page references only, return, and
    then the from side processing completes and drops the SKB ref.

    The page based reference to skb->data is not enough to keep the
    kmalloc() buffer backing it from being reused.  Yet, that is
    all that the socket send side has at this point.

    This leads to data corruption if the skb->data buffer is reused
    by SLAB before the send side socket actually gets the TX packet
    out to the device.

    The fix employed here is to simply allocate a page and copy the
    skb->data bytes into that page.

    This will hurt performance, but there is no clear way to fix this
    properly without a copy at the present time, and it is important
    to get rid of the data corruption.

    With fixes from Herbert Xu.

    Tested-by: Willy Tarreau <[hidden email]>
    Foreseen-by: Changli Gao <[hidden email]>
    Diagnosed-by: Willy Tarreau <[hidden email]>
    Reported-by: Willy Tarreau <[hidden email]>
    Fixed-by: Jens Axboe <[hidden email]>
    Signed-off-by: Jarek Poplawski <[hidden email]>
    Signed-off-by: David S. Miller <[hidden email]>
    Signed-off-by: Greg Kroah-Hartman <[hidden email]>
========================================================================================



-----Original Message-----
From: Jeremy Allison [mailto:[hidden email]]
Sent: Wednesday, February 02, 2011 1:58 AM
To: Jacky Lam
Cc: [hidden email]; [hidden email]
Subject: Re: Zero-copy patch

On Tue, Feb 01, 2011 at 09:03:06AM -0500, Jacky Lam wrote:
> I am curious about that as well. But I have double check most of the time spending while writing to samba server is that two splice call. And the kernel profile is showing that __copy_user() is using 25% CPU time during that period of time.
>
> I don't know if it is platform dependent problem.....have you try to turn on splice and do a kernel profiling?

Please do and let us know why it isn't working right.
I can help bug the kernel devs to look at it.

FYI for everyone else, I've sent Jacky the kernel
patch to test off-list.

Jeremy.

IMPORTANT CONFIDENTIALITY NOTICE
This message and any attached documents contain information from ViXS Systems, Inc. and are confidential and privileged and further subject to any confidentiality agreement between the parties. The information is intended to be viewed only by the individual(s) or entity(ies) to whom the message is addressed. If you are not the intended recipient, be aware that reading, disclosing, copying, distributing or using the contents of this transmission is prohibited. Please notify us immediately if you have received this transmission in error, and delete this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Jeremy Allison
On Tue, Feb 08, 2011 at 01:53:14AM -0500, Jacky Lam wrote:

> Jeremy,
>
>         After tracing the kernel code, I come up with a disappointing conclusion.
>
>         The heavy memory copy are coming from two places in kernel:
>
>         1. net/core/skbuff.c: linear_to_page()
>         2. fs/splice.c: pipe_to_file()
>
>         The first one is actually introduced in 2.6.28.6/2.6.29 to fix a data corruption bug when splice from socket to socket. I quote the kernel change log at the end of mail. This one seems don't bother Samba, so I roll back the fix and get a 10% improvement.
>
>         The second one do memcopy when
>                 *       Destination page already exists in the address space and there
>                 *       are users of it. For that case we have no other option that
>                 *       copying the data. Tough luck.
>         But I am curious who the users are (mistake?). So, I just comment out the memory copy and expect data corruption. I want to check what can I achieve if all these things are probably fixed.
>
>         Finally, I get the throughput more or less the same as using read()/write(). It doesn't worthwhile to bother all those trouble at all. Maybe the data flow of current implementation of socket splice are inefficient enough indeed.
>
>         If there is not any other idea, I think I would rather spend time on how to make the patches you send me to work.

Another OEM (who shall remain nameless) mentioned
that changing the kernel scheduler to the deadline
scheduler helps performance (you may already have
done this of course).

Let me know if you haven't and if it helps.

Jeremy.
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2


> -----Original Message-----
> From: Jeremy Allison [mailto:[hidden email]]
> Sent: Thursday, February 10, 2011 9:43 AM
> To: Jacky Lam
> Cc: Jeremy Allison; [hidden email]; samba-
> [hidden email]
> Subject: Re: Zero-copy patch
>
> On Tue, Feb 08, 2011 at 01:53:14AM -0500, Jacky Lam wrote:
> > Jeremy,
> >
> >         After tracing the kernel code, I come up with a disappointing
> conclusion.
> >
> >         The heavy memory copy are coming from two places in kernel:
> >
> >         1. net/core/skbuff.c: linear_to_page()
> >         2. fs/splice.c: pipe_to_file()
> >
> >         The first one is actually introduced in 2.6.28.6/2.6.29 to
> fix a data corruption bug when splice from socket to socket. I quote
> the kernel change log at the end of mail. This one seems don't bother
> Samba, so I roll back the fix and get a 10% improvement.
> >
> >         The second one do memcopy when
> >                 *       Destination page already exists in the
> address space and there
> >                 *       are users of it. For that case we have no
> other option that
> >                 *       copying the data. Tough luck.
> >         But I am curious who the users are (mistake?). So, I just
> comment out the memory copy and expect data corruption. I want to check
> what can I achieve if all these things are probably fixed.
> >
> >         Finally, I get the throughput more or less the same as using
> read()/write(). It doesn't worthwhile to bother all those trouble at
> all. Maybe the data flow of current implementation of socket splice are
> inefficient enough indeed.
> >
> >         If there is not any other idea, I think I would rather spend
> time on how to make the patches you send me to work.
>
> Another OEM (who shall remain nameless) mentioned
> that changing the kernel scheduler to the deadline
> scheduler helps performance (you may already have
> done this of course).
>
> Let me know if you haven't and if it helps.
>
> Jeremy.

I have tried before and just now again. No noticeable changes.

BTW, I applied the patch to kernel of my desktop. The patch is applied cleanly but obviously missing something. It add splice_direct_from_socket() for splice(), but have nothing change for sendfile(). Anyway, I do a quick fix for that and finally got sendfile() do what we want. The result is the throughput is about 5% faster than read/write implementation. But it is far lower than my expectation. Is this reported result of the patch?

Jacky


IMPORTANT CONFIDENTIALITY NOTICE
This message and any attached documents contain information from ViXS Systems, Inc. and are confidential and privileged and further subject to any confidentiality agreement between the parties. The information is intended to be viewed only by the individual(s) or entity(ies) to whom the message is addressed. If you are not the intended recipient, be aware that reading, disclosing, copying, distributing or using the contents of this transmission is prohibited. Please notify us immediately if you have received this transmission in error, and delete this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2
Jeremy,

        I try again the patch on my MIPS platform and happily find that the throughput improves by 20%. However, I get some data corruption. I find there is some chance the order of two 64K data packets is reversed. Who is the author of this patch? Need to submit the bug to him?

        Thanks.

BR,
Jacky

> -----Original Message-----
> From: [hidden email] [mailto:samba-technical-
> [hidden email]] On Behalf Of Jacky Lam
> Sent: Thursday, February 10, 2011 2:29 PM
> To: Jeremy Allison
> Cc: [hidden email]; [hidden email]
> Subject: RE: Zero-copy patch
>
>
>
> > -----Original Message-----
> > From: Jeremy Allison [mailto:[hidden email]]
> > Sent: Thursday, February 10, 2011 9:43 AM
> > To: Jacky Lam
> > Cc: Jeremy Allison; [hidden email]; samba-
> > [hidden email]
> > Subject: Re: Zero-copy patch
> >
> > On Tue, Feb 08, 2011 at 01:53:14AM -0500, Jacky Lam wrote:
> > > Jeremy,
> > >
> > >         After tracing the kernel code, I come up with a
> disappointing
> > conclusion.
> > >
> > >         The heavy memory copy are coming from two places in kernel:
> > >
> > >         1. net/core/skbuff.c: linear_to_page()
> > >         2. fs/splice.c: pipe_to_file()
> > >
> > >         The first one is actually introduced in 2.6.28.6/2.6.29 to
> > fix a data corruption bug when splice from socket to socket. I quote
> > the kernel change log at the end of mail. This one seems don't bother
> > Samba, so I roll back the fix and get a 10% improvement.
> > >
> > >         The second one do memcopy when
> > >                 *       Destination page already exists in the
> > address space and there
> > >                 *       are users of it. For that case we have no
> > other option that
> > >                 *       copying the data. Tough luck.
> > >         But I am curious who the users are (mistake?). So, I just
> > comment out the memory copy and expect data corruption. I want to
> check
> > what can I achieve if all these things are probably fixed.
> > >
> > >         Finally, I get the throughput more or less the same as
> using
> > read()/write(). It doesn't worthwhile to bother all those trouble at
> > all. Maybe the data flow of current implementation of socket splice
> are
> > inefficient enough indeed.
> > >
> > >         If there is not any other idea, I think I would rather
> spend
> > time on how to make the patches you send me to work.
> >
> > Another OEM (who shall remain nameless) mentioned
> > that changing the kernel scheduler to the deadline
> > scheduler helps performance (you may already have
> > done this of course).
> >
> > Let me know if you haven't and if it helps.
> >
> > Jeremy.
>
> I have tried before and just now again. No noticeable changes.
>
> BTW, I applied the patch to kernel of my desktop. The patch is applied
> cleanly but obviously missing something. It add
> splice_direct_from_socket() for splice(), but have nothing change for
> sendfile(). Anyway, I do a quick fix for that and finally got sendfile()
> do what we want. The result is the throughput is about 5% faster than
> read/write implementation. But it is far lower than my expectation. Is
> this reported result of the patch?
>
> Jacky
>
>
> IMPORTANT CONFIDENTIALITY NOTICE
> This message and any attached documents contain information from ViXS
> Systems, Inc. and are confidential and privileged and further subject
> to any confidentiality agreement between the parties. The information
> is intended to be viewed only by the individual(s) or entity(ies) to
> whom the message is addressed. If you are not the intended recipient,
> be aware that reading, disclosing, copying, distributing or using the
> contents of this transmission is prohibited. Please notify us
> immediately if you have received this transmission in error, and delete
> this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Jeremy Allison
On Thu, Feb 10, 2011 at 10:39:44PM -0500, Jacky Lam wrote:
> Jeremy,
>
> I try again the patch on my MIPS platform and happily find that the throughput improves by 20%. However, I get some data corruption. I find there is some chance the order of two 64K data packets is reversed. Who is the author of this patch? Need to submit the bug to him?

Wow - that's interesting. Why the change ?

I'm currently coordinating the patch so send the
fix to me. I've sent it to Christoph Hellwig, and
I'll make him aware of your fix.

Thanks !

Jeremy.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Jeremy Allison
On Thu, Feb 10, 2011 at 07:42:45PM -0800, Jeremy Allison wrote:
> On Thu, Feb 10, 2011 at 10:39:44PM -0500, Jacky Lam wrote:
> > Jeremy,
> >
> > I try again the patch on my MIPS platform and happily find that the throughput improves by 20%. However, I get some data corruption. I find there is some chance the order of two 64K data packets is reversed. Who is the author of this patch? Need to submit the bug to him?
>
> Wow - that's interesting. Why the change ?

And in this I mean why the change from the 5% you
originally reported to the 20% ? What changed
between the two tests ?

Jeremy.
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2
No change is source level. I just change the test platform from x86 to MIPS. Maybe my MIPS's throughput level is CPU power and this patch helps that.

I am still investigating the problem and will send you the patch if I fix it.

Jacky

> -----Original Message-----
> From: Jeremy Allison [mailto:[hidden email]]
> Sent: Friday, February 11, 2011 11:45 AM
> To: Jeremy Allison
> Cc: Jacky Lam; [hidden email]; samba-
> [hidden email]
> Subject: Re: Zero-copy patch
>
> On Thu, Feb 10, 2011 at 07:42:45PM -0800, Jeremy Allison wrote:
> > On Thu, Feb 10, 2011 at 10:39:44PM -0500, Jacky Lam wrote:
> > > Jeremy,
> > >
> > >   I try again the patch on my MIPS platform and happily find that
> the throughput improves by 20%. However, I get some data corruption. I
> find there is some chance the order of two 64K data packets is reversed.
> Who is the author of this patch? Need to submit the bug to him?
> >
> > Wow - that's interesting. Why the change ?
>
> And in this I mean why the change from the 5% you
> originally reported to the 20% ? What changed
> between the two tests ?
>
> Jeremy.

IMPORTANT CONFIDENTIALITY NOTICE
This message and any attached documents contain information from ViXS Systems, Inc. and are confidential and privileged and further subject to any confidentiality agreement between the parties. The information is intended to be viewed only by the individual(s) or entity(ies) to whom the message is addressed. If you are not the intended recipient, be aware that reading, disclosing, copying, distributing or using the contents of this transmission is prohibited. Please notify us immediately if you have received this transmission in error, and delete this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Jeremy Allison
On Thu, Feb 10, 2011 at 10:47:42PM -0500, Jacky Lam wrote:
> No change is source level. I just change the test platform from x86 to MIPS. Maybe my MIPS's throughput level is CPU power and this patch helps that.

Ah ok - I misunderstood. That is indeed a great help !
Did you also look into the scheduler change I mentioned ?

> I am still investigating the problem and will send you the patch if I fix it.

Thanks - much appreciated !

Jeremy.
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2


> -----Original Message-----
> From: Jeremy Allison [mailto:[hidden email]]
> Sent: Friday, February 11, 2011 11:49 AM
> To: Jacky Lam
> Cc: Jeremy Allison; [hidden email]; samba-
> [hidden email]
> Subject: Re: Zero-copy patch
>
> On Thu, Feb 10, 2011 at 10:47:42PM -0500, Jacky Lam wrote:
> > No change is source level. I just change the test platform from x86
> to MIPS. Maybe my MIPS's throughput level is CPU power and this patch
> helps that.
>
> Ah ok - I misunderstood. That is indeed a great help !
> Did you also look into the scheduler change I mentioned ?

Yes, I change the scheduler but no change in throughput or CPU loading.....Any extra tuning of scheduler parameter needed?

>
> > I am still investigating the problem and will send you the patch if I
> fix it.
>
> Thanks - much appreciated !
>
> Jeremy.

IMPORTANT CONFIDENTIALITY NOTICE
This message and any attached documents contain information from ViXS Systems, Inc. and are confidential and privileged and further subject to any confidentiality agreement between the parties. The information is intended to be viewed only by the individual(s) or entity(ies) to whom the message is addressed. If you are not the intended recipient, be aware that reading, disclosing, copying, distributing or using the contents of this transmission is prohibited. Please notify us immediately if you have received this transmission in error, and delete this message along with any attached files.
Reply | Threaded
Open this post in threaded view
|

RE: Zero-copy patch

Jacky Lam-2
Jeremy,

        I make a mistake, the data corruption is due to my change to the patch which let sys_sendfile() to call do_splice_from_socket() when appropriate. I think you should have this already, otherwise, the patch won't have any effect to sendfile(). Anyway, I attached in case you are interested.

        Also, for the samba patch to lib/recvfile.c, I suggest to make the following change:

-#if defined(HAVE_SENDFILE) && defined(LINUX_SENDFILE_API)
+#if (defined(HAVE_SENDFILE64) || defined(HAVE_SENDFILE)) && defined(LINUX_SENDFILE_API)

        Thanks.

Jacky

> -----Original Message-----
> From: [hidden email] [mailto:samba-technical-
> [hidden email]] On Behalf Of Jacky Lam
> Sent: Friday, February 11, 2011 11:51 AM
> To: Jeremy Allison
> Cc: [hidden email]; [hidden email]
> Subject: RE: Zero-copy patch
>
>
>
> > -----Original Message-----
> > From: Jeremy Allison [mailto:[hidden email]]
> > Sent: Friday, February 11, 2011 11:49 AM
> > To: Jacky Lam
> > Cc: Jeremy Allison; [hidden email]; samba-
> > [hidden email]
> > Subject: Re: Zero-copy patch
> >
> > On Thu, Feb 10, 2011 at 10:47:42PM -0500, Jacky Lam wrote:
> > > No change is source level. I just change the test platform from x86
> > to MIPS. Maybe my MIPS's throughput level is CPU power and this patch
> > helps that.
> >
> > Ah ok - I misunderstood. That is indeed a great help !
> > Did you also look into the scheduler change I mentioned ?
>
> Yes, I change the scheduler but no change in throughput or CPU
> loading.....Any extra tuning of scheduler parameter needed?
>
> >
> > > I am still investigating the problem and will send you the patch if
> I
> > fix it.
> >
> > Thanks - much appreciated !
> >
> > Jeremy.
>
> IMPORTANT CONFIDENTIALITY NOTICE
> This message and any attached documents contain information from ViXS
> Systems, Inc. and are confidential and privileged and further subject
> to any confidentiality agreement between the parties. The information
> is intended to be viewed only by the individual(s) or entity(ies) to
> whom the message is addressed. If you are not the intended recipient,
> be aware that reading, disclosing, copying, distributing or using the
> contents of this transmission is prohibited. Please notify us
> immediately if you have received this transmission in error, and delete
> this message along with any attached files.

fix.patch (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

wangdbang
In reply to this post by Jeremy Allison
Hi, Jacky/Jeremy

     Which kernel version based for this patch?  or would you like to send me a full code for the function do_splice_from_socket?

thanks,
Daobang Wang.
Reply | Threaded
Open this post in threaded view
|

Re: Zero-copy patch

Jackki
In reply to this post by Jeremy Allison
Hi, Jacky/Jeremy

Could you please release the code of do_splice_from_socket() for kernel patch?
Or is it available somewhere else?
I've tried to find the similar patch to eliminate the high cpu usage of __copy_user() while using splice() syscall.
But only found an old post which is based on linux kernel 2.6.13.
http://lwn.net/Articles/152424/

Thanks,

Jackki