Quantcast

libsmb2, meet samba

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

libsmb2, meet samba

ronnie sahlberg
samba, please meet libsmb2
libsmb2 please meet samba


As you know already I am am for various reasons, to benefit linux, working on a
smb2 client library.

https://github.com/sahlberg/libsmb2


Why, and what requirements?
----------------------------------------
I want there to be a highly portable and free smb2 client that any/all
applications can use.

It should be small and nimble. Currently clocks in at ~50kb compiled
(though that excludes MIT kerberos).

$ ls -l lib/.libs/libsmb2.a -rw-rw-r-- 1 sahlberg sahlberg 54188 Jan
24 17:40 lib/.libs/libsmb2.a

It should need very few external dependencies. Right now I target
basic posix libc plus MIT. Nothing else.

It should be so portable that it compiles native on both Win32/64 as
well as Amiga Kickstart 1.3.

It should provide an API similar to libnfs so that porting between nfs
and smb wrappers should be as painless as possible.

SMB2+ only. No smb1 support. RDMA support will be nice to add in the future.

Zero-copy for [p]read/[p]write.


Also, I have been a bit bored recently and what better reason for
building a lightweight SMB2 client? Boredom is the fuel for
innovation.


Licence
----------
For licence, gnu licences lgplv2.1 only or lgpl2.1-plus are
acceptable. QMU licence compatibility is not negotiable. (Make me
angry and I will consider 2-clause BSD, or worse :-) )
Nah lgpl2.1 is fine :-)

Kodi support is kind of not negotiable as well, but they are
compatible with a lot mode different licences so that should not be an
issue.



Current status
-------------------
The codebase is in hue flux and will need a major refactoring and also
audits over the next few weeks until the implementation as well as the
APIs will settle down.

Mimicing the libnfs APIs it provides fair coverage of the common posix
file api already:
smb2_close
smb2_close_async
smb2_closedir
smb2_cmd_close_async
smb2_cmd_create_async
smb2_cmd_echo_async
smb2_cmd_logoff_async
smb2_cmd_negotiate_async
smb2_cmd_query_directory_async
smb2_cmd_query_info_async
smb2_cmd_session_setup_async
smb2_cmd_tree_connect_async
smb2_connect_async
smb2_connect_share
smb2_connect_share_async
smb2_destroy_context
smb2_destroy_url
smb2_fstat
smb2_fstat_async
smb2_get_client_guid
smb2_get_error
smb2_get_fd
smb2_init_context
smb2_mkdir
smb2_mkdir_async
smb2_open
smb2_open_async
smb2_opendir
smb2_opendir_async
smb2_parse_url
smb2_pread
smb2_pread_async
smb2_pwrite
smb2_pwrite_async
smb2_read
smb2_read_async
smb2_readdir
smb2_rewinddir
smb2_rmdir
smb2_rmdir_async
smb2_seek
smb2_seekdir
smb2_service
smb2_set_security_mode
smb2_stat
smb2_stat_async
smb2_telldir
smb2_unlink
smb2_unlink_async
smb2_which_events
smb2_write
smb2_write_async

Not complete yet but sufficient for building a filemanager or something similar.

It uses MIT + simo's GSS-NLTMSSP for authentication. It does not yet
do actual sign or seal
but I think that is low priority for the initial version. Patches welcome.

It leaks memory from the gss contexts that are used during session
setup. Again patches welcome as I am completely ignorant about MIT and
its GSS layer, and wish to remain so as I am too old and lazy to learn
new stuff.


These funcitons work pretty well and even run valgrind clean (aside
from the two leaks form the mis-use of the MIT library).
Woohoo for that.


Qemu and Kodi and all the other users of libnfs
---------------------------------------------------------------
It supports all the posix fs functions that any of those apps use
today for libnfs so it should be fairly easy for those folks to take
theur libnfs wrappers, do some sed-magic and end up with an almost
functional smb2 wrapper.

Easy porting from libnfs to libsmb2 is another requirement that I just made up.


Examples
-------------
There are a bunch of examples in the examples subdirectory to show the
async as well as the sync api. Please have look.


Fire extinguisher
----------------------
This is not a replacement for libsmbclient. This is aimed as a minimal
library that is
easy to port and use. With limited functionality but an alternative to
those apps that just need the most basic smb2 support and where
libsmb2 client is overkill.

Well, on amiga Kickstart 1.3 it might be a replacement as I doubt
anyone wants to port non-extremely-ancient versions of libsmbclient to
the amiga.



Interested?
---------------
It is functional right now. More posix function emulation is needed.
It probably need a re-write and re-factor as well as the internal APIs
settle down.

Want to help out? Contact me and check out the github.
Contacting me first before starting to prepare any pull requests. I
expect that the code base will
be in major flux for the next few weeks until things stabilize.

There is a lot of work that still needs doing.


Where to host?
--------------------
I kind of like github/sahlberg as your on-stop-shop for everything
related to network storage client software.
However, hosting this on samba.org at some stage might be a better choice.

You open for that idea?





regards, your friend in the file/block client space
ronnie sahlberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

David Disseldorp-2
Hi Ronnie,

On Tue, 24 Jan 2017 19:31:33 -0800, ronnie sahlberg wrote:

...

> It should be small and nimble. Currently clocks in at ~50kb compiled
> (though that excludes MIT kerberos).
>
> $ ls -l lib/.libs/libsmb2.a -rw-rw-r-- 1 sahlberg sahlberg 54188 Jan
> 24 17:40 lib/.libs/libsmb2.a
>
> It should need very few external dependencies. Right now I target
> basic posix libc plus MIT. Nothing else.
>
> It should be so portable that it compiles native on both Win32/64 as
> well as Amiga Kickstart 1.3.
>
> It should provide an API similar to libnfs so that porting between nfs
> and smb wrappers should be as painless as possible.
>
> SMB2+ only. No smb1 support. RDMA support will be nice to add in the future.
>
> Zero-copy for [p]read/[p]write.

Always good to see a new FOSS client in the wild - Particularly with
async support and smb1 abandonment :-)

> Licence
> ----------
> For licence, gnu licences lgplv2.1 only or lgpl2.1-plus are
> acceptable. QMU licence compatibility is not negotiable. (Make me
> angry and I will consider 2-clause BSD, or worse :-) )
> Nah lgpl2.1 is fine :-)
>
> Kodi support is kind of not negotiable as well, but they are
> compatible with a lot mode different licences so that should not be an
> issue.

So this means that the libsmb2 client code could also be used by the
cifs.ko kernel client? I think at the very least it'd be good to have
each of these projects using the Samba idl for network<->req/rsp struct
marshalling.
talloc/tevent usage would also be a nice to have IMO, so you don't end
up rewriting a generic event library, or hierarchical allocator
(libiscsi seems to have gone a little that way ;-).

> Where to host?
> --------------------
> I kind of like github/sahlberg as your on-stop-shop for everything
> related to network storage client software.
> However, hosting this on samba.org at some stage might be a better choice.
>
> You open for that idea?

I can't speak for others on the team, but I'd certainly be in favour of
hosting it under samba.org.

Cheers, David

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Jeremy Allison
In reply to this post by ronnie sahlberg
On Tue, Jan 24, 2017 at 07:31:33PM -0800, ronnie sahlberg wrote:

> samba, please meet libsmb2
> libsmb2 please meet samba
>
> Where to host?
> --------------------
> I kind of like github/sahlberg as your on-stop-shop for everything
> related to network storage client software.
> However, hosting this on samba.org at some stage might be a better choice.
>
> You open for that idea?

Hi Ronnie - great work ! Looks really nice and
meets a need.

Personally I think hosting on samba.org is a great
idea, we are kind of the one-stop-shop for SMB+ tech
(kind of like Costco, but for interop :-). Github
is also a proprietary site, and while I have nothing
against them it'd be nice to see a Free Software project
hosted on a free software site :-).

Cheers,

        Jeremy.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

ronnie sahlberg
On Wed, Jan 25, 2017 at 1:41 PM, Jeremy Allison <[hidden email]> wrote:

> On Tue, Jan 24, 2017 at 07:31:33PM -0800, ronnie sahlberg wrote:
>> samba, please meet libsmb2
>> libsmb2 please meet samba
>>
>> Where to host?
>> --------------------
>> I kind of like github/sahlberg as your on-stop-shop for everything
>> related to network storage client software.
>> However, hosting this on samba.org at some stage might be a better choice.
>>
>> You open for that idea?
>
> Hi Ronnie - great work ! Looks really nice and
> meets a need.
>
> Personally I think hosting on samba.org is a great
> idea, we are kind of the one-stop-shop for SMB+ tech
> (kind of like Costco, but for interop :-). Github
> is also a proprietary site, and while I have nothing
> against them it'd be nice to see a Free Software project
> hosted on a free software site :-).

I am very happy to hear that.
I would like it to benefit from your build farms and your test infrastructure.
That would be win-win.

Thanks!
ronnie sahlberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Steve French-2
I would also strongly recommend hosting it on samba.org, and there is
some value if (at least) some of the headers can be shared.  Although
cifs code had been ported to user space for testing (for spec e.g.), I
like this better.

My only nit (and it is a small one), is noting that at this stage
there is near zero value (and some disadvantage in additional testing
and coding) in enabling SMB2 dialect, it is easy enough to focus 100%
on SMB3 and later dialects (or SMB2.1 or later). Leaving off the
optional features of SMB3 until they are ready is fine, but there is
almost zero value in ever negotiating SMB2 at this stage - but there
would be HUGE value in having a good user space SMB3.0 or SMB3.11
library that could be more broadly used by open source components that
won't link to Samba today.

On Wed, Jan 25, 2017 at 3:50 PM, ronnie sahlberg
<[hidden email]> wrote:

> On Wed, Jan 25, 2017 at 1:41 PM, Jeremy Allison <[hidden email]> wrote:
>> On Tue, Jan 24, 2017 at 07:31:33PM -0800, ronnie sahlberg wrote:
>>> samba, please meet libsmb2
>>> libsmb2 please meet samba
>>>
>>> Where to host?
>>> --------------------
>>> I kind of like github/sahlberg as your on-stop-shop for everything
>>> related to network storage client software.
>>> However, hosting this on samba.org at some stage might be a better choice.
>>>
>>> You open for that idea?
>>
>> Hi Ronnie - great work ! Looks really nice and
>> meets a need.
>>
>> Personally I think hosting on samba.org is a great
>> idea, we are kind of the one-stop-shop for SMB+ tech
>> (kind of like Costco, but for interop :-). Github
>> is also a proprietary site, and while I have nothing
>> against them it'd be nice to see a Free Software project
>> hosted on a free software site :-).
>
> I am very happy to hear that.
> I would like it to benefit from your build farms and your test infrastructure.
> That would be win-win.
>
> Thanks!
> ronnie sahlberg
>



--
Thanks,

Steve

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Steve French-2
On Fri, Jan 27, 2017 at 12:11 PM, Steve French <[hidden email]> wrote:

> I would also strongly recommend hosting it on samba.org, and there is
> some value if (at least) some of the headers can be shared.  Although
> cifs code had been ported to user space for testing (for spec e.g.), I
> like this better.
>
> My only nit (and it is a small one), is noting that at this stage
> there is near zero value (and some disadvantage in additional testing
> and coding) in enabling SMB2 dialect, it is easy enough to focus 100%
> on SMB3 and later dialects (or SMB2.1 or later). Leaving off the
> optional features of SMB3 until they are ready is fine, but there is
> almost zero value in ever negotiating SMB2 at this stage - but there
> would be HUGE value in having a good user space SMB3.0 or SMB3.11
> library that could be more broadly used by open source components that
> won't link to Samba today.

This e.g. in your libsmb2.c

memset(&req, 0, sizeof(struct smb2_negotiate_request));
req.struct_size = SMB2_NEGOTIATE_REQUEST_SIZE;
req.dialect_count = SMB2_NUM_DIALECTS;
req.security_mode = smb2->security_mode;
req.dialects[0] = SMB2_VERSION_0202;
req.dialects[1] = SMB2_VERSION_0210;



> On Wed, Jan 25, 2017 at 3:50 PM, ronnie sahlberg
> <[hidden email]> wrote:
>> On Wed, Jan 25, 2017 at 1:41 PM, Jeremy Allison <[hidden email]> wrote:
>>> On Tue, Jan 24, 2017 at 07:31:33PM -0800, ronnie sahlberg wrote:
>>>> samba, please meet libsmb2
>>>> libsmb2 please meet samba
>>>>
>>>> Where to host?
>>>> --------------------
>>>> I kind of like github/sahlberg as your on-stop-shop for everything
>>>> related to network storage client software.
>>>> However, hosting this on samba.org at some stage might be a better choice.


--
Thanks,

Steve

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

ronnie sahlberg
In reply to this post by Steve French-2
On Fri, Jan 27, 2017 at 10:11 AM, Steve French <[hidden email]> wrote:
> I would also strongly recommend hosting it on samba.org, and there is
> some value if (at least) some of the headers can be shared.  Although
> cifs code had been ported to user space for testing (for spec e.g.), I
> like this better.
>

Thanks Steve. I appreciate your input.

> My only nit (and it is a small one), is noting that at this stage
> there is near zero value (and some disadvantage in additional testing
> and coding) in enabling SMB2 dialect, it is easy enough to focus 100%
> on SMB3 and later dialects (or SMB2.1 or later). Leaving off the
> optional features of SMB3 until they are ready is fine, but there is
> almost zero value in ever negotiating SMB2 at this stage - but there
> would be HUGE value in having a good user space SMB3.0 or SMB3.11
> library that could be more broadly used by open source components that
> won't link to Samba today.

ACK, Switching to SMB3+ only support is a good idea. I absolutely also
want to get RDMA offload into this library just like libnfs.


I will do that. I do not have any real time to work on this this
weekend but will next weekend.
For now, see this as a q-n-d proof of concept.
I will need to spend time to clean things up a bit first.
For example, not having offsets hardcoded. Not even have offsets
exposed in the structures exposed in the API
but have them utomatically computed in the marshalling/unmarshalling functions/
There is a whole lot of cleaning up to do.
I need to audit all error paths, check for memory leaks, and buffer
overruns. etc.


I agree that there is immense value in a small, and liberally
licenced, smb3+ client.



After that I will want to have a look at your kernel code.
I think it would be very useful if we can share code. In particular
sharing the code to marshall/unmarshall smb2 pdus and infolevels are
obvious points where we can work together.
Also codepaths where we do things such as the mapping from things like
O_RDWR/O_...    to the arguments to the NTFS API are other areas.

I am very open to do this but you have to bear with me until I get
time to pick this up again in a week or two.


regards
ronnie sahlberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Samba - samba-technical mailing list
In reply to this post by ronnie sahlberg
Update. Start of unstructured brain dump:


Libsmb2 is nearing its first official release. Which means it has all
the features that QEMU and KODI (main consumers) need.
Please have a look at my github repo.


Libsmb2 is fully async and non-blocking.
It does zero-copy for both READ and WRITE commands, in the sense that
it reads/writes the application buffer directly to/from the socket
without any memcpy() being done in the library itself.  True zero-copy
will have to wait until RDMA support comes.

It also supports compounded requests making the common pattern of
CREATE/do-something/CLOSE execute in a single roundtrip.
This should make a lot of metadata operations fast-ish.

Not bloated yet, almost fitting in memory of a Sinclair Spectrum:
$ ls -l lib/.libs/libsmb2.so.1.0.0
-rwxrwxr-x. 1 sahlberg sahlberg 51816 Apr 25 16:05 lib/.libs/libsmb2.so.1.0.0


No external dependencies outside from MIT and basic libc making it,
very portable.
As I am not fluent in Visual Studio I would welcome any help I can get
to set up a proper VS project file.


At this point I think the library is in a usable state and the API
should be declared stable.
Some features should still be added but as far as main consumers, i.e.
QEMU and KODI, are concerned we
are in a releasable state for 0.1 feature wise.


I was chatting with Jeremy about features that would be nice to add
after the 0.1 release and I would be happy to work with anyone
that wants to contribute.
My idea of such post 0.1 features would be :
* krb5 support. Currently only gss-ntlmssp is supported but adding
krb5 support should be trivial.
* sing and seal.
* visual studio 1* project files and support.
* SET_INFO support
* a simple cifs fuse module.
* RDMA (this is probably not a near term goal)



Now, at the end, I just want to show what the compound API ended up
looking like.
Not too shabby me thinks.

Example of using the compound request API:
==================================
        /* CREATE command */
        memset(&cr_req, 0, sizeof(struct smb2_create_request));
        cr_req.requested_oplock_level = SMB2_OPLOCK_LEVEL_NONE;
        cr_req.impersonation_level = SMB2_IMPERSONATION_IMPERSONATION;
        cr_req.desired_access = SMB2_READ_CONTROL;
        cr_req.file_attributes = 0;
        cr_req.share_access = SMB2_FILE_SHARE_READ | SMB2_FILE_SHARE_WRITE;
        cr_req.create_disposition = SMB2_FILE_OPEN;
        cr_req.create_options = 0;
        cr_req.name = path;

        pdu = smb2_cmd_create_async(smb2, &cr_req, stat_cb_1, stat_data);
        if (pdu == NULL) {
                fprintf(stderr, "Failed to create create command");
                free(stat_data);
                return -1;
        }

        /* QUERY INFO command */
        memset(&qi_req, 0, sizeof(struct smb2_query_info_request));
        qi_req.info_type = SMB2_0_INFO_SECURITY;
        qi_req.output_buffer_length = 65535;
        qi_req.additional_information =
                SMB2_OWNER_SECURITY_INFORMATION |
                SMB2_GROUP_SECURITY_INFORMATION |
                SMB2_DACL_SECURITY_INFORMATION;
        memcpy(qi_req.file_id, compound_file_id, SMB2_FD_SIZE);

        next_pdu = smb2_cmd_query_info_async(smb2, &qi_req,
                                             stat_cb_2, stat_data);
        if (next_pdu == NULL) {
                fprintf(stderr, "Failed to create query command");
                free(stat_data);
                smb2_free_pdu(smb2, pdu);
                return -1;
        }
        smb2_add_compound_pdu(smb2, pdu, next_pdu);

        /* CLOSE command */
        memset(&cl_req, 0, sizeof(struct smb2_close_request));
        cl_req.flags = SMB2_CLOSE_FLAG_POSTQUERY_ATTRIB;
        memcpy(cl_req.file_id, compound_file_id, SMB2_FD_SIZE);

        next_pdu = smb2_cmd_close_async(smb2, &cl_req, stat_cb_3, stat_data);
        if (next_pdu == NULL) {
                stat_data->cb(smb2, -1, NULL, stat_data->cb_data);
                free(stat_data);
                smb2_free_pdu(smb2, pdu);
                return -1;
        }
        smb2_add_compound_pdu(smb2, pdu, next_pdu);

        smb2_queue_pdu(smb2, pdu);

regards
ronnie sahlberg



On Tue, Jan 24, 2017 at 7:31 PM, ronnie sahlberg
<[hidden email]> wrote:

> samba, please meet libsmb2
> libsmb2 please meet samba
>
>
> As you know already I am am for various reasons, to benefit linux, working on a
> smb2 client library.
>
> https://github.com/sahlberg/libsmb2
>
>
> Why, and what requirements?
> ----------------------------------------
> I want there to be a highly portable and free smb2 client that any/all
> applications can use.
>
> It should be small and nimble. Currently clocks in at ~50kb compiled
> (though that excludes MIT kerberos).
>
> $ ls -l lib/.libs/libsmb2.a -rw-rw-r-- 1 sahlberg sahlberg 54188 Jan
> 24 17:40 lib/.libs/libsmb2.a
>
> It should need very few external dependencies. Right now I target
> basic posix libc plus MIT. Nothing else.
>
> It should be so portable that it compiles native on both Win32/64 as
> well as Amiga Kickstart 1.3.
>
> It should provide an API similar to libnfs so that porting between nfs
> and smb wrappers should be as painless as possible.
>
> SMB2+ only. No smb1 support. RDMA support will be nice to add in the future.
>
> Zero-copy for [p]read/[p]write.
>
>
> Also, I have been a bit bored recently and what better reason for
> building a lightweight SMB2 client? Boredom is the fuel for
> innovation.
>
>
> Licence
> ----------
> For licence, gnu licences lgplv2.1 only or lgpl2.1-plus are
> acceptable. QMU licence compatibility is not negotiable. (Make me
> angry and I will consider 2-clause BSD, or worse :-) )
> Nah lgpl2.1 is fine :-)
>
> Kodi support is kind of not negotiable as well, but they are
> compatible with a lot mode different licences so that should not be an
> issue.
>
>
>
> Current status
> -------------------
> The codebase is in hue flux and will need a major refactoring and also
> audits over the next few weeks until the implementation as well as the
> APIs will settle down.
>
> Mimicing the libnfs APIs it provides fair coverage of the common posix
> file api already:
> smb2_close
> smb2_close_async
> smb2_closedir
> smb2_cmd_close_async
> smb2_cmd_create_async
> smb2_cmd_echo_async
> smb2_cmd_logoff_async
> smb2_cmd_negotiate_async
> smb2_cmd_query_directory_async
> smb2_cmd_query_info_async
> smb2_cmd_session_setup_async
> smb2_cmd_tree_connect_async
> smb2_connect_async
> smb2_connect_share
> smb2_connect_share_async
> smb2_destroy_context
> smb2_destroy_url
> smb2_fstat
> smb2_fstat_async
> smb2_get_client_guid
> smb2_get_error
> smb2_get_fd
> smb2_init_context
> smb2_mkdir
> smb2_mkdir_async
> smb2_open
> smb2_open_async
> smb2_opendir
> smb2_opendir_async
> smb2_parse_url
> smb2_pread
> smb2_pread_async
> smb2_pwrite
> smb2_pwrite_async
> smb2_read
> smb2_read_async
> smb2_readdir
> smb2_rewinddir
> smb2_rmdir
> smb2_rmdir_async
> smb2_seek
> smb2_seekdir
> smb2_service
> smb2_set_security_mode
> smb2_stat
> smb2_stat_async
> smb2_telldir
> smb2_unlink
> smb2_unlink_async
> smb2_which_events
> smb2_write
> smb2_write_async
>
> Not complete yet but sufficient for building a filemanager or something similar.
>
> It uses MIT + simo's GSS-NLTMSSP for authentication. It does not yet
> do actual sign or seal
> but I think that is low priority for the initial version. Patches welcome.
>
> It leaks memory from the gss contexts that are used during session
> setup. Again patches welcome as I am completely ignorant about MIT and
> its GSS layer, and wish to remain so as I am too old and lazy to learn
> new stuff.
>
>
> These funcitons work pretty well and even run valgrind clean (aside
> from the two leaks form the mis-use of the MIT library).
> Woohoo for that.
>
>
> Qemu and Kodi and all the other users of libnfs
> ---------------------------------------------------------------
> It supports all the posix fs functions that any of those apps use
> today for libnfs so it should be fairly easy for those folks to take
> theur libnfs wrappers, do some sed-magic and end up with an almost
> functional smb2 wrapper.
>
> Easy porting from libnfs to libsmb2 is another requirement that I just made up.
>
>
> Examples
> -------------
> There are a bunch of examples in the examples subdirectory to show the
> async as well as the sync api. Please have look.
>
>
> Fire extinguisher
> ----------------------
> This is not a replacement for libsmbclient. This is aimed as a minimal
> library that is
> easy to port and use. With limited functionality but an alternative to
> those apps that just need the most basic smb2 support and where
> libsmb2 client is overkill.
>
> Well, on amiga Kickstart 1.3 it might be a replacement as I doubt
> anyone wants to port non-extremely-ancient versions of libsmbclient to
> the amiga.
>
>
>
> Interested?
> ---------------
> It is functional right now. More posix function emulation is needed.
> It probably need a re-write and re-factor as well as the internal APIs
> settle down.
>
> Want to help out? Contact me and check out the github.
> Contacting me first before starting to prepare any pull requests. I
> expect that the code base will
> be in major flux for the next few weeks until things stabilize.
>
> There is a lot of work that still needs doing.
>
>
> Where to host?
> --------------------
> I kind of like github/sahlberg as your on-stop-shop for everything
> related to network storage client software.
> However, hosting this on samba.org at some stage might be a better choice.
>
> You open for that idea?
>
>
>
>
>
> regards, your friend in the file/block client space
> ronnie sahlberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Samba - samba-technical mailing list
On Tue, Apr 25, 2017 at 04:26:48PM -0700, ronnie sahlberg via samba-technical wrote:

> Update. Start of unstructured brain dump:
>
>
> Libsmb2 is nearing its first official release. Which means it has all
> the features that QEMU and KODI (main consumers) need.
> Please have a look at my github repo.
>
>
> Libsmb2 is fully async and non-blocking.
> It does zero-copy for both READ and WRITE commands, in the sense that
> it reads/writes the application buffer directly to/from the socket
> without any memcpy() being done in the library itself.  True zero-copy
> will have to wait until RDMA support comes.
>
> It also supports compounded requests making the common pattern of
> CREATE/do-something/CLOSE execute in a single roundtrip.
> This should make a lot of metadata operations fast-ish.
>
> Not bloated yet, almost fitting in memory of a Sinclair Spectrum:
> $ ls -l lib/.libs/libsmb2.so.1.0.0
> -rwxrwxr-x. 1 sahlberg sahlberg 51816 Apr 25 16:05 lib/.libs/libsmb2.so.1.0.0
>
>
> No external dependencies outside from MIT and basic libc making it,
> very portable.
> As I am not fluent in Visual Studio I would welcome any help I can get
> to set up a proper VS project file.
>
>
> At this point I think the library is in a usable state and the API
> should be declared stable.
> Some features should still be added but as far as main consumers, i.e.
> QEMU and KODI, are concerned we
> are in a releasable state for 0.1 feature wise.
>
>
> I was chatting with Jeremy about features that would be nice to add
> after the 0.1 release and I would be happy to work with anyone
> that wants to contribute.
> My idea of such post 0.1 features would be :
> * krb5 support. Currently only gss-ntlmssp is supported but adding
> krb5 support should be trivial.
> * sing and seal.
> * visual studio 1* project files and support.
> * SET_INFO support
> * a simple cifs fuse module.
> * RDMA (this is probably not a near term goal)
>
>
>
> Now, at the end, I just want to show what the compound API ended up
> looking like.
> Not too shabby me thinks.
>
> Example of using the compound request API:
> ==================================
>         /* CREATE command */
>         memset(&cr_req, 0, sizeof(struct smb2_create_request));
>         cr_req.requested_oplock_level = SMB2_OPLOCK_LEVEL_NONE;
>         cr_req.impersonation_level = SMB2_IMPERSONATION_IMPERSONATION;
>         cr_req.desired_access = SMB2_READ_CONTROL;
>         cr_req.file_attributes = 0;
>         cr_req.share_access = SMB2_FILE_SHARE_READ | SMB2_FILE_SHARE_WRITE;
>         cr_req.create_disposition = SMB2_FILE_OPEN;
>         cr_req.create_options = 0;
>         cr_req.name = path;
>
>         pdu = smb2_cmd_create_async(smb2, &cr_req, stat_cb_1, stat_data);
>         if (pdu == NULL) {
>                 fprintf(stderr, "Failed to create create command");
>                 free(stat_data);
>                 return -1;
>         }
>
>         /* QUERY INFO command */
>         memset(&qi_req, 0, sizeof(struct smb2_query_info_request));
>         qi_req.info_type = SMB2_0_INFO_SECURITY;
>         qi_req.output_buffer_length = 65535;
>         qi_req.additional_information =
>                 SMB2_OWNER_SECURITY_INFORMATION |
>                 SMB2_GROUP_SECURITY_INFORMATION |
>                 SMB2_DACL_SECURITY_INFORMATION;
>         memcpy(qi_req.file_id, compound_file_id, SMB2_FD_SIZE);
>
>         next_pdu = smb2_cmd_query_info_async(smb2, &qi_req,
>                                              stat_cb_2, stat_data);
>         if (next_pdu == NULL) {
>                 fprintf(stderr, "Failed to create query command");
>                 free(stat_data);
>                 smb2_free_pdu(smb2, pdu);
>                 return -1;
>         }
>         smb2_add_compound_pdu(smb2, pdu, next_pdu);
>
>         /* CLOSE command */
>         memset(&cl_req, 0, sizeof(struct smb2_close_request));
>         cl_req.flags = SMB2_CLOSE_FLAG_POSTQUERY_ATTRIB;
>         memcpy(cl_req.file_id, compound_file_id, SMB2_FD_SIZE);
>
>         next_pdu = smb2_cmd_close_async(smb2, &cl_req, stat_cb_3, stat_data);
>         if (next_pdu == NULL) {
>                 stat_data->cb(smb2, -1, NULL, stat_data->cb_data);
>                 free(stat_data);
>                 smb2_free_pdu(smb2, pdu);
>                 return -1;
>         }
>         smb2_add_compound_pdu(smb2, pdu, next_pdu);
>
>         smb2_queue_pdu(smb2, pdu);
>
> regards
> ronnie sahlberg

This looks nice Ronnie ! Do you want to make it a
project under samba.org somewhere ?

Makes all the libraries available in one place !

Jeremy.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Wednesday, 26 April 2017 01:26:48 CEST ronnie sahlberg via samba-technical
wrote:

> Update. Start of unstructured brain dump:
>
>
> Libsmb2 is nearing its first official release. Which means it has all
> the features that QEMU and KODI (main consumers) need.
> Please have a look at my github repo.
>
>
> Libsmb2 is fully async and non-blocking.
> It does zero-copy for both READ and WRITE commands, in the sense that
> it reads/writes the application buffer directly to/from the socket
> without any memcpy() being done in the library itself.  True zero-copy
> will have to wait until RDMA support comes.
>
> It also supports compounded requests making the common pattern of
> CREATE/do-something/CLOSE execute in a single roundtrip.
> This should make a lot of metadata operations fast-ish.
>
> Not bloated yet, almost fitting in memory of a Sinclair Spectrum:
> $ ls -l lib/.libs/libsmb2.so.1.0.0
> -rwxrwxr-x. 1 sahlberg sahlberg 51816 Apr 25 16:05
> lib/.libs/libsmb2.so.1.0.0
>
>
> No external dependencies outside from MIT and basic libc making it,
> very portable.
> As I am not fluent in Visual Studio I would welcome any help I can get
> to set up a proper VS project file.

For something like that I really suggest to switch to the cmake build system.
cmake is a metabuild system, it produces make, ninja, msys, mingw or VS
project files. Once you have everything working on Linux, Windows is just a
few lines of cmake away.

Here is my highly portable unit testing framework called cmocka.

Website: https://cmocka.org/
Git:     https://git.cryptomilk.org/projects/cmocka.git/


You might want to use that to write unit tests ;)


        Andreas

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
Hi Ronnie,

Am 26.04.2017 um 01:26 schrieb ronnie sahlberg via samba-technical:

> Update. Start of unstructured brain dump:
>
>
> Libsmb2 is nearing its first official release. Which means it has all
> the features that QEMU and KODI (main consumers) need.
> Please have a look at my github repo.
>
>
> Libsmb2 is fully async and non-blocking.
> It does zero-copy for both READ and WRITE commands, in the sense that
> it reads/writes the application buffer directly to/from the socket
> without any memcpy() being done in the library itself.  True zero-copy
> will have to wait until RDMA support comes.
>
> It also supports compounded requests making the common pattern of
> CREATE/do-something/CLOSE execute in a single roundtrip.
> This should make a lot of metadata operations fast-ish.
Can you point me to the code that takes care of the credit handling?

Do you support large reads and writes using SMB2_CAP_LARGE_MTU?
Or is this basically just the SMB 2.02 feature set?

> Not bloated yet, almost fitting in memory of a Sinclair Spectrum:
> $ ls -l lib/.libs/libsmb2.so.1.0.0
> -rwxrwxr-x. 1 sahlberg sahlberg 51816 Apr 25 16:05 lib/.libs/libsmb2.so.1.0.0
>
>
> No external dependencies outside from MIT and basic libc making it,
> very portable.
> As I am not fluent in Visual Studio I would welcome any help I can get
> to set up a proper VS project file.
>
>
> At this point I think the library is in a usable state and the API
> should be declared stable.
I'd propose to use 'uint8_t' instead of 'char' for buffers.

And also add as much 'const' to pointers within structs and
function arguments.

Global variables should also be [static] const.

> Some features should still be added but as far as main consumers, i.e.
> QEMU and KODI, are concerned we
> are in a releasable state for 0.1 feature wise.

While I don't really like the callback based api, I guess it's that
way because libnfs uses something like this, which makes it easier
to adopt for the main consumers.

> I was chatting with Jeremy about features that would be nice to add
> after the 0.1 release and I would be happy to work with anyone
> that wants to contribute.
> My idea of such post 0.1 features would be :
> * krb5 support. Currently only gss-ntlmssp is supported but adding
> krb5 support should be trivial.
> * sing and seal.

You should also ask gss_init_sec_context for GSS_C_INTEG_FLAG,
even without signing in use.

> * visual studio 1* project files and support.
> * SET_INFO support
> * a simple cifs fuse module.
> * RDMA (this is probably not a near term goal)

As the api is very low level and (as far as I can see)
only based on SMB 2.02 features, I guess it will be hard
to add features like encryption, multi-channel and smb-direct.

But it's very good to have a start that seems to work,
it can always be improved later.

Do you have some QEMU code that uses this api?

Thanks!
metze




signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Samba - samba-technical mailing list
Stephan, Thanks for your review.

On Tue, Apr 25, 2017 at 11:53 PM, Stefan Metzmacher <[hidden email]> wrote:

> Hi Ronnie,
>
> Am 26.04.2017 um 01:26 schrieb ronnie sahlberg via samba-technical:
>> Update. Start of unstructured brain dump:
>>
>>
>> Libsmb2 is nearing its first official release. Which means it has all
>> the features that QEMU and KODI (main consumers) need.
>> Please have a look at my github repo.
>>
>>
>> Libsmb2 is fully async and non-blocking.
>> It does zero-copy for both READ and WRITE commands, in the sense that
>> it reads/writes the application buffer directly to/from the socket
>> without any memcpy() being done in the library itself.  True zero-copy
>> will have to wait until RDMA support comes.
>>
>> It also supports compounded requests making the common pattern of
>> CREATE/do-something/CLOSE execute in a single roundtrip.
>> This should make a lot of metadata operations fast-ish.
>
> Can you point me to the code that takes care of the credit handling?
>
> Do you support large reads and writes using SMB2_CAP_LARGE_MTU?
> Or is this basically just the SMB 2.02 feature set?

Yes, and yes. I just added this in :
https://github.com/sahlberg/libsmb2/commit/0a8066f86fa1f9088372a5564ab92fc9ab49f12f
https://github.com/sahlberg/libsmb2/commit/4bb794def87a25ff3af00849402699c136177d54
https://github.com/sahlberg/libsmb2/commit/4bb794def87a25ff3af00849402699c136177d54

I still need to enforce the 64k for !multi_credit in the
smb2_[p]{read|write} functions.


>
>> Not bloated yet, almost fitting in memory of a Sinclair Spectrum:
>> $ ls -l lib/.libs/libsmb2.so.1.0.0
>> -rwxrwxr-x. 1 sahlberg sahlberg 51816 Apr 25 16:05 lib/.libs/libsmb2.so.1.0.0
>>
>>
>> No external dependencies outside from MIT and basic libc making it,
>> very portable.
>> As I am not fluent in Visual Studio I would welcome any help I can get
>> to set up a proper VS project file.
>>
>>
>> At this point I think the library is in a usable state and the API
>> should be declared stable.
>
> I'd propose to use 'uint8_t' instead of 'char' for buffers.
>
> And also add as much 'const' to pointers within structs and
> function arguments.
>
> Global variables should also be [static] const.

Good hygiene suggestions. Done.

>
>> Some features should still be added but as far as main consumers, i.e.
>> QEMU and KODI, are concerned we
>> are in a releasable state for 0.1 feature wise.
>
> While I don't really like the callback based api, I guess it's that
> way because libnfs uses something like this, which makes it easier
> to adopt for the main consumers.
>
>> I was chatting with Jeremy about features that would be nice to add
>> after the 0.1 release and I would be happy to work with anyone
>> that wants to contribute.
>> My idea of such post 0.1 features would be :
>> * krb5 support. Currently only gss-ntlmssp is supported but adding
>> krb5 support should be trivial.
>> * sing and seal.
>
> You should also ask gss_init_sec_context for GSS_C_INTEG_FLAG,
> even without signing in use.
>
>> * visual studio 1* project files and support.
>> * SET_INFO support
>> * a simple cifs fuse module.
>> * RDMA (this is probably not a near term goal)
>
> As the api is very low level and (as far as I can see)
> only based on SMB 2.02 features, I guess it will be hard
> to add features like encryption, multi-channel and smb-direct.

Like libnfs, there are both highlevel posix-like APIs (sync and async)
as well as a raw low level smb API.
For now it is mostly <=2.02 level, except for the large_mtu support.


>
> But it's very good to have a start that seems to work,
> it can always be improved later.
>
> Do you have some QEMU code that uses this api?

It will basically be almost identical to
http://git.qemu.org/?p=qemu.git;a=blob;f=block/nfs.c;h=6541dec1fcc1afd46a6226ae7c0381b6b9f727f5;hb=81b2d5ceb0cfb4cdc2163492e3169ed714b0cda9
Once I cut the first initial pre-release Peter (the guy that now
maintains qemu nfs and iscsi) will create smb2.c for qemu.

Later on it would be very nice to add both SMB Direct as well as
discard/unmap support as both features are very important for
cloud/enterprise/qemu users.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: libsmb2, meet samba

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Apr 26, 2017 02:27, "ronnie sahlberg via samba-technical" <
[hidden email]> wrote:


My idea of such post 0.1 features would be :
* krb5 support. Currently only gss-ntlmssp is supported but adding
krb5 support should be trivial.


Note that gss_init would block on network IO when krb5 mechanism is used.
Loading...