[ANNOUNCE] Samba 4.0 beta5

[Andrew Bartlett]

We are proud to a announce another beta release of Samba 4.0, beta5
(the required ldb 1.1.9 release will follow shortly).


What's new in Samba 4.0 beta5
=============================

Samba 4.0 will be the next version of the Samba suite and incorporates
all the technology found in both the Samba4 alpha series and the
stable 3.x series. The primary additional features over Samba 3.6 are
support for the Active Directory logon protocols used by Windows 2000
and above.


WARNINGS
========

Samba 4.0 beta5 is not a final Samba release, however we are now making
good progress towards a Samba 4.0 release, of which this is a preview.
Be aware the this release contains the best of all of Samba's
technology parts, both a file server (that you can reasonably expect
to upgrade existing Samba 3.x releases to) and the AD domain
controller work previously known as 'samba4'.

Samba 4.0 is subjected to an awesome battery of tests on an automated
basis, we have found Samba 4.0 to be very stable in it's behavior.
However, we still recommend against upgrading production servers from
Samba 3.x release to Samba 4.0 beta at this stage.

In particular note that the new default configuration 's3fs' may have
different stability characteristics compared with our previous default
file server.  We are making this release so that we can find and fix
any of these issues that arise in the real world.  New AD DC
installations can provision or join with --use-ntvfs to obtain the
previous default file server.  See below how to continue using ntvfs
in an existing installation.

If you are upgrading, or looking to develop, test or deploy Samba 4.0
beta releases, you should backup all configuration and data.


UPGRADING
=========

Users upgrading from Samba 3.x domain controllers and wanting to use
Samba 4.0 as an AD DC should use the 'samba-tool domain
classicupgrade' command.  See the wiki for more details:
https://wiki.samba.org/index.php/Samba4/samba3upgrade/HOWTO 

Users upgrading from Samba 4.0 alpha and beta releases since alpha15
should run 'samba-tool dbcheck --cross-ncs --fix'.  Users upgrading
from earlier alpha releases should contact the team for advice.


NEW FEATURES
============

Samba 4.0 beta supports the server-side of the Active Directory logon
environment used by Windows 2000 and later, so we can do full domain
join and domain logon operations with these clients.

Our Domain Controller (DC) implementation includes our own built-in
LDAP server and Kerberos Key Distribution Center (KDC) as well as the
Samba3-like logon services provided over CIFS.  We correctly generate
the infamous Kerberos PAC, and include it with the Kerberos tickets we
issue.

Samba 4.0 beta ships with two distinct file servers.  We now use the
file server from the Samba 3.x series 'smbd' for all file serving by
default.  For pure file server work, the binaries users would expect
from that series (nmbd, winbindd, smbpasswd) continue to be available.

Samba 4.0 also ships with the 'NTVFS' file server.  This file server
is what was used in all previous alpha releases of Samba 4.0, and is
tuned to match the requirements of an AD domain controller.  We
continue to support this, not only to provide continuity to
installations that have deployed it as part of an AD DC, but also as a
running example of the NT-FSA architecture we expect to move smbd to in
the longer term.  

As mentioned above, this change to the default file server may cause
instability, as we learn about the real-world interactions between
these two key components.

As DNS is an integral part of Active Directory, we also provide a DNS
solution, using the BIND DLZ mechanism in versions 9.8 and 9.9.
During the provision, a configuration file will be generated for bind
to make it use this plugin.  We also have a project to provide a
minimal internal DNS server from within the Samba process, for easier
'out of the box' configuration.  Note however that this is not yet
complete (pending addition of secure DNS update support).

To provide accurate timestamps to Windows clients, we integrate with
the NTP project to provide secured NTP replies.

Finally, a new scripting interface has been added to Samba 4, allowing
Python programs to interface to Samba's internals, and many tools and
internal workings of the DC code is now implemented in python.


CHANGES SINCE beta4
=====================

For a list of changes since beta4, please see the git log.

$ git clone git://git.samba.org/samba.git
$ cd samba.git
$ git log samba-4.0.0beta4..samba-4.0.0beta5

Some major user-visible changes include:

- The issue with beta4 being unable to build with a released version of
  ldb has been resolved.

- The two parameter tables for our two smb.conf parsing engines have
  been merged.  This removes the ugly (but harmless) "unknown
  parameter xxx" warnings, particularly from the smbd child process.

- Major issues have been fixed in conflict and missing/deleted parent
  handling in or DRS replication engine.

- Safety improvements to prevent corruption of read-write replicas
  by manual replication from a read-only replica.

- Improvements to dbcheck to correct incorrect instanceType values from
  the above and to relocate objects with missing parents.

- smbd no longer places all accounts in the 'Domain Users' of the AD
  domain to which it is joined

- AES support in NETLOGON Schannel

- DCE/RPC timeout handling no longer crashes

- "socket address" is now "nbt client socket address" as it only
  controls the binding of the NetBIOS client, not other protocols.
  See 'bind interfaces only = yes'.  This parameter is also now depricated.

- nmbd now always binds to it's broadcast sockets explicitly, rather
  than just relying on the socket address above.

Less visible, but important changes under the hood include:

- Continued work to support SMB2 and SMB3

- Continued work to use async IO to improve file server performance.

- Patches to ensure that talloc_tos() and talloc_stackframe() are
  always used correctly.

KNOWN ISSUES
============

- This release makes the s3fs file server the default, as this is the
  file server combination we will use for the Samba 4.0 release.

- Modifying of group policies by members of the Domain Administrators
  group is not possible with the s3fs file server, only with the ntvfs
  file server.  This is due to the underlying POSIX ACL not being set
  at provision time.

- For similar reasons, sites with ACLs stored by the ntvfs file server
  may wish to continue to use that file server implementation, as a
  posix ACL will similarly not be set in this case.

- Replication of DNS data from one AD server to another may not work.
  The DNS data used by the internal DNS server and bind9_dlz is stored
  in an application partition in our directory.  The replication of
  this partition is not yet reliable.

- Replication may fail on FreeBSD due to getaddrinfo() rejecting names
  containing _.  A workaround will be in the next beta.

- upgradeprovision should not be run when upgrading to this release
  from a recent release.  No important database format changes have
  been made since alpha16.  

- Installation on systems without a system iconv (and developer
  headers at compile time) is known to cause errors when dealing with
  non-ASCII characters.

- Domain member support in the 'samba' binary is in it's infancy, and
  is not comparable to the support found in winbindd.  As such, do not
  use the 'samba' binary (provided for the AD server) on a member
  server.

- There is no NetBIOS browsing support (network neighbourhood) in the
  'samba' binary (use nmbd and smbd instead)

- Clock Synchronisation is critical.  Many 'wrong password' errors are
  actually due to Kerberos objecting to a clock skew between client
  and server.  (The NTP work in the previous alphas are partly to assist
  with this problem).

- The DRS replication code may fail.  Please contact the team if you
  experience issues with DRS replication, as we have fixed many issues
  here in response to feedback from our production users.


RUNNING Samba 4.0 as an AD DC
=============================

A short guide to setting up Samba 4 as an AD DC can be found on the wiki:

  http://wiki.samba.org/index.php/Samba4/HOWTO

#######################################
Reporting bugs & Development Discussion
#######################################

Please discuss this release on the samba-technical mailing list or by
joining the #samba-technical IRC channel on irc.freenode.net.

If you do report problems then please try to send high quality
feedback. If you don't provide vital information to help us track down
the problem then you will probably be ignored.  All bug reports should
be filed under the Samba 4.0 product in the project's Bugzilla
database (https://bugzilla.samba.org/).


======================================================================
== Our Code, Our Bugs, Our Responsibility.
== The Samba Team
======================================================================




Download Details
================

The release tarball is available from the following location:
 * http://download.samba.org/samba/ftp/samba4/samba-4.0.0beta5.tar.gz

This release has been signed using GPG with Andrew's GPG key 28B436BB).

 * http://download.samba.org/samba/ftp/samba4/samba-4.0.0beta5.tar.asc

To verify that the signature is correct, make sure that the tarball has
been unzipped and run:

$ gpg --verify samba-4.0.0beta5.tar.asc

Happy testing!

The Samba team
--
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org

[scottj]

I'm a little confused on this whole ACL and s3fs file system issue.

 >  Modifying of group policies by members of the Domain Administrators
 >  group is not possible with the s3fs file server, only with the ntvfs
 >  file server.  This is due to the underlying POSIX ACL not being set
 >  at provision time.

I have a production site that's running and now using s3fs. I elected to
create a whole new, clean domain using beta4 (they had a Win2k3 domain).
There's only 15 users/computers, so it wasn't too hard to re-create. The
server was previously acting as a file server, running Samba3 and acting
as a member server to the older Win2k3 AD domain (The samba server,
BTW,  is Ubuntu 10.04 LTS x64)..

To enable GPOs, is there a way to use setfacl set the necessary ACL
default values after the provisioning? If so, what ACLs need to be set?
Do you set ACLs on all files/directories in the file shares or just the
ones in SYSVOL? It's also a little confusing on how Windows ACLs map to
Posix ACLs. What ACL values need to be set? I need to clean up file
access as the files/folders still hold old S3 IDMAP entries.

-- Scott

[Andrew Bartlett]

On Wed, 2012-08-01 at 15:49 -0700, Scott Jordahl wrote:

> I'm a little confused on this whole ACL and s3fs file system issue.
>
>  >  Modifying of group policies by members of the Domain Administrators
>  >  group is not possible with the s3fs file server, only with the ntvfs
>  >  file server.  This is due to the underlying POSIX ACL not being set
>  >  at provision time.
>
> I have a production site that's running and now using s3fs. I elected to
> create a whole new, clean domain using beta4 (they had a Win2k3 domain).
> There's only 15 users/computers, so it wasn't too hard to re-create. The
> server was previously acting as a file server, running Samba3 and acting
> as a member server to the older Win2k3 AD domain (The samba server,
> BTW,  is Ubuntu 10.04 LTS x64)..
>
> To enable GPOs, is there a way to use setfacl set the necessary ACL
> default values after the provisioning? If so, what ACLs need to be set?
> Do you set ACLs on all files/directories in the file shares or just the
> ones in SYSVOL? It's also a little confusing on how Windows ACLs map to
> Posix ACLs. What ACL values need to be set? I need to clean up file
> access as the files/folders still hold old S3 IDMAP entries.

So, what is set by provision will allow GPOs to work, because from the
client the read access is all correct, and the ACLs match.

However, for writes it only works as administrator, because I
misunderstood the smbd ACL model.  In smbd, the POSIX ACL trumps all
(except in very exceptional circumstances), where as in the ntvfs file
server, the NT ACL trumps all, overriding an incorrect POSIX ACL.

I know how to call the code to set the POSIX ACL, I just need to sort
out some remaining details and implement it.  It remains one of my
high-priority TODO items.

Andrew Bartlett

--
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org

[Andrew Bartlett]

On Thu, 2012-08-02 at 09:02 +1000, Andrew Bartlett wrote:

> On Wed, 2012-08-01 at 15:49 -0700, Scott Jordahl wrote:
> > I'm a little confused on this whole ACL and s3fs file system issue.
> >
> >  >  Modifying of group policies by members of the Domain Administrators
> >  >  group is not possible with the s3fs file server, only with the ntvfs
> >  >  file server.  This is due to the underlying POSIX ACL not being set
> >  >  at provision time.
> >
> > I have a production site that's running and now using s3fs. I elected to
> > create a whole new, clean domain using beta4 (they had a Win2k3 domain).
> > There's only 15 users/computers, so it wasn't too hard to re-create. The
> > server was previously acting as a file server, running Samba3 and acting
> > as a member server to the older Win2k3 AD domain (The samba server,
> > BTW,  is Ubuntu 10.04 LTS x64)..
> >
> > To enable GPOs, is there a way to use setfacl set the necessary ACL
> > default values after the provisioning? If so, what ACLs need to be set?
> > Do you set ACLs on all files/directories in the file shares or just the
> > ones in SYSVOL? It's also a little confusing on how Windows ACLs map to
> > Posix ACLs. What ACL values need to be set? I need to clean up file
> > access as the files/folders still hold old S3 IDMAP entries.
>
> So, what is set by provision will allow GPOs to work, because from the
> client the read access is all correct, and the ACLs match.
>
> However, for writes it only works as administrator, because I
> misunderstood the smbd ACL model.  In smbd, the POSIX ACL trumps all
> (except in very exceptional circumstances), where as in the ntvfs file
> server, the NT ACL trumps all, overriding an incorrect POSIX ACL.
>
> I know how to call the code to set the POSIX ACL, I just need to sort
> out some remaining details and implement it.  It remains one of my
> high-priority TODO items.

I've finally made a start on this, and it shows in part that we need to
be quite careful to have good tests for the NT -> POSIX ACL layer (we
don't so far).  

I've pushed what I have so far to:
https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/posix-acl-provision

This fails, because the POSIX ACL generated is invalid (multiple primary
group entries).  I'll need to fix that, and fix up a unit test to prove
correct behaviour here.

Once this is done, I'll probably add a --gpo-acl and --gpo-sync option
to dbcheck, to have it compare the group policy object ACLs with the AD
ACLs (or force re-syncing them).  The former will just fix up
inconsistancies but the latter will be really important as a tool to fix
up existing installations to have a correct POSIX ACL.

Andrew Bartlett

Andrew Bartlett

--
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org

[Andrew Bartlett]

On Thu, 2012-08-02 at 20:08 +1000, Andrew Bartlett wrote:

> On Thu, 2012-08-02 at 09:02 +1000, Andrew Bartlett wrote:
> >
> > So, what is set by provision will allow GPOs to work, because from the
> > client the read access is all correct, and the ACLs match.
> >
> > However, for writes it only works as administrator, because I
> > misunderstood the smbd ACL model.  In smbd, the POSIX ACL trumps all
> > (except in very exceptional circumstances), where as in the ntvfs file
> > server, the NT ACL trumps all, overriding an incorrect POSIX ACL.
> >
> > I know how to call the code to set the POSIX ACL, I just need to sort
> > out some remaining details and implement it.  It remains one of my
> > high-priority TODO items.
>
> I've finally made a start on this, and it shows in part that we need to
> be quite careful to have good tests for the NT -> POSIX ACL layer (we
> don't so far).  
>
> I've pushed what I have so far to:
> https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/posix-acl-provision
>
> This fails, because the POSIX ACL generated is invalid (multiple primary
> group entries).  I'll need to fix that, and fix up a unit test to prove
> correct behaviour here.

I've updated my branch at posix-acl-provision.  This now sets the posix
ACL using the VFS layer during provision.  (in turn, this also moves us
closer to supporting other ACL backends in the AD DC).

As this changes the base posix ACL mapping layer, it needs review.  Once
reviewed it will allow us to properly support group policy modifications
in smbd after a fresh provision.

I'll continue to update my branch with any tests I can manage to come up
with.  Hopefully the motivation is clear:  Multiple things map to gid 0
in a standard Samba4 provision (and even if we fix that, existing idmap
databases will have such a mapping).

As the POSIX ACL rules are enforced by the posix ID, the comparison
should be of the posix ID.

Thanks,

Andrew Bartlett

--
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org

[Andrew Bartlett]

On Tue, 2012-08-07 at 15:41 +1000, Andrew Bartlett wrote:

> On Thu, 2012-08-02 at 20:08 +1000, Andrew Bartlett wrote:
> > On Thu, 2012-08-02 at 09:02 +1000, Andrew Bartlett wrote:
> > >
> > > So, what is set by provision will allow GPOs to work, because from the
> > > client the read access is all correct, and the ACLs match.
> > >
> > > However, for writes it only works as administrator, because I
> > > misunderstood the smbd ACL model.  In smbd, the POSIX ACL trumps all
> > > (except in very exceptional circumstances), where as in the ntvfs file
> > > server, the NT ACL trumps all, overriding an incorrect POSIX ACL.
> > >
> > > I know how to call the code to set the POSIX ACL, I just need to sort
> > > out some remaining details and implement it.  It remains one of my
> > > high-priority TODO items.
> >
> > I've finally made a start on this, and it shows in part that we need to
> > be quite careful to have good tests for the NT -> POSIX ACL layer (we
> > don't so far).  
> >
> > I've pushed what I have so far to:
> > https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/posix-acl-provision
> >
> > This fails, because the POSIX ACL generated is invalid (multiple primary
> > group entries).  I'll need to fix that, and fix up a unit test to prove
> > correct behaviour here.
>
> I've updated my branch at posix-acl-provision.  This now sets the posix
> ACL using the VFS layer during provision.  (in turn, this also moves us
> closer to supporting other ACL backends in the AD DC).
>
> As this changes the base posix ACL mapping layer, it needs review.  Once
> reviewed it will allow us to properly support group policy modifications
> in smbd after a fresh provision.
>
> I'll continue to update my branch with any tests I can manage to come up
> with.  Hopefully the motivation is clear:  Multiple things map to gid 0
> in a standard Samba4 provision (and even if we fix that, existing idmap
> databases will have such a mapping).
>
> As the POSIX ACL rules are enforced by the posix ID, the comparison
> should be of the posix ID.

Jeremy,

I've updated the branch with the other patches that you punted on a
couple of months ago.  The NFSv4 patch is required if you want us to
support being an AD DC on ZFS, but is not as critical as the rest.

https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/posix-acl-provision

These are needed so that we can write posix ACLs out during provision
(not included in this branch, as I need to set the right VFS modules and
write my tests to ensure they can be read back).

Andrew Bartlett
--
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org

[Jeremy Allison]

On Fri, Aug 10, 2012 at 04:38:22PM +1000, Andrew Bartlett wrote:

> Jeremy,
>
> I've updated the branch with the other patches that you punted on a
> couple of months ago.  The NFSv4 patch is required if you want us to
> support being an AD DC on ZFS, but is not as critical as the rest.
>
> https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/posix-acl-provision
>
> These are needed so that we can write posix ACLs out during provision
> (not included in this branch, as I need to set the right VFS modules and
> write my tests to ensure they can be read back).

Sorry for the delay. These all LGTM and I've pushed them
to autobuild.

Cheers,

        Jeremy.