DNS partitions replication on secondary DC is not full

[danicat]

Hi samba team,
I've some problems with the dns of the secondary DC.

I have 2 samba4 DCs: kdc01 and kdc02 (respectively Version
4.0.0alpha21-GIT-7b55ec2 and Version 4.0.0alpha21-GIT-8026550).
I have successfully joined the secondary DC and replication seems to be
working fine.

As said in another thread I see that replication between DNS zones is
not full:

[root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc01
_msdcs.saitelitalia.local @ ALL -U administrator
...
  Name=, Records=2, Children=0
    NS: kdc01.saitelitalia.local. (flags=600000f0, serial=1, ttl=900)
    SOA: serial=147, refresh=900, retry=600, expire=86400,
ns=kdc01.saitelitalia.local., email=hostmaster.saitelitalia.local.
(flags=600000f0, serial=146, ttl=3600)
  Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=1, Children=0
    CNAME: kdc02.saitelitalia.local. (flags=f0, serial=284, ttl=900)
  Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=1, Children=0
    CNAME: kdc01.saitelitalia.local. (flags=f0, serial=1, ttl=900)
  Name=dc, Records=0, Children=2
  Name=domains, Records=0, Children=1
  Name=gc, Records=0, Children=2
  Name=kdc01, Records=1, Children=0
    NS: 192.168.12.5. (flags=f0, serial=62, ttl=900)
  Name=pdc, Records=0, Children=1

[root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc02
_msdcs.saitelitalia.local @ ALL -U administrator
...
  Name=, Records=0, Children=0
  Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=0, Children=0
  Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=0, Children=0
  Name=dc, Records=0, Children=2
  Name=domains, Records=0, Children=1
  Name=gc, Records=0, Children=2
  Name=kdc01, Records=0, Children=0
  Name=pdc, Records=0, Children=1

If I shutdown kdc01, kdc02 is not able to keep things working (no _ldap,
_kerberos and other records are present in secondary DNS).

samba_dnsupdate --verbose works fine on secondary DC while primary is on
but if I remove from resolv.conf the address of the primary DC/DNS and
leave just the address of the secondary DC/DNS it (takes a long time)
says that all records are missing and when it tries to auth to krb it
fails (again no _kerberos.udp... record present).

I tried to add these records by hand to see if something goes better but
if I try to add records on secondary DC, samba-tool fails always saying:
[root@kdc02:/usr/local/samba/private]# samba-tool dns add kdc02
saitelitalia.local kdc01 A 192.168.12.5 -U administrator
...
ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
  File
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
line 160, in _run
    return self.run(*args, **kwargs)
  File
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
1055, in run
    None)

while it works fine on primary.

I'm a little bit confused by the error message because
WERR_INTERNAL_DB_ERROR seems to be related to an error in adding the
record to the DB but in line 1055 of .../samba/netcmd/dns.py it seems
that the problem is related to some missing/wrong argument to the update
record call.

Am I doing something wrong?

I'll be happy to contribute but need to be addressed how.

Thanks,
Daniele.

[danicat]

On Mon, 2012-05-07 at 11:42 +0200, Daniele Dario wrote:

> Hi samba team,
> I've some problems with the dns of the secondary DC.
>
> I have 2 samba4 DCs: kdc01 and kdc02 (respectively Version
> 4.0.0alpha21-GIT-7b55ec2 and Version 4.0.0alpha21-GIT-8026550).
> I have successfully joined the secondary DC and replication seems to be
> working fine.
>
> As said in another thread I see that replication between DNS zones is
> not full:
>
> [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc01
> _msdcs.saitelitalia.local @ ALL -U administrator
> ...
>   Name=, Records=2, Children=0
>     NS: kdc01.saitelitalia.local. (flags=600000f0, serial=1, ttl=900)
>     SOA: serial=147, refresh=900, retry=600, expire=86400,
> ns=kdc01.saitelitalia.local., email=hostmaster.saitelitalia.local.
> (flags=600000f0, serial=146, ttl=3600)
>   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=1, Children=0
>     CNAME: kdc02.saitelitalia.local. (flags=f0, serial=284, ttl=900)
>   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=1, Children=0
>     CNAME: kdc01.saitelitalia.local. (flags=f0, serial=1, ttl=900)
>   Name=dc, Records=0, Children=2
>   Name=domains, Records=0, Children=1
>   Name=gc, Records=0, Children=2
>   Name=kdc01, Records=1, Children=0
>     NS: 192.168.12.5. (flags=f0, serial=62, ttl=900)
>   Name=pdc, Records=0, Children=1
>
> [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc02
> _msdcs.saitelitalia.local @ ALL -U administrator
> ...
>   Name=, Records=0, Children=0
>   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=0, Children=0
>   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=0, Children=0
>   Name=dc, Records=0, Children=2
>   Name=domains, Records=0, Children=1
>   Name=gc, Records=0, Children=2
>   Name=kdc01, Records=0, Children=0
>   Name=pdc, Records=0, Children=1
>
> If I shutdown kdc01, kdc02 is not able to keep things working (no _ldap,
> _kerberos and other records are present in secondary DNS).
>
> samba_dnsupdate --verbose works fine on secondary DC while primary is on
> but if I remove from resolv.conf the address of the primary DC/DNS and
> leave just the address of the secondary DC/DNS it (takes a long time)
> says that all records are missing and when it tries to auth to krb it
> fails (again no _kerberos.udp... record present).
>
> I tried to add these records by hand to see if something goes better but
> if I try to add records on secondary DC, samba-tool fails always saying:
> [root@kdc02:/usr/local/samba/private]# samba-tool dns add kdc02
> saitelitalia.local kdc01 A 192.168.12.5 -U administrator
> ...
> ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
>   File
> "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
> line 160, in _run
>     return self.run(*args, **kwargs)
>   File
> "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
> 1055, in run
>     None)
>
> while it works fine on primary.
>
> I'm a little bit confused by the error message because
> WERR_INTERNAL_DB_ERROR seems to be related to an error in adding the
> record to the DB but in line 1055 of .../samba/netcmd/dns.py it seems
> that the problem is related to some missing/wrong argument to the update
> record call.
>
> Am I doing something wrong?
>
> I'll be happy to contribute but need to be addressed how.
>
> Thanks,
> Daniele.
>

After some other tries, I've seen that an update (or for linux boxes
with fixed addresses a delete+add) of records on the zones of the
primary DC/DNS, records have appeared also on secondary DC/DNS.

Next step I'll try to stop primary DC/DNS to see if secondary keeps the
domain up.

Daniele.

[danicat]

On Mon, 2012-05-07 at 16:46 +0200, Daniele Dario wrote:

> On Mon, 2012-05-07 at 11:42 +0200, Daniele Dario wrote:
> > Hi samba team,
> > I've some problems with the dns of the secondary DC.
> >
> > I have 2 samba4 DCs: kdc01 and kdc02 (respectively Version
> > 4.0.0alpha21-GIT-7b55ec2 and Version 4.0.0alpha21-GIT-8026550).
> > I have successfully joined the secondary DC and replication seems to be
> > working fine.
> >
> > As said in another thread I see that replication between DNS zones is
> > not full:
> >
> > [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc01
> > _msdcs.saitelitalia.local @ ALL -U administrator
> > ...
> >   Name=, Records=2, Children=0
> >     NS: kdc01.saitelitalia.local. (flags=600000f0, serial=1, ttl=900)
> >     SOA: serial=147, refresh=900, retry=600, expire=86400,
> > ns=kdc01.saitelitalia.local., email=hostmaster.saitelitalia.local.
> > (flags=600000f0, serial=146, ttl=3600)
> >   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=1, Children=0
> >     CNAME: kdc02.saitelitalia.local. (flags=f0, serial=284, ttl=900)
> >   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=1, Children=0
> >     CNAME: kdc01.saitelitalia.local. (flags=f0, serial=1, ttl=900)
> >   Name=dc, Records=0, Children=2
> >   Name=domains, Records=0, Children=1
> >   Name=gc, Records=0, Children=2
> >   Name=kdc01, Records=1, Children=0
> >     NS: 192.168.12.5. (flags=f0, serial=62, ttl=900)
> >   Name=pdc, Records=0, Children=1
> >
> > [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc02
> > _msdcs.saitelitalia.local @ ALL -U administrator
> > ...
> >   Name=, Records=0, Children=0
> >   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=0, Children=0
> >   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=0, Children=0
> >   Name=dc, Records=0, Children=2
> >   Name=domains, Records=0, Children=1
> >   Name=gc, Records=0, Children=2
> >   Name=kdc01, Records=0, Children=0
> >   Name=pdc, Records=0, Children=1
> >
> > If I shutdown kdc01, kdc02 is not able to keep things working (no _ldap,
> > _kerberos and other records are present in secondary DNS).
> >
> > samba_dnsupdate --verbose works fine on secondary DC while primary is on
> > but if I remove from resolv.conf the address of the primary DC/DNS and
> > leave just the address of the secondary DC/DNS it (takes a long time)
> > says that all records are missing and when it tries to auth to krb it
> > fails (again no _kerberos.udp... record present).
> >
> > I tried to add these records by hand to see if something goes better but
> > if I try to add records on secondary DC, samba-tool fails always saying:
> > [root@kdc02:/usr/local/samba/private]# samba-tool dns add kdc02
> > saitelitalia.local kdc01 A 192.168.12.5 -U administrator
> > ...
> > ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
> >   File
> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
> > line 160, in _run
> >     return self.run(*args, **kwargs)
> >   File
> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
> > 1055, in run
> >     None)
> >
> > while it works fine on primary.
> >
> > I'm a little bit confused by the error message because
> > WERR_INTERNAL_DB_ERROR seems to be related to an error in adding the
> > record to the DB but in line 1055 of .../samba/netcmd/dns.py it seems
> > that the problem is related to some missing/wrong argument to the update
> > record call.
> >
> > Am I doing something wrong?
> >
> > I'll be happy to contribute but need to be addressed how.
> >
> > Thanks,
> > Daniele.
> >
>
> After some other tries, I've seen that an update (or for linux boxes
> with fixed addresses a delete+add) of records on the zones of the
> primary DC/DNS, records have appeared also on secondary DC/DNS.
>
> Next step I'll try to stop primary DC/DNS to see if secondary keeps the
> domain up.
>
> Daniele.
>

Just an update to my tests:
     1. changed order of nameserver in resolv.conf of secondary DC
        (first will be secondary DC itself)
     2. run samba_nsupdate --verbose to see which records were missing
        on secondary DC
     3. updated all records on primary DC/DNS (using W2k3 tools from a
        joined WXP box)

After that, all records appeared on zones of the secondary DC/DNS and
samba_dnsupdate works successfully (because nsupdate does not have to
update any record).

Any update performed on secondary DC (with nsupdate and/or with
samba-tool dns add) will fail.

samba-tool dns add kdc02 12.168.192.in-addr.arpa 220 PTR
alaska.saitelitalia.local. -U administrator
GENSEC backend 'gssapi_spnego' registered
GENSEC backend 'gssapi_krb5' registered
GENSEC backend 'gssapi_krb5_sasl' registered
GENSEC backend 'schannel' registered
GENSEC backend 'spnego' registered
GENSEC backend 'ntlmssp' registered
GENSEC backend 'krb5' registered
GENSEC backend 'fake_gssapi_krb5' registered
Using binding ncacn_ip_tcp:kdc02[,sign]
Password for [SAITELITALIA\administrator]:
ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
  File
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
line 160, in _run
    return self.run(*args, **kwargs)
  File
"/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
1055, in run
    None)

Using nsupdate -g just tells SERVFAIL but looking in named log I've
found

database: error: samba_dlz: failed to modify
DC=@,DC=12.168.192.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=saitelitalia,DC=local - cannot change replicated attribute on partial replica at ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:1400

So while for "DC=saitelitalia,DC=local"
"CN=Configuration,DC=saitelitalia,DC=local" and
"CN=Schema,CN=Configuration,DC=saitelitalia,DC=local" partitions I have
full replica and I'm able to add users groups (I'll try to join a box)
on secondary DC and the changes are replicated to primary as well as
doing it on primary, DNS partitions appear to be a partial replica and
only primary will be able to modify them.

BTW, once samba_dnsupdate --verbose finds all required records on
secondary DC/DNS I stopped primary DC/DNS and users have been able to
login to the domain, see network shares so krb, ldap and other stuff
worked fine.

Hope these tests help to solve the problem.

Regards,
Daniele

[Amitay Isaacs]

Hi Daniele,

Thanks for testing the secondary DNS server set up. Most of the
secondary DNS server issues arise due to the problems of replication
(more details below). The solution is to fix the replication problems
for application partitions. Unfortunately, I have not been able to
spend much time on DNS/Replication work lately.

Amitay.

On Tue, May 8, 2012 at 5:48 PM, Daniele Dario <[hidden email]> wrote:

> On Mon, 2012-05-07 at 16:46 +0200, Daniele Dario wrote:
>> On Mon, 2012-05-07 at 11:42 +0200, Daniele Dario wrote:
>> > Hi samba team,
>> > I've some problems with the dns of the secondary DC.
>> >
>> > I have 2 samba4 DCs: kdc01 and kdc02 (respectively Version
>> > 4.0.0alpha21-GIT-7b55ec2 and Version 4.0.0alpha21-GIT-8026550).
>> > I have successfully joined the secondary DC and replication seems to be
>> > working fine.
>> >
>> > As said in another thread I see that replication between DNS zones is
>> > not full:
>> >
>> > [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc01
>> > _msdcs.saitelitalia.local @ ALL -U administrator
>> > ...
>> >   Name=, Records=2, Children=0
>> >     NS: kdc01.saitelitalia.local. (flags=600000f0, serial=1, ttl=900)
>> >     SOA: serial=147, refresh=900, retry=600, expire=86400,
>> > ns=kdc01.saitelitalia.local., email=hostmaster.saitelitalia.local.
>> > (flags=600000f0, serial=146, ttl=3600)
>> >   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=1, Children=0
>> >     CNAME: kdc02.saitelitalia.local. (flags=f0, serial=284, ttl=900)
>> >   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=1, Children=0
>> >     CNAME: kdc01.saitelitalia.local. (flags=f0, serial=1, ttl=900)
>> >   Name=dc, Records=0, Children=2
>> >   Name=domains, Records=0, Children=1
>> >   Name=gc, Records=0, Children=2
>> >   Name=kdc01, Records=1, Children=0
>> >     NS: 192.168.12.5. (flags=f0, serial=62, ttl=900)
>> >   Name=pdc, Records=0, Children=1
>> >
>> > [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc02
>> > _msdcs.saitelitalia.local @ ALL -U administrator
>> > ...
>> >   Name=, Records=0, Children=0
>> >   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=0, Children=0
>> >   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=0, Children=0
>> >   Name=dc, Records=0, Children=2
>> >   Name=domains, Records=0, Children=1
>> >   Name=gc, Records=0, Children=2
>> >   Name=kdc01, Records=0, Children=0
>> >   Name=pdc, Records=0, Children=1
>> >
>> > If I shutdown kdc01, kdc02 is not able to keep things working (no _ldap,
>> > _kerberos and other records are present in secondary DNS).
>> >
>> > samba_dnsupdate --verbose works fine on secondary DC while primary is on
>> > but if I remove from resolv.conf the address of the primary DC/DNS and
>> > leave just the address of the secondary DC/DNS it (takes a long time)
>> > says that all records are missing and when it tries to auth to krb it
>> > fails (again no _kerberos.udp... record present).
>> >
>> > I tried to add these records by hand to see if something goes better but
>> > if I try to add records on secondary DC, samba-tool fails always saying:
>> > [root@kdc02:/usr/local/samba/private]# samba-tool dns add kdc02
>> > saitelitalia.local kdc01 A 192.168.12.5 -U administrator
>> > ...
>> > ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
>> >   File
>> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
>> > line 160, in _run
>> >     return self.run(*args, **kwargs)
>> >   File
>> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
>> > 1055, in run
>> >     None)
>> >
>> > while it works fine on primary.
>> >
>> > I'm a little bit confused by the error message because
>> > WERR_INTERNAL_DB_ERROR seems to be related to an error in adding the
>> > record to the DB but in line 1055 of .../samba/netcmd/dns.py it seems
>> > that the problem is related to some missing/wrong argument to the update
>> > record call.
>> >
>> > Am I doing something wrong?
>> >
>> > I'll be happy to contribute but need to be addressed how.
>> >
>> > Thanks,
>> > Daniele.
>> >
>>
>> After some other tries, I've seen that an update (or for linux boxes
>> with fixed addresses a delete+add) of records on the zones of the
>> primary DC/DNS, records have appeared also on secondary DC/DNS.
>>
>> Next step I'll try to stop primary DC/DNS to see if secondary keeps the
>> domain up.
>>
>> Daniele.
>>
> Just an update to my tests:
>     1. changed order of nameserver in resolv.conf of secondary DC
>        (first will be secondary DC itself)
>     2. run samba_nsupdate --verbose to see which records were missing
>        on secondary DC
>     3. updated all records on primary DC/DNS (using W2k3 tools from a
>        joined WXP box)
>
> After that, all records appeared on zones of the secondary DC/DNS and
> samba_dnsupdate works successfully (because nsupdate does not have to
> update any record).
>
> Any update performed on secondary DC (with nsupdate and/or with
> samba-tool dns add) will fail.
>
> samba-tool dns add kdc02 12.168.192.in-addr.arpa 220 PTR
> alaska.saitelitalia.local. -U administrator
> GENSEC backend 'gssapi_spnego' registered
> GENSEC backend 'gssapi_krb5' registered
> GENSEC backend 'gssapi_krb5_sasl' registered
> GENSEC backend 'schannel' registered
> GENSEC backend 'spnego' registered
> GENSEC backend 'ntlmssp' registered
> GENSEC backend 'krb5' registered
> GENSEC backend 'fake_gssapi_krb5' registered
> Using binding ncacn_ip_tcp:kdc02[,sign]
> Password for [SAITELITALIA\administrator]:
> ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
>  File
> "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
> line 160, in _run
>    return self.run(*args, **kwargs)
>  File
> "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
> 1055, in run
>    None)
>
> Using nsupdate -g just tells SERVFAIL but looking in named log I've
> found
>
> database: error: samba_dlz: failed to modify
> DC=@,DC=12.168.192.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=saitelitalia,DC=local - cannot change replicated attribute on partial replica at ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:1400
>
> So while for "DC=saitelitalia,DC=local"
> "CN=Configuration,DC=saitelitalia,DC=local" and
> "CN=Schema,CN=Configuration,DC=saitelitalia,DC=local" partitions I have
> full replica and I'm able to add users groups (I'll try to join a box)
> on secondary DC and the changes are replicated to primary as well as
> doing it on primary, DNS partitions appear to be a partial replica and
> only primary will be able to modify them.

The main cause for this is the replication not working *correctly* for
DNS partitions. May be a replication expert can comment on why there
is an issue with replicating application partitions. The interim fix
is to *disallow* any updates if we don't have a full Replica of DNS
partitions. At least that will report correct information to the user.

> BTW, once samba_dnsupdate --verbose finds all required records on
> secondary DC/DNS I stopped primary DC/DNS and users have been able to
> login to the domain, see network shares so krb, ldap and other stuff
> worked fine.
>
> Hope these tests help to solve the problem.
>
> Regards,
> Daniele
>
>

[danicat]

Hi Amitay, samba-team,
rising debug level to 5 to see more detail about replication this is
what I see every time replication starts from primary to secondary:

[2012/05/10 08:35:39,
5] ../source4/dsdb/kcc/kcc_periodic.c:107(check_MasterNC)
  bdbaecef-ace9-4314-b65e-54933ac8b660._msdcs.saitelitalia.local
msDS-hasMasterNCs match on DC=ForestDnsZones,DC=saitelitalia,DC=loc` in
CN=NTDS
Settings,CN=KDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=saitelitalia,DC=local
[2012/05/10 08:35:39,
5] ../source4/dsdb/kcc/kcc_periodic.c:107(check_MasterNC)
  bdbaecef-ace9-4314-b65e-54933ac8b660._msdcs.saitelitalia.local
msDS-hasMasterNCs match on DC=DomainDnsZones,DC=saitelitalia,DC=loc` in
CN=NTDS
Settings,CN=KDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=saitelitalia,DC=local
[2012/05/10 08:35:39,
5] ../source4/dsdb/kcc/kcc_periodic.c:107(check_MasterNC)
  bdbaecef-ace9-4314-b65e-54933ac8b660._msdcs.saitelitalia.local
msDS-hasMasterNCs match on DC=saitelitalia,DC=localP in CN=NTDS
Settings,CN=KDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=saitelitalia,DC=local
[2012/05/10 08:35:39,
5] ../source4/dsdb/kcc/kcc_periodic.c:107(check_MasterNC)
  bdbaecef-ace9-4314-b65e-54933ac8b660._msdcs.saitelitalia.local
msDS-hasMasterNCs match on CN=Configuration,DC=saitelitalia,DC=loca` in
CN=NTDS
Settings,CN=KDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=saitelitalia,DC=local
[2012/05/10 08:35:39,
5] ../source4/dsdb/kcc/kcc_periodic.c:107(check_MasterNC)
  bdbaecef-ace9-4314-b65e-54933ac8b660._msdcs.saitelitalia.local
msDS-hasMasterNCs match on
CN=Schema,CN=Configuration,DC=saitelitalia,DC=loh in CN=NTDS
Settings,CN=KDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=saitelitalia,DC=local
[2012/05/10 08:35:39,
4] ../source4/dsdb/kcc/kcc_periodic.c:584(kccsrv_periodic_schedule)
  kccsrv_periodic_schedule(300) scheduled for: Thu May 10 08:40:39 2012
CEST
[2012/05/10 08:35:39,  5] ../lib/ldb-samba/ldb_wrap.c:68(ldb_wrap_debug)
  ldb: start ldb transaction (nesting: 0)
[2012/05/10 08:35:39,  5] ../lib/ldb-samba/ldb_wrap.c:68(ldb_wrap_debug)
  ldb: replmd_extended_replicated_objects
 
[2012/05/10 08:35:39,
4] ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4498(replmd_extended_replicated_objects)
  linked_attributes_count=0
[2012/05/10 08:35:39,
4] ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4332(replmd_replicated_uptodate_modify)
  DRS replication uptodate modify message:
  dn: DC=DomainDnsZones,DC=saitelitalia,DC=local
  changetype: modify
  replace: replUpToDateVector
  replUpToDateVector::
AgAAAAAAAAAHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD8IgAAAAAAAAD1o
   av8KM0BjF
+uErseOEyULwvIWhMvRqISAAAAAAAAIvpcRwEBzQFUyo8f7kptRKEIU32z0VGPQw4AAA

AAAADEhxfdL8bMAZ2UcWixCGNNkfCjHamJKDKVDgAAAAAAABjneCF618wBH7KLeMjtfUaJz/ZrZ4Q
   M4QwlAAAAAAAAgO86HXcuzQE7cFmbOdfHTYPv5V5dB2of7RIAAAAAAAAAOnGIVyjNAdai
+csVAUZP
   pOgdhB8vfT7BEQAAAAAAADKcQ82IGc0B
  -
  replace: repsFrom
  repsFrom::
AQAAAAAAAAATAQAAAAAAAI/xuwUDAAAAu/K7BQMAAAAAAAAA0AAAAEMAAAB0AAAAERE

RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER

ERERERERERERERERERERERERERERERERAAAAAAwlAAAAAAAAAAAAAAAAAAAMJQAAAAAAAO/sur3pr

BRDtl5UkzrItmAfsot4yO19RonP9mtnhAzhAAAAAAAAAAAAAAAAAAAAAD8AAABiZGJhZWNlZi1hY2

U5LTQzMTQtYjY1ZS01NDkzM2FjOGI2NjAuX21zZGNzLnNhaXRlbGl0YWxpYS5sb2NhbAA=
  -
 
 
[2012/05/10 08:35:39,  5] ../lib/ldb-samba/ldb_wrap.c:68(ldb_wrap_debug)
  ldb: commit ldb transaction (nesting: 0)
[2012/05/10 08:35:39,
2] ../source4/dsdb/repl/replicated_objects.c:637(dsdb_replicated_objects_commit)
  Replicated 0 objects (0 linked attributes) for
DC=DomainDnsZones,DC=saitelitalia,DC=local
[2012/05/10 08:35:39,
4] ../source4/dsdb/repl/drepl_out_helpers.c:856(dreplsrv_update_refs_done)
  UpdateRefs OK for
06f11708-b11c-4848-879d-565d72adfaf3._msdcs.saitelitalia.local
DC=DomainDnsZones,DC=saitelitalia,DC=local
[2012/05/10 08:35:39,
4] ../source4/dsdb/repl/drepl_out_pull.c:160(dreplsrv_pending_op_callback)
  dreplsrv_op_pull_source(WERR_OK) for
DC=DomainDnsZones,DC=saitelitalia,DC=local

Don't know why messages
from ../source4/dsdb/kcc/kcc_periodic.c:107(check_MasterNC) looks not
correctly displayed.

BTW, what seems wrong to me is that every
time ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4332(replmd_replicated_uptodate_modify) is called it seems to modify the same record.
After a call I suppose the record should be updated but on next call it
seems the record has not been updated.

Am I wrong?

This happens also for other partitions/zones:
DC=saitelitalia,DC=local:
      * [2012/05/10 08:40:40,
4] ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4332(replmd_replicated_uptodate_modify)
          DRS replication uptodate modify message:
          dn: DC=saitelitalia,DC=local
          changetype: modify
          replace: replUpToDateVector
          replUpToDateVector::
AgAAAAAAAAAPAAAAAAAAAIxfrhK7HjhMlC8LyFoTL0bHEgAAAAAAAACIc
           J6zGM0BVMqPH+5KbUShCFN9s9FRj1kOAAAAAAAAAD8lyQ/HzAFW4mwl
+54QS4IUrdAe0X2Ssw4AAA

AAAACAlV7/cRnNAfSdqCa4nL5Hv59a4rdKWJmqDgAAAAAAAIB/ySgZ4MwBLsD5M+nZ
+EieoE5r7Xw
           V0qYOAAAAAAAAgGxrG
+gbzQFQL3FPeeBKQ7rK9nyHsAaKyA4AAAAAAACANtLqZyjNAXUXaVu4yMFA

oXacOxP0EQyRDgAAAAAAAAA4r2y3G80BnZRxaLEIY02R8KMdqYkoMhkOAAAAAAAAgOs0IXrXzAEww
           xtwn+G8TZwsCpm3Xw+Nvw4AAAAAAACAT/yDXijNAR
+yi3jI7X1Gic/2a2eEDOHgJAAAAAAAAADko9
           B3Ls0BbfQcezd1x0KscfKsrty/hIsOAAAAAAAAADLg7qAbzQH8L
+WOUjz/TroDHEPjM9QEYQ4AAAA
           AAACAQknqY/DMATtwWZs518dNg
+/lXl0Hah/eEgAAAAAAAAA6cYhXKM0B1qL5yxUBRk+k6B2EHy99

Pn0OAAAAAAAAgEU5K4kZzQGwoB3on6LOSqyDhZYDb5fgeA4AAAAAAAAAV9KedxnNAQ==
          -
          replace: repsFrom
          repsFrom::
AQAAAAAAAAATAQAAAAAAALzyuwUDAAAA5/O7BQMAAAAAAAAA0AAAAEMAAAB0AAAAERE

RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER

ERERERERERERERERERERERERERERERERAAAAAOAkAAAAAAAAAAAAAAAAAADgJAAAAAAAAO/sur3pr

BRDtl5UkzrItmAfsot4yO19RonP9mtnhAzhAAAAAAAAAAAAAAAAAAAAAD8AAABiZGJhZWNlZi1hY2

U5LTQzMTQtYjY1ZS01NDkzM2FjOGI2NjAuX21zZGNzLnNhaXRlbGl0YWxpYS5sb2NhbAA=
          -
        CN=Schema,CN=Configuration,DC=saitelitalia,DC=local
      * [2012/05/10 08:40:40,
4] ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4332(replmd_replicated_uptodate_modify)
          DRS replication uptodate modify message:
          dn: CN=Schema,CN=Configuration,DC=saitelitalia,DC=local
          changetype: modify
          replace: replUpToDateVector
          replUpToDateVector::
AgAAAAAAAAAPAAAAAAAAAIxfrhK7HjhMlC8LyFoTL0YVBgAAAAAAAIAeC
           Z+zGM0BVMqPH+5KbUShCFN9s9FRjxUGAAAAAAAAAD8lyQ/HzAFW4mwl
+54QS4IUrdAe0X2SFQYAAA

AAAACAlV7/cRnNAfSdqCa4nL5Hv59a4rdKWJkVBgAAAAAAAIB/ySgZ4MwBLsD5M+nZ
+EieoE5r7Xw

V0hUGAAAAAAAAAAMEHOgbzQFQL3FPeeBKQ7rK9nyHsAaKFQYAAAAAAAAAzWrrZyjNAXUXaVu4yMFA

oXacOxP0EQwVBgAAAAAAAIDOR223G80BnZRxaLEIY02R8KMdqYkoMhUGAAAAAAAAAILNIXrXzAEww
           xtwn+G8TZwsCpm3Xw+NFQYAAAAAAAAA5pSEXijNAR
+yi3jI7X1Gic/2a2eEDOFoDgAAAAAAAADko9
           B3Ls0BbfQcezd1x0KscfKsrty/hBUGAAAAAAAAgMh476AbzQH8L
+WOUjz/TroDHEPjM9QEFQYAAAA
           AAACAQknqY/DMATtwWZs518dNg
+/lXl0Hah8VBgAAAAAAAIDQCYlXKM0B1qL5yxUBRk+k6B2EHy99

PhUGAAAAAAAAANzRK4kZzQGwoB3on6LOSqyDhZYDb5fgFQYAAAAAAACA7WqfdxnNAQ==
          -
          replace: repsFrom
          repsFrom::
AQAAAAAAAAATAQAAAAAAALzyuwUDAAAA6PO7BQMAAAAAAAAA0AAAAEMAAAB0AAAAERE

RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER

ERERERERERERERERERERERERERERERERAAAAAGgOAAAAAAAAAAAAAAAAAABoDgAAAAAAAO/sur3pr

BRDtl5UkzrItmAfsot4yO19RonP9mtnhAzhAAAAAAAAAAAAAAAAAAAAAD8AAABiZGJhZWNlZi1hY2

U5LTQzMTQtYjY1ZS01NDkzM2FjOGI2NjAuX21zZGNzLnNhaXRlbGl0YWxpYS5sb2NhbAA=
          -
          CN=Configuration,DC=saitelitalia,DC=local
      * [2012/05/10 08:40:41,
4] ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4332(replmd_replicated_uptodate_modify)
          DRS replication uptodate modify message:
          dn: CN=Configuration,DC=saitelitalia,DC=local
          changetype: modify
          replace: replUpToDateVector
          replUpToDateVector::
AgAAAAAAAAAPAAAAAAAAAIxfrhK7HjhMlC8LyFoTL0bOEgAAAAAAAAC1o
           Z+zGM0BVMqPH+5KbUShCFN9s9FRj1oOAAAAAAAAgNW9yQ/HzAFW4mwl
+54QS4IUrdAe0X2Stg4AAA

AAAAAALPf/cRnNAfSdqCa4nL5Hv59a4rdKWJmnDgAAAAAAAAAWYikZ4MwBLsD5M+nZ
+EieoE5r7Xw

V0jMSAAAAAAAAgJmcHOgbzQFQL3FPeeBKQ7rK9nyHsAaKyQ4AAAAAAAAAzWrrZyjNAXUXaVu4yMFA

oXacOxP0EQyQDgAAAAAAAABl4G23G80BnZRxaLEIY02R8KMdqYkoMg0OAAAAAAAAAILNIXrXzAEww
           xtwn+G8TZwsCpm3Xw+Nwg4AAAAAAACAfC2FXijNAR
+yi3jI7X1Gic/2a2eEDOGVIwAAAAAAAIB6PN
           F3Ls0BbfQcezd1x0KscfKsrty/hIwOAAAAAAAAAF8R8KAbzQH8L
+WOUjz/TroDHEPjM9QEYg4AAAA
           AAAAA2eHqY/DMATtwWZs518dNg
+/lXl0Hah/uEgAAAAAAAABnoolXKM0B1qL5yxUBRk+k6B2EHy99

PvsRAAAAAAAAANzRK4kZzQGwoB3on6LOSqyDhZYDb5fgew4AAAAAAACA7WqfdxnNAQ==
          -
          replace: repsFrom
          repsFrom::
AQAAAAAAAAATAQAAAAAAAL3yuwUDAAAA6fO7BQMAAAAAAAAA0AAAAEMAAAB0AAAAERE

RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER

ERERERERERERERERERERERERERERERERAAAAAJUjAAAAAAAAAAAAAAAAAACVIwAAAAAAAO/sur3pr

BRDtl5UkzrItmAfsot4yO19RonP9mtnhAzhAAAAAAAAAAAAAAAAAAAAAD8AAABiZGJhZWNlZi1hY2

U5LTQzMTQtYjY1ZS01NDkzM2FjOGI2NjAuX21zZGNzLnNhaXRlbGl0YWxpYS5sb2NhbAA=
          -
        DC=ForestDnsZones,DC=saitelitalia,DC=local
      * [2012/05/10 08:40:38,
4] ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:4332(replmd_replicated_uptodate_modify)
          DRS replication uptodate modify message:
          dn: DC=ForestDnsZones,DC=saitelitalia,DC=local
          changetype: modify
          replace: replUpToDateVector
          replUpToDateVector::
AgAAAAAAAAAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJIgAAAAAAAIBeC
           av8KM0BjF
+uErseOEyULwvIWhMvRm4RAAAAAAAAdoPiTQEBzQFUyo8f7kptRKEIU32z0VGPVg4AAA

AAAACmj8FMLMbMAXUXaVu4yMFAoXacOxP0EQwMEgAAAAAAADAtB223G80BH7KLeMjtfUaJz/ZrZ4Q

M4WMkAAAAAAAAALdyz3cuzQE7cFmbOdfHTYPv5V5dB2of9g4AAAAAAACAo9iHVyjNAQ==
          -
          replace: repsFrom
          repsFrom::
AQAAAAAAAAATAQAAAAAAALvyuwUDAAAA5vO7BQMAAAAAAAAA0AAAAEMAAAB0AAAAERE

RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER

ERERERERERERERERERERERERERERERERAAAAAGMkAAAAAAAAAAAAAAAAAABjJAAAAAAAAO/sur3pr

BRDtl5UkzrItmAfsot4yO19RonP9mtnhAzhAAAAAAAAAAAAAAAAAAAAAD8AAABiZGJhZWNlZi1hY2

U5LTQzMTQtYjY1ZS01NDkzM2FjOGI2NjAuX21zZGNzLnNhaXRlbGl0YWxpYS5sb2NhbAA=
          -
       
Thanks,
Daniele


On Tue, 2012-05-08 at 19:00 +1000, Amitay Isaacs wrote:

> Hi Daniele,
>
> Thanks for testing the secondary DNS server set up. Most of the
> secondary DNS server issues arise due to the problems of replication
> (more details below). The solution is to fix the replication problems
> for application partitions. Unfortunately, I have not been able to
> spend much time on DNS/Replication work lately.
>
> Amitay.
>
> On Tue, May 8, 2012 at 5:48 PM, Daniele Dario <[hidden email]> wrote:
> > On Mon, 2012-05-07 at 16:46 +0200, Daniele Dario wrote:
> >> On Mon, 2012-05-07 at 11:42 +0200, Daniele Dario wrote:
> >> > Hi samba team,
> >> > I've some problems with the dns of the secondary DC.
> >> >
> >> > I have 2 samba4 DCs: kdc01 and kdc02 (respectively Version
> >> > 4.0.0alpha21-GIT-7b55ec2 and Version 4.0.0alpha21-GIT-8026550).
> >> > I have successfully joined the secondary DC and replication seems to be
> >> > working fine.
> >> >
> >> > As said in another thread I see that replication between DNS zones is
> >> > not full:
> >> >
> >> > [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc01
> >> > _msdcs.saitelitalia.local @ ALL -U administrator
> >> > ...
> >> >   Name=, Records=2, Children=0
> >> >     NS: kdc01.saitelitalia.local. (flags=600000f0, serial=1, ttl=900)
> >> >     SOA: serial=147, refresh=900, retry=600, expire=86400,
> >> > ns=kdc01.saitelitalia.local., email=hostmaster.saitelitalia.local.
> >> > (flags=600000f0, serial=146, ttl=3600)
> >> >   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=1, Children=0
> >> >     CNAME: kdc02.saitelitalia.local. (flags=f0, serial=284, ttl=900)
> >> >   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=1, Children=0
> >> >     CNAME: kdc01.saitelitalia.local. (flags=f0, serial=1, ttl=900)
> >> >   Name=dc, Records=0, Children=2
> >> >   Name=domains, Records=0, Children=1
> >> >   Name=gc, Records=0, Children=2
> >> >   Name=kdc01, Records=1, Children=0
> >> >     NS: 192.168.12.5. (flags=f0, serial=62, ttl=900)
> >> >   Name=pdc, Records=0, Children=1
> >> >
> >> > [root@kdc02:/usr/local/samba/private]# samba-tool dns query kdc02
> >> > _msdcs.saitelitalia.local @ ALL -U administrator
> >> > ...
> >> >   Name=, Records=0, Children=0
> >> >   Name=06f11708-b11c-4848-879d-565d72adfaf3, Records=0, Children=0
> >> >   Name=bdbaecef-ace9-4314-b65e-54933ac8b660, Records=0, Children=0
> >> >   Name=dc, Records=0, Children=2
> >> >   Name=domains, Records=0, Children=1
> >> >   Name=gc, Records=0, Children=2
> >> >   Name=kdc01, Records=0, Children=0
> >> >   Name=pdc, Records=0, Children=1
> >> >
> >> > If I shutdown kdc01, kdc02 is not able to keep things working (no _ldap,
> >> > _kerberos and other records are present in secondary DNS).
> >> >
> >> > samba_dnsupdate --verbose works fine on secondary DC while primary is on
> >> > but if I remove from resolv.conf the address of the primary DC/DNS and
> >> > leave just the address of the secondary DC/DNS it (takes a long time)
> >> > says that all records are missing and when it tries to auth to krb it
> >> > fails (again no _kerberos.udp... record present).
> >> >
> >> > I tried to add these records by hand to see if something goes better but
> >> > if I try to add records on secondary DC, samba-tool fails always saying:
> >> > [root@kdc02:/usr/local/samba/private]# samba-tool dns add kdc02
> >> > saitelitalia.local kdc01 A 192.168.12.5 -U administrator
> >> > ...
> >> > ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
> >> >   File
> >> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
> >> > line 160, in _run
> >> >     return self.run(*args, **kwargs)
> >> >   File
> >> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
> >> > 1055, in run
> >> >     None)
> >> >
> >> > while it works fine on primary.
> >> >
> >> > I'm a little bit confused by the error message because
> >> > WERR_INTERNAL_DB_ERROR seems to be related to an error in adding the
> >> > record to the DB but in line 1055 of .../samba/netcmd/dns.py it seems
> >> > that the problem is related to some missing/wrong argument to the update
> >> > record call.
> >> >
> >> > Am I doing something wrong?
> >> >
> >> > I'll be happy to contribute but need to be addressed how.
> >> >
> >> > Thanks,
> >> > Daniele.
> >> >
> >>
> >> After some other tries, I've seen that an update (or for linux boxes
> >> with fixed addresses a delete+add) of records on the zones of the
> >> primary DC/DNS, records have appeared also on secondary DC/DNS.
> >>
> >> Next step I'll try to stop primary DC/DNS to see if secondary keeps the
> >> domain up.
> >>
> >> Daniele.
> >>
> > Just an update to my tests:
> >     1. changed order of nameserver in resolv.conf of secondary DC
> >        (first will be secondary DC itself)
> >     2. run samba_nsupdate --verbose to see which records were missing
> >        on secondary DC
> >     3. updated all records on primary DC/DNS (using W2k3 tools from a
> >        joined WXP box)
> >
> > After that, all records appeared on zones of the secondary DC/DNS and
> > samba_dnsupdate works successfully (because nsupdate does not have to
> > update any record).
> >
> > Any update performed on secondary DC (with nsupdate and/or with
> > samba-tool dns add) will fail.
> >
> > samba-tool dns add kdc02 12.168.192.in-addr.arpa 220 PTR
> > alaska.saitelitalia.local. -U administrator
> > GENSEC backend 'gssapi_spnego' registered
> > GENSEC backend 'gssapi_krb5' registered
> > GENSEC backend 'gssapi_krb5_sasl' registered
> > GENSEC backend 'schannel' registered
> > GENSEC backend 'spnego' registered
> > GENSEC backend 'ntlmssp' registered
> > GENSEC backend 'krb5' registered
> > GENSEC backend 'fake_gssapi_krb5' registered
> > Using binding ncacn_ip_tcp:kdc02[,sign]
> > Password for [SAITELITALIA\administrator]:
> > ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
> >  File
> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
> > line 160, in _run
> >    return self.run(*args, **kwargs)
> >  File
> > "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/dns.py", line
> > 1055, in run
> >    None)
> >
> > Using nsupdate -g just tells SERVFAIL but looking in named log I've
> > found
> >
> > database: error: samba_dlz: failed to modify
> > DC=@,DC=12.168.192.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=saitelitalia,DC=local - cannot change replicated attribute on partial replica at ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:1400
> >
> > So while for "DC=saitelitalia,DC=local"
> > "CN=Configuration,DC=saitelitalia,DC=local" and
> > "CN=Schema,CN=Configuration,DC=saitelitalia,DC=local" partitions I have
> > full replica and I'm able to add users groups (I'll try to join a box)
> > on secondary DC and the changes are replicated to primary as well as
> > doing it on primary, DNS partitions appear to be a partial replica and
> > only primary will be able to modify them.
>
> The main cause for this is the replication not working *correctly* for
> DNS partitions. May be a replication expert can comment on why there
> is an issue with replicating application partitions. The interim fix
> is to *disallow* any updates if we don't have a full Replica of DNS
> partitions. At least that will report correct information to the user.
>
> > BTW, once samba_dnsupdate --verbose finds all required records on
> > secondary DC/DNS I stopped primary DC/DNS and users have been able to
> > login to the domain, see network shares so krb, ldap and other stuff
> > worked fine.
> >
> > Hope these tests help to solve the problem.
> >
> > Regards,
> > Daniele
> >
> >