Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Richard Sharpe-2
Hi folks,

We are intermittently seeing NTLM auth failing with
NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:

[2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
0), class=winbind]
../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
  winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
Maybe the trust account password was changed and we didn't know it.
Killing connections to domain SOMEDOM

Now, the real reason seems to be that one of the DCs in that domain
disallows NTLM authentication and whenever winbindd finds that DC we
get this problem.

Is there some way to tell Windindd not to use that DC?

Also, I notice that in some instances in winbind_samlogon_retry_loop
we move to another DC but not in this case. We simply retry with the
same DC.

I suspect that we should move to another DC in this case as well.

Any comments?

Also, perhaps we should retry with as many DCs as we can find?

--
Regards,
Richard Sharpe
(何以解憂?唯有杜康。--曹操)

Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Jeremy Allison
On Thu, Oct 08, 2015 at 04:11:19PM -0700, Richard Sharpe wrote:

> Hi folks,
>
> We are intermittently seeing NTLM auth failing with
> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>
> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
> 0), class=winbind]
> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
> Maybe the trust account password was changed and we didn't know it.
> Killing connections to domain SOMEDOM
>
> Now, the real reason seems to be that one of the DCs in that domain
> disallows NTLM authentication and whenever winbindd finds that DC we
> get this problem.
>
> Is there some way to tell Windindd not to use that DC?
>
> Also, I notice that in some instances in winbind_samlogon_retry_loop
> we move to another DC but not in this case. We simply retry with the
> same DC.
>
> I suspect that we should move to another DC in this case as well.
>
> Any comments?

Yep - getting  ACCESS_DENIED should certainly trigger adding
the DC to the negative connection cache.

Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Stefan Metzmacher-2
Am 09.10.2015 um 01:19 schrieb Jeremy Allison:

> On Thu, Oct 08, 2015 at 04:11:19PM -0700, Richard Sharpe wrote:
>> Hi folks,
>>
>> We are intermittently seeing NTLM auth failing with
>> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>>
>> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
>> 0), class=winbind]
>> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
>> Maybe the trust account password was changed and we didn't know it.
>> Killing connections to domain SOMEDOM
>>
>> Now, the real reason seems to be that one of the DCs in that domain
>> disallows NTLM authentication and whenever winbindd finds that DC we
>> get this problem.
>>
>> Is there some way to tell Windindd not to use that DC?
>>
>> Also, I notice that in some instances in winbind_samlogon_retry_loop
>> we move to another DC but not in this case. We simply retry with the
>> same DC.
>>
>> I suspect that we should move to another DC in this case as well.
>>
>> Any comments?
>
> Yep - getting  ACCESS_DENIED should certainly trigger adding
> the DC to the negative connection cache.
But not an the first failure!

BTW: which Samba version are you using?

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Richard Sharpe-2
On Thu, Oct 8, 2015 at 9:22 PM, Stefan Metzmacher <[hidden email]> wrote:

> Am 09.10.2015 um 01:19 schrieb Jeremy Allison:
>> On Thu, Oct 08, 2015 at 04:11:19PM -0700, Richard Sharpe wrote:
>>> Hi folks,
>>>
>>> We are intermittently seeing NTLM auth failing with
>>> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>>>
>>> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
>>> 0), class=winbind]
>>> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>>>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
>>> Maybe the trust account password was changed and we didn't know it.
>>> Killing connections to domain SOMEDOM
>>>
>>> Now, the real reason seems to be that one of the DCs in that domain
>>> disallows NTLM authentication and whenever winbindd finds that DC we
>>> get this problem.
>>>
>>> Is there some way to tell Windindd not to use that DC?
>>>
>>> Also, I notice that in some instances in winbind_samlogon_retry_loop
>>> we move to another DC but not in this case. We simply retry with the
>>> same DC.
>>>
>>> I suspect that we should move to another DC in this case as well.
>>>
>>> Any comments?
>>
>> Yep - getting  ACCESS_DENIED should certainly trigger adding
>> the DC to the negative connection cache.
>
> But not an the first failure!

Hmmm, why not. If it is returning ACCESS_DENIED either someone has
changed the machine account password without telling us or that DC
does not like NTLM passthrough ...

> BTW: which Samba version are you using?

4.3.0-- and master

--
Regards,
Richard Sharpe
(何以解憂?唯有杜康。--曹操)

Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Richard Sharpe-2
On Thu, Oct 8, 2015 at 9:35 PM, Richard Sharpe
<[hidden email]> wrote:

> On Thu, Oct 8, 2015 at 9:22 PM, Stefan Metzmacher <[hidden email]> wrote:
>> Am 09.10.2015 um 01:19 schrieb Jeremy Allison:
>>> On Thu, Oct 08, 2015 at 04:11:19PM -0700, Richard Sharpe wrote:
>>>> Hi folks,
>>>>
>>>> We are intermittently seeing NTLM auth failing with
>>>> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>>>>
>>>> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
>>>> 0), class=winbind]
>>>> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>>>>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
>>>> Maybe the trust account password was changed and we didn't know it.
>>>> Killing connections to domain SOMEDOM
>>>>
>>>> Now, the real reason seems to be that one of the DCs in that domain
>>>> disallows NTLM authentication and whenever winbindd finds that DC we
>>>> get this problem.
>>>>
>>>> Is there some way to tell Windindd not to use that DC?
>>>>
>>>> Also, I notice that in some instances in winbind_samlogon_retry_loop
>>>> we move to another DC but not in this case. We simply retry with the
>>>> same DC.
>>>>
>>>> I suspect that we should move to another DC in this case as well.
>>>>
>>>> Any comments?
>>>
>>> Yep - getting  ACCESS_DENIED should certainly trigger adding
>>> the DC to the negative connection cache.
>>
>> But not an the first failure!
>
> Hmmm, why not. If it is returning ACCESS_DENIED either someone has
> changed the machine account password without telling us or that DC
> does not like NTLM passthrough ...

OK, so there is a situation where we could get access denied because
the machine account password has changed. Let's say we are already
connected and someone changes it. In that case we would not want to
black-list it, but just connect to another DC.

Maybe what I need to do is to increase the retry count to three.

However the failure I was seeing seemed to occur on the retry as well,
because we found the same DC name again and connected to it.

Maybe all I need to do is deprecate that name? Perhaps remove it from
gencache ...

>> BTW: which Samba version are you using?
>
> 4.3.0-- and master
>
> --
> Regards,
> Richard Sharpe
> (何以解憂?唯有杜康。--曹操)



--
Regards,
Richard Sharpe
(何以解憂?唯有杜康。--曹操)

Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Richard Sharpe-2
On Fri, Oct 9, 2015 at 9:24 AM, Richard Sharpe
<[hidden email]> wrote:

> On Thu, Oct 8, 2015 at 9:35 PM, Richard Sharpe
> <[hidden email]> wrote:
>> On Thu, Oct 8, 2015 at 9:22 PM, Stefan Metzmacher <[hidden email]> wrote:
>>> Am 09.10.2015 um 01:19 schrieb Jeremy Allison:
>>>> On Thu, Oct 08, 2015 at 04:11:19PM -0700, Richard Sharpe wrote:
>>>>> Hi folks,
>>>>>
>>>>> We are intermittently seeing NTLM auth failing with
>>>>> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>>>>>
>>>>> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
>>>>> 0), class=winbind]
>>>>> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>>>>>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
>>>>> Maybe the trust account password was changed and we didn't know it.
>>>>> Killing connections to domain SOMEDOM
>>>>>
>>>>> Now, the real reason seems to be that one of the DCs in that domain
>>>>> disallows NTLM authentication and whenever winbindd finds that DC we
>>>>> get this problem.
>>>>>
>>>>> Is there some way to tell Windindd not to use that DC?
>>>>>
>>>>> Also, I notice that in some instances in winbind_samlogon_retry_loop
>>>>> we move to another DC but not in this case. We simply retry with the
>>>>> same DC.
>>>>>
>>>>> I suspect that we should move to another DC in this case as well.
>>>>>
>>>>> Any comments?
>>>>
>>>> Yep - getting  ACCESS_DENIED should certainly trigger adding
>>>> the DC to the negative connection cache.
>>>
>>> But not an the first failure!
>>
>> Hmmm, why not. If it is returning ACCESS_DENIED either someone has
>> changed the machine account password without telling us or that DC
>> does not like NTLM passthrough ...
>
> OK, so there is a situation where we could get access denied because
> the machine account password has changed. Let's say we are already
> connected and someone changes it. In that case we would not want to
> black-list it, but just connect to another DC.
>
> Maybe what I need to do is to increase the retry count to three.
>
> However the failure I was seeing seemed to occur on the retry as well,
> because we found the same DC name again and connected to it.
>
> Maybe all I need to do is deprecate that name? Perhaps remove it from
> gencache ...

Having stared at that code a lot now, I think the thing to do is call
saf_delete(domain->name).

This will remove affinity to that name and allow us to try a different DC.

There could still be intermittent successes if there was one server
that allowed pass through auth while the others didn't.

--
Regards,
Richard Sharpe
(何以解憂?唯有杜康。--曹操)

Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Uri Simchoni-4


On 10/09/2015 11:56 PM, Richard Sharpe wrote:

> On Fri, Oct 9, 2015 at 9:24 AM, Richard Sharpe
> <[hidden email]> wrote:
>> OK, so there is a situation where we could get access denied because
>> the machine account password has changed. Let's say we are already
>> connected and someone changes it. In that case we would not want to
>> black-list it, but just connect to another DC.
>>
>> Maybe what I need to do is to increase the retry count to three.
>>
>> However the failure I was seeing seemed to occur on the retry as well,
>> because we found the same DC name again and connected to it.
>>
>> Maybe all I need to do is deprecate that name? Perhaps remove it from
>> gencache ...
> Having stared at that code a lot now, I think the thing to do is call
> saf_delete(domain->name).
>
> This will remove affinity to that name and allow us to try a different DC.
>
> There could still be intermittent successes if there was one server
> that allowed pass through auth while the others didn't.
>
You want saf_delete(domain->alt_name) too, because that's the DNS name
and that's what counts in DNS searches.

Calling winbind_add_failed_connection_entry() will call saf_delete() for
you and also blacklist the failed DC. The blacklisting (lasting for one
minute) is what guarantees that the next attempt will not try this DC.

Once the AD setup is screwed up (inconsistent configuration between
servers) there's no possibility of a "perfect" behavior on our part -
perhaps the best we can do is make it easiest to spot - as you point out
in the log patch.


Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Stefan Metzmacher-2
In reply to this post by Richard Sharpe-2
Hi Richard,

>>>> We are intermittently seeing NTLM auth failing with
>>>> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>>>>
>>>> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
>>>> 0), class=winbind]
>>>> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>>>>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
>>>> Maybe the trust account password was changed and we didn't know it.
>>>> Killing connections to domain SOMEDOM
>>>>
>>>> Now, the real reason seems to be that one of the DCs in that domain
>>>> disallows NTLM authentication and whenever winbindd finds that DC we
>>>> get this problem.
>>>>
>>>> Is there some way to tell Windindd not to use that DC?
>>>>
>>>> Also, I notice that in some instances in winbind_samlogon_retry_loop
>>>> we move to another DC but not in this case. We simply retry with the
>>>> same DC.
>>>>
>>>> I suspect that we should move to another DC in this case as well.
>>>>
>>>> Any comments?
>>>
>>> Yep - getting  ACCESS_DENIED should certainly trigger adding
>>> the DC to the negative connection cache.
>>
>> But not an the first failure!
>
> Hmmm, why not. If it is returning ACCESS_DENIED either someone has
> changed the machine account password without telling us or that DC
> does not like NTLM passthrough ...
I'd assume that we need to distinguish between ACCESS_DENIED in response
to a netr_ServerAuthenticate*() where we could be rejected because
of a changed machine password (verify unlikely to happen) and other calls.

If other calls return ACCESS_DENIED (which can happen if the dc restarts)
we need to destroy the connection and netlogon_creds_cli.tdb entry and
reauthenticate.

The question is which request returns ACCESS_DENIED in the situation
where the DC rejects NTLM authentication.

Do we have a capture and level 10 logs?

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Stefan Metzmacher-2
Am 12.11.2015 um 14:21 schrieb Stefan Metzmacher:

> Hi Richard,
>
>>>>> We are intermittently seeing NTLM auth failing with
>>>>> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>>>>>
>>>>> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
>>>>> 0), class=winbind]
>>>>> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>>>>>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
>>>>> Maybe the trust account password was changed and we didn't know it.
>>>>> Killing connections to domain SOMEDOM
>>>>>
>>>>> Now, the real reason seems to be that one of the DCs in that domain
>>>>> disallows NTLM authentication and whenever winbindd finds that DC we
>>>>> get this problem.
>>>>>
>>>>> Is there some way to tell Windindd not to use that DC?
>>>>>
>>>>> Also, I notice that in some instances in winbind_samlogon_retry_loop
>>>>> we move to another DC but not in this case. We simply retry with the
>>>>> same DC.
>>>>>
>>>>> I suspect that we should move to another DC in this case as well.
>>>>>
>>>>> Any comments?
>>>>
>>>> Yep - getting  ACCESS_DENIED should certainly trigger adding
>>>> the DC to the negative connection cache.
>>>
>>> But not an the first failure!
>>
>> Hmmm, why not. If it is returning ACCESS_DENIED either someone has
>> changed the machine account password without telling us or that DC
>> does not like NTLM passthrough ...
>
> I'd assume that we need to distinguish between ACCESS_DENIED in response
> to a netr_ServerAuthenticate*() where we could be rejected because
> of a changed machine password (verify unlikely to happen) and other calls.
>
> If other calls return ACCESS_DENIED (which can happen if the dc restarts)
> we need to destroy the connection and netlogon_creds_cli.tdb entry and
> reauthenticate.
>
> The question is which request returns ACCESS_DENIED in the situation
> where the DC rejects NTLM authentication.
>
> Do we have a capture and level 10 logs?
[MS-APDS] and [MS-NLMP] contain STATUS_NTLM_BLOCKED, I'm wondering
why we don't get that instead of STATUS_ACCESS_DENIED...

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Richard Sharpe-2
On Sun, Nov 15, 2015 at 3:55 AM, Stefan Metzmacher <[hidden email]> wrote:

> Am 12.11.2015 um 14:21 schrieb Stefan Metzmacher:
>> Hi Richard,
>>
>>>>>> We are intermittently seeing NTLM auth failing with
>>>>>> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>>>>>>
>>>>>> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
>>>>>> 0), class=winbind]
>>>>>> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>>>>>>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
>>>>>> Maybe the trust account password was changed and we didn't know it.
>>>>>> Killing connections to domain SOMEDOM
>>>>>>
>>>>>> Now, the real reason seems to be that one of the DCs in that domain
>>>>>> disallows NTLM authentication and whenever winbindd finds that DC we
>>>>>> get this problem.
>>>>>>
>>>>>> Is there some way to tell Windindd not to use that DC?
>>>>>>
>>>>>> Also, I notice that in some instances in winbind_samlogon_retry_loop
>>>>>> we move to another DC but not in this case. We simply retry with the
>>>>>> same DC.
>>>>>>
>>>>>> I suspect that we should move to another DC in this case as well.
>>>>>>
>>>>>> Any comments?
>>>>>
>>>>> Yep - getting  ACCESS_DENIED should certainly trigger adding
>>>>> the DC to the negative connection cache.
>>>>
>>>> But not an the first failure!
>>>
>>> Hmmm, why not. If it is returning ACCESS_DENIED either someone has
>>> changed the machine account password without telling us or that DC
>>> does not like NTLM passthrough ...
>>
>> I'd assume that we need to distinguish between ACCESS_DENIED in response
>> to a netr_ServerAuthenticate*() where we could be rejected because
>> of a changed machine password (verify unlikely to happen) and other calls.
>>
>> If other calls return ACCESS_DENIED (which can happen if the dc restarts)
>> we need to destroy the connection and netlogon_creds_cli.tdb entry and
>> reauthenticate.
>>
>> The question is which request returns ACCESS_DENIED in the situation
>> where the DC rejects NTLM authentication.
>>
>> Do we have a capture and level 10 logs?
>
> [MS-APDS] and [MS-NLMP] contain STATUS_NTLM_BLOCKED, I'm wondering
> why we don't get that instead of STATUS_ACCESS_DENIED...

This is a good question.

I will have to get QA to repro the problem so I can check this ...
probably early in December is the earliest I can do it though.

--
Regards,
Richard Sharpe
(何以解憂?唯有杜康。--曹操)

Reply | Threaded
Open this post in threaded view
|

Re: Intermittent failure to authenticate using NTLM with NT_STATUS_ACCESS_DENIED

Samba - samba-technical mailing list
In reply to this post by Richard Sharpe-2
Hi Richard,

> We are intermittently seeing NTLM auth failing with
> NT_STATUS_ACCESS_DENIED and we see this message in winbindd.log:
>
> [2015/10/08 15:34:33.393987,  3, pid=3549, effective(0, 0), real(0,
> 0), class=winbind]
> ../source3/winbindd/winbindd_pam.c:1426(winbind_samlogon_retry_loop)
>   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.
> Maybe the trust account password was changed and we didn't know it.
> Killing connections to domain SOMEDOM
>
> Now, the real reason seems to be that one of the DCs in that domain
> disallows NTLM authentication and whenever winbindd finds that DC we
> get this problem.
I just come across something similar.

Are you should really sure the disabled NTLM authentication was the
reason here. As far as I remember a DC would return
NT_STATUS_NTLM_BLOCKED instead of NT_STATUS_ACCESS_DENIED in such
a situation.

See [MS-APDS] 3.1.5 Message Processing Events and Sequencing Rules:

  ...

  If NTLMServerDomainBlocked == TRUE, the NTLM server SHOULD<7> return
  STATUS_NTLM_BLOCKED to the NTLM client.

  If the DC is of the resource domain:
  * If ResourceDCBlocked == TRUE, and the NTLM server's name is not
    equal to any of the DCBlockExceptions server names, the DC SHOULD<8>
    return STATUS_NTLM_BLOCKED.

  If the DC is of the account domain:
  * If AccountDCBlocked == TRUE, the APDS server SHOULD<9> return
    STATUS_NTLM_BLOCKED.
  * If the domainControllerFunctionality attribute
    ([MS-ADTS] section 3.1.1.3.2.25) returns a value that is >= 6,
    the account is not also the NTLM server's account, and the APDS
    server determines that an authentication policy setting ([MS-KILE]
    section 3.3.5.5) applies, then:
    * If AllowedToAuthenticateTo is not NULL, an access check SHOULD<10>
      be performed to determine whether the user has the ACL granting
      ACTRL_DS_CONTROL_ACCESS ([MS-SAMR] section 2.2.1.17). If the
      access check fails, APDS MUST return
      STATUS_AUTHENTICATION_FIREWALL_FAILED.

  ...

The only situation I saw NT_STATUS_ACCESS_DENIED from
NetrLogonSamLogonEx was when the DC was installed correctly
and still had SYSVOLReady = 0. See
https://lists.samba.org/archive/cifs-protocol/2017-September/003075.html

And I think this is a situation where we should ban that DC.

Also with our current netlogon_cli_creds.tdb infrastructure
I can't see how we could ever get NT_STATUS_ACCESS_DENIED
from NetrLogonSamLogon[WithFlags]() when using the credential
chain.

metze


signature.asc (853 bytes) Download Attachment