UnauthorizedAccessException when using MSDTC

Hello everyone,

We have a 3-tier application, the client, the server and the databaser server. It was developed using C# 2.0 and .Net 2.0, the database server is MS-SQL 2000 and we have some MSMQ in there. All 3 are located on different machines. The client is Windows XP SP-2, the application server is Windows Server 2003 SP1 and the daatabase server is also Windows server 2003 SP1.

Most of the time when the client performs an operation to the application server the database gets updated and something is written to a queue. The writing to the database and queues is performed in the context of a transaction, we use the System.Transactions facilities and MSDTC. When the server performs the writing operation it 'impersonates' the client, although this should not have any impact since our database connection string contains a database user id and password.

This works great in almost all our test sites... but lately we have run some tests using VM's for the client and the server application. Under some specific configuration we have run into a situation where we would get an exception:


Inner Exception: UnauthorizedAccessException
Properties:
Message = 'Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))'
Data (Dictionary with 0 entrie(s))
TargetSite = 'Void ConnectToProxy(System.String, System.Guid, IntPtr, Boolean ByRef, UInt32 ByRef, System.Transactions.Oletx.CoTaskMemHandle ByRef, System.Transactions.Oletx.IResourceManagerShim ByRef)'
HelpLink = null
Source = 'System.Transactions'
Stack Trace:
at System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier, Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle& whereaboutsBuffer, IResourceManagerShim& resourceManagerShim)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions properties)
at System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction tx)
at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
at System.Transactions.Transaction.Promote()
at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)
at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx)
at System.Data.SqlClient.SqlInternalConnectionTds.Activate(Transaction transaction)
at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()


We have tripple checked the MSDTC configuration, we have run DTCping, we cannot find the problem.

We have also activated the 'Audit on Failure' and we got these errors in the event viewer:

BLITZ is our domain controller
C8SERVER is our application server
dispatcher2 is the user logged on the client pc.

Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 15/08/2006
Time: 1:41:28 PM
User: BLITZ\dispatcher2
Computer: C8SERVER
Description:
Object Open:
Object Server: SC Manager
Object Type: SC_MANAGER OBJECT
Object Name: ServicesActive
Handle ID: -
Operation ID: {0,19314977}
Process ID: 452
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: C8SERVER$
Primary Domain: BLITZ
Primary Logon ID: (0x0,0x3E7)
Client User Name: dispatcher2
Client Domain: BLITZ
Client Logon ID: (0x0,0x126A67F)
Accesses: READ_CONTROL
Connect to service controller
Enumerate services
Query service database lock state

Privileges: -
Restricted Sid Count: 0
Access Mask: 0x20015

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 15/08/2006
Time: 1:41:28 PM
User: BLITZ\dispatcher2
Computer: C8SERVER
Description:
Object Open:
Object Server: SC Manager
Object Type: SC_MANAGER OBJECT
Object Name: ServicesActive
Handle ID: -
Operation ID: {0,19314980}
Process ID: 452
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: C8SERVER$
Primary Domain: BLITZ
Primary Logon ID: (0x0,0x3E7)
Client User Name: dispatcher2
Client Domain: BLITZ
Client Logon ID: (0x0,0x126A67F)
Accesses: READ_CONTROL
Connect to service controller
Enumerate services
Query service database lock state

Privileges: -
Restricted Sid Count: 0
Access Mask: 0x20015

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 15/08/2006
Time: 1:41:28 PM
User: BLITZ\dispatcher2
Computer: C8SERVER
Description:
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
Handle ID: -
Operation ID: {0,19315216}
Process ID: 452
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: C8SERVER$
Primary Domain: BLITZ
Primary Logon ID: (0x0,0x3E7)
Client User Name: dispatcher2
Client Domain: BLITZ
Client Logon ID: (0x0,0x126A67F)
Accesses: Query status of service

Privileges: -
Restricted Sid Count: 0
Access Mask: 0x4

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 15/08/2006
Time: 1:41:28 PM
User: BLITZ\dispatcher2
Computer: C8SERVER
Description:
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
Handle ID: -
Operation ID: {0,19315253}
Process ID: 452
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: C8SERVER$
Primary Domain: BLITZ
Primary Logon ID: (0x0,0x3E7)
Client User Name: dispatcher2
Client Domain: BLITZ
Client Logon ID: (0x0,0x126A67F)
Accesses: Query status of service

Privileges: -
Restricted Sid Count: 0
Access Mask: 0x4

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 15/08/2006
Time: 1:41:28 PM
User: BLITZ\dispatcher2
Computer: C8SERVER
Description:
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
Handle ID: -
Operation ID: {0,19315260}
Process ID: 452
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: C8SERVER$
Primary Domain: BLITZ
Primary Logon ID: (0x0,0x3E7)
Client User Name: dispatcher2
Client Domain: BLITZ
Client Logon ID: (0x0,0x126A67F)
Accesses: Query status of service

Privileges: -
Restricted Sid Count: 0
Access Mask: 0x4

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 15/08/2006
Time: 1:41:28 PM
User: BLITZ\dispatcher2
Computer: C8SERVER
Description:
Object Open:
Object Server: SC Manager
Object Type: SC_MANAGER OBJECT
Object Name: ServicesActive
Handle ID: -
Operation ID: {0,19315269}
Process ID: 452
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: C8SERVER$
Primary Domain: BLITZ
Primary Logon ID: (0x0,0x3E7)
Client User Name: dispatcher2
Client Domain: BLITZ
Client Logon ID: (0x0,0x126A67F)
Accesses: Connect to service controller
Query service database lock state

Privileges: -
Restricted Sid Count: 0
Access Mask: 0x11

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 15/08/2006
Time: 1:41:28 PM
User: BLITZ\dispatcher2
Computer: C8SERVER
Description:
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
Handle ID: -
Operation ID: {0,19315273}
Process ID: 452
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: C8SERVER$
Primary Domain: BLITZ
Primary Logon ID: (0x0,0x3E7)
Client User Name: dispatcher2
Client Domain: BLITZ
Client Logon ID: (0x0,0x126A67F)
Accesses: Query service configuration information

Privileges: -
Restricted Sid Count: 0
Access Mask: 0x1

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

At the time we thought that this was a Virtual server specific problem, but we see one of our customer with the same problem on and off. The customer will exprience the same problem for a short period of time on a daily basis. We are currently trying to figure what goes on at this specific location.

Can anybody shed some ligth or give us a clue as what migth cause this problem?

Best Regards

-Martin

[12500 byte] By [MartinPar] at [2008-3-5]
# 1

You might be hitting a issue fixed in the latest COM+ QFE: http://support.microsoft.com/kb/908473/

Can you try installing the hotfix mentioned at the end of the article to see if the issue goes away?

HTH

FlorinLazar-MSFT at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 2

Thank you Florin,

I'll try to get my hands on the hotfix and I'll let you know.

-Martin

MartinPar at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 3

Hi Florin,

I just installed the hotfix in our lab and it did not resolve the issue. One thing I forgot to mention is that the server in our is running as a virtual machine, while our customeris running a 'real' windows 2003 server.

I'll continue investigating, if you have any other ideas they are more than welcome.

-Regards

Martin

MartinPar at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 4

Hi Martin,

Sorry to hear it doesn't fix your problem.

Can you give more details on what is the special config you are seeing this issue: "This works great in almost all our test sites... but lately we have run some tests using VM's for the client and the server application. Under some specific configuration we have run into a situation where we would get an exception"?

Is it an intermittent issue or in that particular config, it reproduces all the time?

Thanks!

FlorinLazar-MSFT at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 5

Ok, here it goes....

In our lab we have always tested our server application on real machines running Windows Server 2003 SP1, we always use the same database server. We have never experienced that problem in these kind of setups.

About 3 weeks ago we started testing using virtual machines. It tooks us a while to realize that the version of Windows Server 2003 that we were using was not SP1... this is when the problem started to appear. We have upgraded on of our virtual servers to SP1 and from that point on we have encountered that problem. Note that in our lab this problem is not intermitent. We have this problem every single time.

Now, last week we have deployed at a customer site. The customer is running the server application on a Windows 2003 Server R2 SP1(whatever R2 means). We are experiencing this situation every day between 10:30am and noon.... I know this sounds very stupid but that is the fact. The rest of the day the application runs without a problem. To circumvent this problem we have changed our connection string and have set "Enlist=false".

So bottom line is that we were not too worried because we were seeing this behavior only when running on virtual machine, now we are more concerned since this occurs in a production environment and a real server.

I hope this clarifies the situation.

Thank you for your help

-Martin

MartinPar at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 6

The fact that your customer is seeing the issue between 10:30am and 12:00pm is very strange. Is there anything else happening in that period of time? Like backups/domain controllers rebooted etc?

As for the virtual machines issues, what SKU of Windows Server 2003 are you using on the virtual machine? Is it by chance the Web Server SKU?

Can you run the following command on the virtual machine experiencing the access denied issue and post back the output:

"sc sdshow msdtc"

Thanks.

FlorinLazar-MSFT at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 7

Hi Florin,

I am not familiar with the term SKU, we are running "Windows Server 2003 Standard Edition Service Pack 1".

Here is the output you requested.

D:(A;;CCLCSWRPLOCRRC;;;S-1-2-0)(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)(A;;CR;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)(A;;CCLCSWRPLORC;;;NS)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)

Thank you for your help.

We are still trying to figure out what is going on between 10:30 and 12:00... the customer claims that not much special is going on, it needs be I will try to put a sniffer to see what is happening.

-Martin

MartinPar at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 8

Hi Martin,

Somebody/Something messed up the security access for Authenticated Users. You have:

(A;;CR;;;AU)

And it should be something similar to:

(A;;CCLCSWRPLOCRRC;;;AU)

Here are a few pointers with descriptions:

SC command: http://technet2.microsoft.com/WindowsServer/en/library/1e71d513-bc67-4928-889f-9654f5afbfab1033.mspx?mfr=true

ACE Strings: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/secauthz/security/ace_strings.asp

The simplest way to do this is to run the following command in one line (no spaces in the ACL list):

"sc sdset MSDTC D:(A;;CCLCSWRPLOCRRC;;;S-1-2-0)(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)(A;;CCLCSWRPLOCRRC;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)(A;;CCLCSWRPLORC;;;NS)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)"

NOTE: For other people looking at the sc sdset command - please do not use it as is, since this ACL list is specific to Martin's machine. What I did was to replace (A;;CR;;;AU) with (A;;CCLCSWRPLOCRRC;;;AU) in his original output from "sc sdshow MSDTC".

I hope this helps.

FlorinLazar-MSFT at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 9

Thank you Florin,

Changing the secutiry descriptior did the trick!!! We will keep a close watch on the sd when we will create a new virtual machine, just in case.

This does not resolve the issue at our custmoer site, but we will monitor this situation as weel.

Thank you!

MartinPar at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...
# 10

Outstanding - I was about to open a support ticket when I came across this. This was our exact problem as well and the proposed solution is working in our test environment.

Thanks!

Lane

spacattac at 2007-8-30 > top of Msdn Tech,Software Development for Windows Vista,Transactions Programming...

Software Development for Windows Vista

Site Classified