Designing SSB queues, EA sample and unreliable interdependant NTFS tasks
Our current project involved into managing NTFS hierarchical folders structured by multi-level customers and associated projects for them. Essentially each folder level represents one level of customer hierarchy or project root. Both have different subfolders and user access rights for them based on generic XML templates. The folders resided on file servers across the country and should be accessible in ordinary way NTFS file shares are allowing. LAN/Intranet MS AD Win2003 / SQL2005 environment.
The folder management system basically have to keep the folder structure in tact with changes in underlying managing application logic. That involves such operations as creating a new folder with subfolders, copying a folder with its content into another folder branch (which may be on the same or another server and place), deleting a folder/content, renaming a folder, applying NTFS access rights to folders and subfolders for users according generic templates. As all these actions are unreliable and some tasks may take hours to complete, SSB approach seems to be the viable solution. Some tasks involved have to be done within 10 minutes, others (are prolonged) have to be scheduled for overdnight run. Windows service like EA sample may be used to start the actual NTFS related tasks. Content transfer involves WMI remoting with robocopy tool on target machine (for better network utilization), other related tasks make use of WMI APIs and probably do direct (i.e. synchronous) calls to the remote target file server.
At this stage, making atomary executable modules that do just one functionally isolated task like DirCopy, DirCreator, DirRename, DirDelete, DirUserAccess seems like somewhat logical choice.
The questions starting to arise from SSB queues planning to adoption of ExternalActivator sample to run these atomic executables. The problem is that if SSB messages contain atomic tasks for these executables, they needed to be syncronized in two ways - by execution precedence (create or copy dir first, then apply users' access rights) and transactionally (only when all related tasks succeed, the appropriate feedback action and event log writing can be taken.I can imagine two implementation scenarios below.
Case A. Create common queue for directory creating, content transfer and access rights.
In that case ExternalActivator has to be either extended & re-designed or it has to activate another Activator-Coordinator (middleway) executable, that would actually read the queue and, based upon the message type, run the appropriate atomary executable. In that scenario, the queue processing would be stalled because if the previous task was content transfer and lasts, say access rights to them can't be started before the transfer task has been finished OK. That in turn probably will require running multiple instances of the atomary tasks and using conversation groups, wouldn't it? What would be the most reliable and simple way to achive that?
Case B. Dedicated queues for each atomary executable modules.
Neither EA changes nor extra Activator-Coordinator middleway executable required. But because the atomary task-oriented queues are not syncronized with each other the queues internal conversation groups wouldn't help much... That means if a directory not yet exists, userRights module have to wait. But, what if we are transferring directories from path X to Y, based on what the userRights module knows to wait? With zero asynchronous design experience I'm lost here...
Hope I described the app domain understandably, thanks for hints leading to the working solution!
[4740 byte] By [
Vikh] at [2008-3-1]
The conversation semantic used in SSB gaurantees the following:
1> Message ordering -- so if you send the DirCreate message before the DirCopy message, you are gauranteed that they will be received in that order.
2> Isolation/Serializability -- so that if you receive a DirCreate message as part of one transaction in one thread, until you've processed that message (and ended the transaction), other threads cannot receive the DirCopy message in a different transaction.
This second property is provided by means of locking. But instead of locking conversations, we provision locking at a more granular level -- conversation group. Typically, each conversation will have it's own unique conversation group, but in some applications, the user might want to relate two conversations and serialize access to them and so we provision locking at the group level. (In your case, you can just map converation = conversation group).
So in your case, you might send DirCreate and DirCopy on the same conversation in the order you wish them to be executed, but you could send DirUserAccess on a different conversation after the target service sends a 'DirCreated' response message after having created the directory. That way the DirUserAccess and DirCopy command may be run in parallel.
The external activator does not RECEIVE messages from the target queue. It actually launches an exe to do that. So as described in Case A, you will need to write an exe that the external activator will launch. I don't understand how you plan to use the external activator in case B without any service exe.
Finally, you don't want to be holding the database transaction for as long as it takes to complete your job... especially for jobs like DirCopy that could last minutes. In that case, you would set the state of that conversation (state could be stored in a table and associated with the converation handle) to 'DirCopying' and commit the transaction while you are still performing the copy in background. If another message arrives on that conversation, and you find yourself in this DirCopying state, you could simply log the message to a pending operations table. When the copying task completes, it can lookup at the pending operations table for commands that were received while it was busy copying and execute those in order.
To preserve order between messages one must use a dialog, so Case A is the only one that would fit.
The simplest way to achieve this is like this:
1. begin a transaction
2. waitfor to receive a message from the queue
3. inspect the message type and payload and start the appropiate file operation (DirCreate, DirCopy, DirUserAccess etc)
4. wait for the file operation to complete
5. if this operation completes a set of related tasks, send feedback
6. commit the transaction
7. go back to step 1
This way, if anything happens before the step 6, the message is pushed back on the queue and will be received again on the next attempt. Also, only one client process or thread processes one set of related tasks at once.
Because the processing can crash anytime between step 2 and 5 causing the message to be processed again, the file operations and the send feedback tasks has to be either idempotent (executing the operation twice give the same result as execurting it once) either testable (one can test wether the operation was already executed once before). All file operations I can think of can satisfy this. As about the feedback, it depends on how the feedback is sent to the user. My sugestion is for the response to be sent back as an SSB response message on the dialog. When the sender receives this response, it knows that the whole set of related operations completed and can initiate the feedback to the user (via mail or show an update on the UI or whatever).
The problem with this is the fact that you are holding long lived database locks. Each RECEIVE will place an exclusive database lock on the received messages (think at them as rows in a table) and on the conversation group. This is fine with in regard to other RECEIVE statements (they are designed to avoid each other locks w/o blocking), you might run into problem if you application does other things like scans over the message queues (SELECT from a message queue), which may block for a long time waiting for the blocking RECEIVE to commit.
HTH,
~ Remus
 Rushi Desai wrote: |
| The conversation semantic used in SSB gaurantees the following: 1> Message ordering -- so if you send the DirCreate message before the DirCopy message, you are gauranteed that they will be received in that order. 2> Isolation/Serializability -- so that if you receive a DirCreate message as part of one transaction in one thread, until you've processed that message (and ended the transaction), other threads cannot receive the DirCopy message in a different transaction. |
|
You mean 2> when DirCreate and DirCopy are sent on the same conversation dialog?
| Rushi Desai wrote: |
| This second property is provided by means of locking. But instead of locking conversations, we provision locking at a more granular level -- conversation group. Typically, each conversation will have it's own unique conversation group, but in some applications, the user might want to relate two conversations and serialize access to them and so we provision locking at the group level. (In your case, you can just map converation = conversation group). |
|
My thought was to use conversation groups to prevent executing new conversation dialog if it has common file paths with the previous (currently in queue) conversations that is critical to avoid the simultaneous access clashing. Say you can't rename dir \\srv1\share2\dir3\subdir4 to subdir 44 if some other process currently does \\srv1\share2\dir3 to \\srv2\share3\dir4\dir5 copy action. The other way is just keep a separate table of all queued paths and lookup whether any clashes can occure, is this any better solution, won't it somewhat defeat the SSB purpose?
| Rushi Desai wrote: |
| So in your case, you might send DirCreate and DirCopy on the same conversation in the order you wish them to be executed, but you could send DirUserAccess on a different conversation after the target service sends a 'DirCreated' response message after having created the directory. That way the DirUserAccess and DirCopy command may be run in parallel. |
|
How the queues and the services have to be designed, would we have separate RequestQueue and TaskQueue with assosiated services or just one common queue? Would we get any benefit when separating the common task queue into almost instant one (for operations that take place almost instant and last few minutes max) and the overnight queue for scheduled tasks? Where/when the best way to set scheduling timer for prolonged tasks?
| Rushi Desai wrote: |
| The external activator does not RECEIVE messages from the target queue. It actually launches an exe to do that. So as described in Case A, you will need to write an exe that the external activator will launch. I don't understand how you plan to use the external activator in case B without any service exe. |
|
I'm aware the EA sample does neither read/select nor receive from the target queues. It probably just consumes [ExternalActivatorQueue] somehow. The case B means the activation of a dedicated service.exe depending on the target queue (from the appropriate configuration record from app.xml config file), i.e. EA sample just maps the target queue and target service executable in relation 1:1, am I wrong? The whole case B with EA sample as it is works only if there are no any kind of dependances between activated services, no? In our case, unfortunately, that's not true.
| Rushi Desai wrote: |
| Finally, you don't want to be holding the database transaction for as long as it takes to complete your job... especially for jobs like DirCopy that could last minutes. In that case, you would set the state of that conversation (state could be stored in a table and associated with the converation handle) to 'DirCopying' and commit the transaction while you are still performing the copy in background. If another message arrives on that conversation, and you find yourself in this DirCopying state, you could simply log the message to a pending operations table. When the copying task completes, it can lookup at the pending operations table for commands that were received while it was busy copying and execute those in order. |
|
In our case prolonged DirCopy can take hours, not minutes (but not always). Does it mean the holding transaction for hours will keep db locks and no SELECT from a message queue would be possible?
Where I can find a sample of storing/retrieving of conversation state? Wouldn't the keeping pending operations table defeat the SSB usage feasibility in the first place? That implementation would distribute the business logic from initiator (that resides in stored procedures in db) to target services (that initially were planned as a simple executables to do a single NTFS task). Plus, if I understood correctly, EA should be re-designed to actually RECEIVE messages from the common queue to be able to activate the right executable based on the message data. Then some mechanism would be required to transfer begin transaction, wait for receive from EA to activating executable further... Or should the activating module be just a dll (a separate VS2005 project in the solution) started in dedicated thread of EA?
Hi, unfortunately yet more questions than sentences following 
| Remus Rusanu wrote: |
| To preserve order between messages one must use a dialog, so Case A is the only one that would fit. |
|
Yes, that seems to me too. Unless we want to shift the massive part of the business logic to the target part of SSB (with only high level tasks in the queue that have to be splitted after receiving, thus eventially re-reating the case A).
| Remus Rusanu wrote: |
| The simplest way to achieve this is like this:
1. begin a transaction
2. waitfor to receive a message from the queue
3. inspect the message type and payload and start the appropiate file operation (DirCreate, DirCopy, DirUserAccess etc)
4. wait for the file operation to complete
5. if this operation completes a set of related tasks, send feedback
6. commit the transaction
7. go back to step 1 |
|
Does that mean the External Activator service works as a coordinator/distributor and executes steps 1-3 or even 1-7?
What the mentioned in 5. set of related tasks means, is it say CreateDir and SetUserRights messages sent by initiator on the same dialog?
| Remus Rusanu wrote: |
| This way, if anything happens before the step 6, the message is pushed back on the queue and will be received again on the next attempt. Also, only one client process or thread processes one set of related tasks at once. |
|
How this pushing back should be implemented, by conversation timer? How the next attempt should be scheduled for retry? The problem with dialog = always new conversation group approach is that we have to avoid the situations I mentioned to Rushi for directory paths clashes between separate conversations, or concurrent application tasks targeted to the common dir paths.
| Remus Rusanu wrote: |
| Because the processing can crash anytime between step 2 and 5 causing the message to be processed again, the file operations and the send feedback tasks has to be either idempotent (executing the operation twice give the same result as execurting it once) either testable (one can test wether the operation was already executed once before). All file operations I can think of can satisfy this. As about the feedback, it depends on how the feedback is sent to the user. My sugestion is for the response to be sent back as an SSB response message on the dialog. When the sender receives this response, it knows that the whole set of related operations completed and can initiate the feedback to the user (via mail or show an update on the UI or whatever). |
|
Executing the operations like DirCopy twice may take several hours so is out of question. What are the possible testability implementation approaches? Can we just change some part/node of XML message showing the stage of execution / status in-between? Or, should we keep extra table per dialog conversation somewhere? Also, sending feedback as response message to the dialog that transaction was committed (as Rushi suggested) - maybe we can use the same extra table mentioned? Who and when that table should be cleaned then, does the whole approach starting to have more housekeeping than the benefit SSB brings, maybe old trusty single task table would work better in this case?
| Remus Rusanu wrote: |
| The problem with this is the fact that you are holding long lived database locks. Each RECEIVE will place an exclusive database lock on the received messages (think at them as rows in a table) and on the conversation group. This is fine with in regard to other RECEIVE statements (they are designed to avoid each other locks w/o blocking), you might run into problem if you application does other things like scans over the message queues (SELECT from a message queue), which may block for a long time waiting for the blocking RECEIVE to commit. HTH,
~ Remus |
|
Does that mean select from a queue is practically out of question, what are the implications of that regarding reliability of the EventNotification mechanism used in EA or other parts? Doesn't that mean we have to exclude RECEIVE locks by committing just after dedicated service.exe has been started? If that case, seems that the SSB part should be finished and some extra task table used. How about feedback message to initiator then?
| Vikh wrote: |
| Does that mean the External Activator service works as a coordinator/distributor and executes steps 1-3 or even 1-7? |
|
Let's keep the EA out of the picture for a moment. For simplicity, consider you have a process running that executes steps 1-6 in a loop. These steps cannot be split between processes because they're one transaction.
| Vikh wrote: |
| What the mentioned in 5. set of related tasks means, is it say CreateDir and SetUserRights messages sent by initiator on the same dialog? |
|
Yes, related tasks are sent ar messages on the same dialog to guarantee correct order of execution.
| Vikh wrote: |
| How this pushing back should be implemented, by conversation timer? How the next attempt should be scheduled for retry? |
|
Since the transaction that dequeued the message is not commited, the message dequeue operation is rolled back. The next attempt happens automatically becase the rolled back message is again available for dequeue and the activation mechanism will launch again the processing task.
| Vikh wrote: |
| What are the possible testability implementation approaches? Can we just change some part/node of XML message showing the stage of execution / status in-between? |
|
What I meant is that given a message payload tha, for example, says 'Create directory \foo\bar', by simply testing the NTFS system for existance of '\foo\bar', you can determine if this test had already completed before.
| Vikh wrote: |
| Does that mean select from a queue is practically out of question, what are the implications of that regarding reliability of the EventNotification mechanism used in EA or other parts? Doesn't that mean we have to exclude RECEIVE locks by committing just after dedicated service.exe has been started? |
|
User SELECTs are the ones in trouble. Even they can still use READPAST lock hint to avoid locked rows. The Event Notifications system is not affected by these locks.
My initial understanding was that the messages are sent to remote SSB services, one for each file server. This kind of distributed application is where SSB advantages are clear. But now I realise that you probably mean only local messages (within same database), as a way to provide a flow control system. If your design is the later, then is true that you can use some benefits from SSB like message order guarantee and conversation_group lock exclusion, but this is not what SSB was primarily designed for.
HTH,
~ Remus
| Remus Rusanu wrote: |
| | Vikh wrote: | | How this pushing back should be implemented, by conversation timer? How the next attempt should be scheduled for retry? | | Since the transaction that dequeued the message is not commited, the message dequeue operation is rolled back. The next attempt happens automatically becase the rolled back message is again available for dequeue and the activation mechanism will launch again the processing task. |
|
I thought there is risk of queue deactivation due to poison messages with rollback mechanism in place. Immediate retry would have no meaning in the app context. Also, aren't your advice shown in this link is more appropriate in this case as well?
| Remus Rusanu wrote: |
| | Vikh wrote: | | What are the possible testability implementation approaches? Can we just change some part/node of XML message showing the stage of execution / status in-between? | | What I meant is that given a message payload tha, for example, says 'Create directory \foo\bar', by simply testing the NTFS system for existance of '\foo\bar', you can determine if this test had already completed before. |
|
I got that, it just not always feasible to do, especially in case with NTFS UserAccessRights that might be just too complex to test. Sure, some shortcut strategy is possible... That's why I asked for possibilities of mantaining state of the separate actions in the single conversation's action/message chain.
| Remus Rusanu wrote: |
| | Vikh wrote: | | Does that mean select from a queue is practically out of question, what are the implications of that regarding reliability of the EventNotification mechanism used in EA or other parts? Doesn't that mean we have to exclude RECEIVE locks by committing just after dedicated service.exe has been started? | | User SELECTs are the ones in trouble. Even they can still use READPAST lock hint to avoid locked rows. The Event Notifications system is not affected by these locks. My initial understanding was that the messages are sent to remote SSB services, one for each file server. This kind of distributed application is where SSB advantages are clear. But now I realise that you probably mean only local messages (within same database), as a way to provide a flow control system. If your design is the later, then is true that you can use some benefits from SSB like message order guarantee and conversation_group lock exclusion, but this is not what SSB was primarily designed for. HTH,
~ Remus |
|
Yes, you nailed my exact intentions now! The system requirement was and still is that NONE extra services, software installations, etc. were allowed on the file servers. That way, we are using SSB on the same db instance that the rest of applications/system utilizes. As I wrote in my message to Rushi, I think using conversation groups to eliminate clashes that might occure when susequent tasks / conversations trying to use the same common NTFS path/dir for their actions; maybe setting that path that the conversation actions supposed to apply as the conversation group ID/name. Do you think such use of SSB is feasible or practical?
Maybe I shoud just use a regular table as a common queue with just SSB EventNotification mecanism to avoid polling the table from the target services? Note: all the target NTFS action servises are supposed to be on the same server (that may be different machine that SQLServer or Web managing application server). Or, does SQLServer2005 has simplier (not involving SSB queues) mechanisms just for event notifications?
In any case, I think the folder management system like this would benefit from asunchronous communication mechanism and allows more SOA-oriented design, what do you think?
Also, I'd appreciate if I get some hints regarding ExternalActivator sample usability in the system (what and in what way EA sample should be modified as its clear now it won't serve as is); BTW I didn't find ServiceBrokerInterface code or samples from our SQLServer Developer release DVD, are there any samples at all?
Please comment/criticize my plan that can be found here. There are cases A and B as a possible solution. As shown in case A, before initiating new dialog the initiator service has to check (from an extra user table or maybe just using the table where stored messages bodies that target services use?) whether file path for the new task could clash/conflict with the currently queued conversation dialogs - if that's the case, the new dialog should be created within the same conversation_group_id (or, which should be essentially the same, with the related dialog_id in the begin dialog clause). If extra table used (still not sure why not to use the single one that target services might use as well), then either initiator (on processing feedback message) or target service has to cleanup that extra table... Thank you!
I don't think scenario B gives you any advantage. The purpose of the EA sample is to show how one process can monitor one single queue (the notifications queue) to start reader processes on a anumber of service queues, by having all the service queues deliver the activation events to one single notification queue.
In your case by introducing thr EA it only complicates things w/o any benefit. Or at least I don't see any benefit.
About Case A, it appears to me that you expect the conversation groups to travel with the conversations from NtfsTaskInitiatorQueue to NtfsTakQueue. Conversation groups are always local to one endpoint of the conversations, so the target conversation endpoints (the ones on NtfsTaskQueue) will all be unrelated, each in its own group. Also, since you plan to have only one receiver for the NtfsTaskQueue service, why are you concerned with concurency/lcoking issues?
Frankly, Service Broker will give you little benefits in you design. Almost everything you're doing could be achieved by simple tables and DML operations. The sources of unreliability are mostly in the remote nature of the the file operations, to a lesser degree, in the launch of and comunication with the local Activated NTFS task.
Just for the record, I believe that SSB would benefit such a scenario a lot if it would be deployed at the communication with the file servers themselfes. Just think how much easier the design would look like if the 'activated ntfs task' operation would be guaranteed with reliability, eliminating the need for retries in the scheduler.
HTH,
~ Remus
Hi, now seems that things starting to get their feasibility shapes and we are almost but not yet done with all of those my completely uneducated and unworkable plans 
What I need is either the working solution based on SSB (w/out SqlServer setups on remote file servers) or the answer why SSB wouln't serve better than common ordinary approach with polling and updating a task table. In latter case, the information on better alternatives to table polling would be greatly appreciated.
| Remus Rusanu wrote: |
I don't think scenario B gives you any advantage. The purpose of the EA sample is to show how one process can monitor one single queue (the notifications queue) to start reader processes on a anumber of service queues, by having all the service queues deliver the activation events to one single notification queue.
In your case by introducing thr EA it only complicates things w/o any benefit. Or at least I don't see any benefit. |
|
I thought that in case B the EA used to instantiate up to N instances of TaskDistributor.exe's just to be able to deal with different conversation groups in parallel. Unfortunately, as you said, these groups are local to endpoint, thus we have to group arriving messages on NtfsTakQueue, not on the initiator side... and that seems to be impossible as no begin dialog there, am I wrong?
| About Case A, it appears to me that you expect the conversation groups to travel with the conversations from NtfsTaskInitiatorQueue to NtfsTakQueue. Conversation groups are always local to one endpoint of the conversations, so the target conversation endpoints (the ones on NtfsTaskQueue) will all be unrelated, each in its own group. Also, since you plan to have only one receiver for the NtfsTaskQueue service, why are you concerned with concurency/lcoking issues? |
|
So seems there is no means to employ conversation groups to detect when different tasks set to use the same NTFS path that potentially can cause NTFS task clashes?
Sure, in case A the single receiver used. But, as the receiver transaction committed before the actual task ended (thus no waiting in receiver loop, i.e. non-blocking queue processing assumed), we have to somehow ensure that only unrelated messages/tasks can be run in parallel. If we would rather have to wait for the NTFS task ending inside receive loop, we couldn't process more than one task at a time, or should we just use multiply readers waiting for the task completion that pushed by EA? How we can achieve the common path clash detecting then?
Edit 2nd March 2006:
As you somewhere said "the order of conversations within a group is random" so we have to send related (on the UNC path) messages within the same dialog/conversation. Couldn't we just use the same table used for storing the message bodies (we can have dialog GUID and its critical path(s) there)?
But then we need multiply receiver instances to handle different dialogs like shown in case B, don't we? Further, receive loop on target service (task distributor) should be classical blocking type, i.e. we have to start transaction and wait until the task on the same conversation finished before processing with the next message. Would this work well?
Frankly, Service Broker will give you little benefits in you design. Almost everything you're doing could be achieved by simple tables and DML operations. The sources of unreliability are mostly in the remote nature of the the file operations, to a lesser degree, in the launch of and comunication with the local Activated NTFS task.
Just for the record, I believe that SSB would benefit such a scenario a lot if it would be deployed at the communication with the file servers themselfes. Just think how much easier the design would look like if the 'activated ntfs task' operation would be guaranteed with reliability, eliminating the need for retries in the scheduler. |
|
The remote nature of NTFS tasks, even when implemented as SQLServer instance per file server, doesn't eliminate the unreliable nature of NTFS operations (failures due to file user locks, another processes/tools may do something with partitions/file shares, etc.). I don't getting how the supposed multi-instance SSB design could eliminate the need for re-sheduling some NTFS task that failures. Would the SomeNtfsTask.exe that executed on remote file server and reads/receives the queue messages by itself from the single db instance (as in my diagrams) perform better in that regard? Isn't that "almost everything" except the event notification that allows bypass continious table polling (any better than polling alternatives)?
