Multiple pending callouts at ALE layers
I noticed on a different thread it was said (emphasis added):
 Biao Wang [MSFT] wrote: |
| You need to use FwpsPendOperation0/FwpsCompleteOperation0 to pend ALE_Xxx invocations as long as FWP_CONDITION_FLAG_IS_REAUTHORIZE is not set. (re-auth can not be pended). Whenever a packet is indicated to ALE_Xxx layers, it needs to be cloned and re-injected back after FwpsCompleteOperation0 using either FwpsInjectTransportSendAsync0/FwpsInjectTransportReceiveAsync0 depending on packet direction. |
|
This begs the question: is having multiple callouts on an ALE layer which need to do out-of-band inspection not supported? If there are two callouts, the first has its classify function called, pends and absorbs the packet, and returns. Later it completes the operation, knows on reauth that it should allow the packet, and does so. Once it allows the packet, the packet continues down the chain to the other callout. The other callout wants to do out-of-band inspection, and thus needs to pend the packet and absorb, but the packet is still flagged as reauth (and indeed, the classify function of the second callout is invoked indirectly from FwpsCompleteOperation0). Does this mean that two (or more) out-of-band inspection callouts on the same ALE layer are not supported?
I have to say, that in my code I am currently putting two out-of-band inspection callouts on the same ALE layer, and it seems to work ok for TCP (as long as I am careful). However, I mentioned before that I got a bluescreen on WS2k8 when I tried to do this with UDP (as UDP does not pend/resume correctly on Vista due to a bug). I have not gotten this since I restructured my code to do DNS lookups differently, but it probably still exists and this limitation is the most probable cause.
Having multiple out-of-band inspection callouts on an ALE layer is a requirement for my application, as it needs to make usermode checks both before and after processing other rules.
Thanks,
Jeremy
Your second, lower-weight, callout can also pend the initial ALE_AUTH_CONNECT (is there a reason you cannot?) If packet is indicated (e.g. in case of UDP), then only one of them should clone, however. If the higher weight callout clones an NBL, it should block it with the FWPS_RIGHT_ACTION_WRITE bit cleared and the lower weight guy should not touch the layerData in this case.
WFP will re-auth the layer only after both callouts completes the pend. And the ALE flow will be opened for traffic when both callouts permit the re-auth.
Note that you should defer re-injecting the clone until after both callouts permit the re-auth -- during classifyFn of either callout the flow is still at PENDING state and the packet injected from witin classifyFn will be dropped.
So I think your scenario should work.
Biao.W.
This has confused me. Your description seems to imply a tight coupling of the two callouts, where they need to know that the other exists and what it is doing, that does not seem correct.
Assume for the sake of argument that the two callouts are written by different people. They have no way to know about the other. How can I write an out-of-band inspection callout that is robust enough to work in the case that there exists zero or more out-of-band inspection callouts with higher weight, lower weight, or both.
I only want the second callout invoked if all previous rules did not provide an action for the packet (either did not match, or returned ACTION_CONTINUE), including the result of the first callout. I do not see how that could be possible if I pend the operation twice on the first indication. The only way I can know the response of all prior rules is to wait until all earlier callouts have completed. Even if I remove the requirement of knowing what the first callout will do, I still have no way of knowing what all of the other rules in between would have done, as setting the action to block, clearing the write right, and setting absorb for pending the operation will make all normal rules be completely hands off of the indication, and thus not tell me if there was some higher weight rule that would have blocked or allowed the packet.
| Jeremy Drake wrote: |
| This has confused me. Your description seems to imply a tight coupling of the two callouts, where they need to know that the other exists and what it is doing, that does not seem correct. Assume for the sake of argument that the two callouts are written by different people. They have no way to know about the other. How can I write an out-of-band inspection callout that is robust enough to work in the case that there exists zero or more out-of-band inspection callouts with higher weight, lower weight, or both. I only want the second callout invoked if all previous rules did not provide an action for the packet (either did not match, or returned ACTION_CONTINUE), including the result of the first callout. I do not see how that could be possible if I pend the operation twice on the first indication. The only way I can know the response of all prior rules is to wait until all earlier callouts have completed. Even if I remove the requirement of knowing what the first callout will do, I still have no way of knowing what all of the other rules in between would have done, as setting the action to block, clearing the write right, and setting absorb for pending the operation will make all normal rules be completely hands off of the indication, and thus not tell me if there was some higher weight rule that would have blocked or allowed the packet. | |
| Jeremy Drake wrote: |
| Your description seems to imply a tight coupling of the two callouts, where they need to know that the other exists and what it is doing, that does not seem correct. | |
I was just commenting on your specific example. And you are right callouts registering at the same layer will likely to be independent and WFP's arbitration model was designed with that in mind.
| Jeremy Drake wrote: |
| I only want the second callout invoked if all previous rules did not provide an action for the packet... | |
You meant you don't want the 2nd callout to pend the classifyFn when invoked right? The 2nd callout is not dynamically added during the first pend correct? I don't see how you can add a filter to achieve the behavior I am quoting.
I don't see anything wrong to always pend the initial auth regardless of the classifyOut result thus far. You shouldn't touch the packet once it is hard blocked but ALE_AUTH_CONNECT is really authorizing both the connection and the packet so I don't see anything wrong with saying "hey I need some time to think about this connection" by pending the classifyFn.
When everybody makes up their minds (by completing), everbody will have a chance to inspect again -- at that time the lower weight people will know what higher weight people say and act accordingly.
Now if you want to know the finally arbitration outcome, you can add a callout at the FWPS_SUBLAYER_INSPECTION (although from here you can't change the outcome). But you could schedule a packet re-injection from there, for example.
Please furthur clarify if I am not understanding the issue correctly.
Thanks,
Biao.W.
Let me try to explain the functionality I am trying to implement in general terms and perhaps that will better indicate what I am doing.
My software has a mode where, if no existing rule matches, the user is prompted about a connection attempt, where they can allow or deny this attempt, or create a new rule. The user is only to be prompted if the connection attempt did not match any existing rules.
This is the 2nd, lower weight callout (in fact, I make sure it has the lowest weight of all of my filters on the layer). When a packet is indicated to this callout with the write right set, I pend the connection attempt and indicate it to user mode to prompt the user, and when the user responds I continue and re-inject if there was an NBL to clone.
This works fine, and is perfectly reasonable, until you add another callout which pends operations to the layer. This callout exists at the very highest weight, and is invoked for every connection regardless of eventual result. What this callout does is unimportant to the discussion, except that it is also out-of-band, and may also prompt the user.
In order to know whether or not to prompt the user from the lower weight callout, I need to know if some higher weight rule matched. My understanding is, I would then either have to pend a reauth, which is what my code does now and seems to work, or what you just said, put my callout on FWPS_SUBLAYER_INSPECTION, but then I cannot change the outcome.
Does this make sense? Do you have any ideas as to the proper way to implement this?
Yes it makes sense now. Basically the lower weight callout needs to pend only after all higher weight callouts complete their pends, hence the need to pend a completion-triggered re-auth.
We do have provision to support such a re-auth. So to correct my earlier post that you quoted -- FwpsCompleteOperation0- triggered re-auth can be pended (your implementation is an example) but policy change- triggered re-auth cannot. An example for the latter -- say your outbound connection is authorized by the 2 callouts and traffic start flowing, sometime during the flow a filter is added/removed from the layer (i.e. Policy Change) hence it will be re-auth, during this invocation, only clone-drop-reinject/discard is supported but not Fwps{Pend|Complete}Operation0.
Hope this answers your concern,
Biao.W.
I edited the other thread to reflect the correction made here.
| Biao Wang [MSFT] wrote: |
| You need to use FwpsPendOperation0/FwpsCompleteOperation0 to pend initial ALE_Xxx invocations. (re-auth triggered by policy change cannot and need not be pended). Whenever a packet is indicated to ALE_Xxx layers, it needs to be cloned and re-injected back after FwpsCompleteOperation0 using either FwpsInjectTransportSendAsync0/FwpsInjectTransportReceiveAsync0 depending on packet direction. With the exception of ALE_AUTH_RECV_ACCEPT, All other ALE_Xxx layers will be re-auth'ed following an FwpsCompleteOperation0 call and it is from this re-classify the inspection decision is to be returned. (Packet queued during the initial classify should only be re-injected back after this second classify is authorized). Callout drivers need to maintain enough states to associate the first and second classify and also to store the inspection result. Re-auth triggered by FwpsCompleteOperation0 can furthur be pended. | |
Great. That makes sense. Just a couple more questions:
- How can I differentiate pendable and non-pendable reauths? Attempt to pend and look for an error code?
- How can I tell when I call Continue whether another callout re-Pended? Error code?
- How do I get the cloned NBL to the second callout?
Do I have to break the abstraction of WFP and hard code these cases in my driver between the two callouts, and if so, how do you expect to be able to support multiple third-party callouts?
Thanks for your help
1. Yes when a re-auth is not pendable, WFP should fail the FwpsPendOperation0 call with STATUS_FWP_CANNOT_PEND. However in Vista RTM we are not returning this error reliably. For example when re-auth is triggered by policy change the API does not fail, which can cause reliability issues. The check is now enforced in WS08 Beta 3.
2. By "Continue" you mean FwpsCompleteOperation0 right? And you want to know whether the re-auth triggered by FwpsCompleteOperation0 is pended by a 3rd party by the time FwpsCompleteOperation0 returns? I wonder why you need that infomation. To know whether you can re-inject the clone right away?
3. I don't think this can be done at ALE_AUTH_CONNECT currently. Ideally when the higher weight callout completes the operation, he should be able to re-inject the clone right away and during re-auth the clone would be passed onto the lower weight callout. Currently the support for this hand-off is not there and the re-injected packet will simply be discarded because the connection is still at "pending" state. This is something we need to work on for Win7.
A heuristic for 3. would be to register at the FWPS_SUBLAYER_INSPECTION and schedule the re-injection when everybody permits the classify. I said "schedule" because re-injection from within classifyFn wouldn't solve the problem because during which time the connection is still marked "pending". So you would need to queue a work item to perform the re-injection some short time (e.g. in mS) in the future.
Again thanks for the feedback.
Biao.W.
| Biao Wang [MSFT] wrote: |
| 2. By "Continue" you mean FwpsCompleteOperation0 right? And you want to know whether the re-auth triggered by FwpsCompleteOperation0 is pended by a 3rd party by the time FwpsCompleteOperation0 returns? I wonder why you need that infomation. To know whether you can re-inject the clone right away? | |
Yes, FwpsCompleteOperation0. I need this information, because you said earlier that I cannot reinejct the packet until all callouts which Pended have Completed.
| Biao Wang [MSFT] wrote: |
| 3. I don't think this can be done at ALE_AUTH_CONNECT currently. Ideally when the higher weight callout completes the operation, he should be able to re-inject the clone right away and during re-auth the clone would be passed onto the lower weight callout. Currently the support for this hand-off is not there and the re-injected packet will simply be discarded because the connection is still at "pending" state. This is something we need to work on for Win7. | |
Ouch! Not even 6.1? 
| Biao Wang [MSFT] wrote: |
| A heuristic for 3. would be to register at the FWPS_SUBLAYER_INSPECTION and schedule the re-injection when everybody permits the classify. I said "schedule" because re-injection from within classifyFn wouldn't solve the problem because during which time the connection is still marked "pending". So you would need to queue a work item to perform the re-injection some short time (e.g. in mS) in the future. | |
This should be workable for the short term, I think. Now I need to read up on FWPS_SUBLAYER_INSPECTION in the docs (I don't even remember seeing it).
Either that, or I can probably break the abstraction between my callouts, and assume for the time being that no other callouts will be installed on the system. But this is not a very nice assumption...
Apparently, on reauth due to FwpsCompleteOperation0, the transportEndpointHandle is NULL, when it was valid on the initial indication. This is for UDP on ALE_CONNECT on server 2008 beta3.
Don't know if controlData is preserved or not, as the packet I was looking at did not have it set, and I don't know how to get a packet which has this, but my bet would be that is NULL as well.
You may want to look into this for SP1...
Currently FwpsCompleteOperation0- triggered re-auth is data-less (layerData is NULL) hence there is no transportEndpointHandle nor controlData present in the meta data. A callout driver would need to cache these fields during the initial classify.
We will revisit this design (along with the issues/limitations you mentioned earlier in this thread) in Win7 timeframe.
Thanks,
Biao.W.
