-
Notifications
You must be signed in to change notification settings - Fork 29
Mctp bridge support #71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mctp bridge support #71
Conversation
Thanks for the contribution! I'll get to a proper review shortly. I have some pending changes that rework a lot of the peer, link and network allocation mechanisms. That shouldn't affect your code too much, but I'll request a rebase once that is merged. |
Sure no problem |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the main design point here is how we're handling the pool allocations. It looks like your particular use-case is around static allocations, which I'll focus on here.
As I mentioned in the dbus changes, we cannot add arguments without further version-compatiblity changes. After a bit of chatting with the team, I think a better approach would be to add a new dbus call to explicitly allocate a bridge and a predefined pool (which would include the pool size). Perhaps something like:
AllocateBridgeStatic(addr: ay, pool_start: y, pool_size: y)
- where the
Set Endpoint ID
response must match the expected pool size.
(we would also want purely-dynamic pools to be allocated from SetupEndpoint and friends, but that would result in a dynamic pool allocation. This dynamic pool would be defined either by a toml config option, or via a new TMBO dbus interface. However, we can handle those later, I think)
Would that work?
In general, can you add a bit more of an explanation / rationale as part of your commit messages, instead of just log output? There is some good guidance for commit messages up in the "Patch formatting and changelogs" section of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/5.Posting.rst |
We'll also need to consider the routing setup for bridged endpoints. Ideally we would:
the issue is that there is no kernel support for (2) at present: we need some kernel changes to implement gateway routes. It is possible to create "somewhat-fake" routes for those endpoints, using a neighbour table entry for each (bridged) peer that uses the bridge phys address, but that's a bit suboptimal. I'd prefer not to encode that hack into mctpd if possible. I do have a todo for the kernel changes necessary for that, sounds like I should get onto it! |
IIUC, 1 is what we can achieve with the tools we have today, right? For ex: add route to the bridge itself and then When you say sub-optimal, are you referring to the neighbour lookup that happens in When is the gateway support in kernel for MCTP nets planned? We can help if you have a design in mind. |
Hi Santosh,
Yes, but it requires a lot of workaround to set up.
That isn't adding a neighbour table entry though; just a route. USB is a little different in that there are no neighbour table entries required, because there is no physical addressing. For a bridge, using this scheme would require:
(for USB, we don't need (2) or (4), but that's purely a property of the transport type. We would need those to be supported in mctpd to allow other transport types like i2c). This would work, but it's messy.
No, the neighbour lookups happen in
Not so much faster, more tidier. With a gateway route, we would require:
No fake neighbour table entries are required - since the kernel just looks up the gateway physical address from the gateway's neighbour table entry.
I have it done - will push a development branch shortly. |
|
Hi Jeremy, Thank you for the detailed response.
Ack, I see something like I2C would need a PHY address.
Ack. I should have said the neigh_lookup call that happens in route.c!
Thank you, that does seem cleaner. |
And for the userspace changes, my |
@jk-ozlabs : I think we agree that mctpd has to poll all allocated endpoints with a Get Endpoint ID periodically. I think the first thing we'd need to enable in order to do that is to make MCTP requests and responses asynchronous. Do you have a design in mind to make MCTP requests async (like via a request queue per allocated endpoint)? |
Just as a clarification - not all endpoints, but EIDs within allocated endpoint ranges, which have not yet been enumerated. And this is assuming we expect mctpd to automatically enumerate those bridged devices. I think the latter is reasonable, but we don't have a specific design point around that yet. With that in mind, yes, we probably want to make that async, as those requests are likely to not have a response, and therefore we're at worst-case waiting time. In terms of design: we probably don't want a We don't necessarily need to keep much state for that polling mechanism (ie, between request and response) - receiving a Get Endpoint ID response for anything in that range would trigger the enumeration process. |
Wouldn't we also want to poll enumerated endpoints under the bridge to determine when they "went away"?
Ack. How periodically do you think we should check? Same as the logic for determining when to set endpoint state as degraded (TReclaim/2)? |
No, and we don't do that with directly-attached endpoints either. The current philosophy is that we don't care if an endpoint disappears, until some application calls [I'm okay with revisiting this, or handling bridged endpoints differently, if there's a compelling argument for doing so]
Treclaim/2 seems a bit too often to me, but might be fine as a starting point. I suspect that an ideal approach would be to poll more regularly when a bridge pool is initially allocated, then reduce frequency. However, let's not complicate the initial implementation too much here, and just use a configurable constant. |
.. and speaking of |
So we have a case where we will have to call allocate endpoint ID on the bridge device when not all of its downstream devices are available. In such a case, how do you think we can determine when those downstream EIDs become available unless we poll? |
I am suggesting we poll. Just that we then stop polling once we enumerate the endpoint. |
Ah, ack, then. |
It will need some rework as currently it assumes the peer is a neighbour and uses physical addressing for |
Thanks for that, Andrew. There might be some commonality between the peers undergoing (non-local) recovery, and those EIDs that are behind a bridge, but not-yet enumerated. If a The same should occur for a |
Yep, that sounds sensible. |
686395a
to
f6d0b8a
Compare
Thank you all for taking out time to look into the PR, I've addressed to the asked comments on previous commit, added new commit for MCTP Bridge design doc, need to push Polling mechanism now |
Thanks for the updates! A couple of comments:
|
bf8f331
to
fa59ed7
Compare
Hello Jeremy Thank you for looking over the commits, based on your comment # 1 I have removed the new .md file which captured MCTP Bridge support details on PR and updated the existing mctpd.md file with new information about dbus api AssignBridgeStatic. Regarding user consumable document, I'm not much sure what this could be, if you could let me know what this document should be I can create one and update the PR. I recently got the permission for PMCI WG, have glanced over what was stated on issue #1540, basically the Idea is to split the BusOwner EID pool and segregate a chunk of eids for Bridge's downstream pool on the higher end of Busowner pool while keeping lower end for non-bridge devices. This would be helpful for Dynamic EID assignment of downstream pool devices incase multiple Bridge's are there under same network. My current implementation involves finding of contiguous eid chunk of min(requested pool size, bridge's pool size capability) from available BusOwner's pool but we begin looking from Asked pool_start (Static) or from next to Bridge's EID (Dynamic) and we look till we get right sized chunk and mark that eid as pool_start. I did based this from the same line of spec for which you raised the issue.
I can create a new PR for Endpoint polling if thats what you mean and skip for this PR. Also for adding route for the downstream endpoint to Bridge, we would need your implementation implementation to be merged for both linux kernel and mctpd. Internally I've tested my polling logic with your pulled changes, but for this PR I haven't picked them up so discovery of downstream EID via LearnEndpoint would probably not be possible with only this PR
I've updated the patch set now for easier review, hope it helps, let me know if I can do anything else to further ease the review. Thanks for your .clang format, once that is pushed I would reply those onto my change |
fa59ed7
to
2766a52
Compare
* updated mctpd.md with new mctp bridge support for dynamic eid assignment from AssignEndpoint d-bus call Signed-off-by: Faizan Ali <[email protected]>
Add new test for validating AssignEndpoint D-Bus method to verify bridge endpoint EID allocation being contiguous to its downstream eids. Add Allocate Endpoint control message support with new endpoint property for allocated pool size also assign dynamic eid contiguous to bridge during Allocate Endpoint control message. Signed-off-by: Faizan Ali <[email protected]>
New endpoint object interface au.com.codeconstruct.MCTP.Bridge1 which will capture details of bridge type endpoint such as pool start, pool end. Update test framework with new test methods to validate bridge pool assignemnt. [Minor rebase rework from Jeremy Kerr <[email protected]>; don't fail the allocation on signal emit failure] Signed-off-by: Faizan Ali <[email protected]>
Currently, test_assign_dynamic_bridge_eid test both the bridge assignment, and conflicts against static EIDs. Instead, split this into two smaller tests, which provide a base for future bridge-conflict tests. Signed-off-by: Jeremy Kerr <[email protected]>
In addition to the static assignments, we want to ensure that LearnEndpoint does not result in EID conflicts. Signed-off-by: Jeremy Kerr <[email protected]>
… ep assignment We want to ensure that running out of bridge range space does not cause a failure to allocate a non-bridge EID. We speculatively allocate before we determine bridge/non-bridge status, so this may cause issues. Signed-off-by: Jeremy Kerr <[email protected]>
add_peer() returns -EEXIST if a proposed EID is already allocated to a peer, but -EADDRNOTAVAIL if it is allocated to a bridge. Callers are expecting EEXIST for conflicts, so use that instead. Signed-off-by: Jeremy Kerr <[email protected]>
…range Signed-off-by: Jeremy Kerr <[email protected]>
ebe93fc
to
5a90891
Compare
Spacing fixes, and a simplification for the Bridge1 interface description. Re-order pool properties to describe in start -> end order. Signed-off-by: Jeremy Kerr <[email protected]>
The spec currenty requires this, but that may change. So, don't bind the dbus API to requiring contiguous EIDs. Signed-off-by: Jeremy Kerr <[email protected]>
…thod Signed-off-by: Jeremy Kerr <[email protected]>
Some failure paths log internally, others do not. Make this consistent, and do all logging inside the function. Signed-off-by: Jeremy Kerr <[email protected]>
Currently, we only allow bridge pool allocation through an AssignEndpoint call, as that is guaranteed to involve a Set Endpoint ID command, required to start the pool allocation process. LearnEndpoint is intended to never modify endpoint state, so we keep that as-is. SetupEndpoint has always been a convenience method, intended to do a LearnEndpoint if possible, or fall back to AssignEndpoint if not. Because of this, we are not making any assurances about preserving state with SetupEndpoint. With the new bridge support, the current distinction between AssignEndpoint (which will allocate a bridge pool) and SetupEndpoint (which will not) is a potential point of confusion. Instead, allow bridge allocations through SetupEndpoint. We have a new conditional path here: we try the LearnEndpoint (ie, a Get Endpoint ID, to see if we can use that EID) first, but add a new check to determine if this is a bridge EID type. Is so, we force the fallback to Set Endpoint ID, which will allow a bridge EID allocation too. Signed-off-by: Jeremy Kerr <[email protected]>
Currently, if an endpoint needs reassigning due to a bridge conflict, SetupEndpoint may publish an endpoint, then remove it. This is because we're calling get_endpoint_peer() for the Get Endpoint ID function (which adds the peer), but later check for bridge compatibility. Prevent this by open-coding the parts we need from get_endpoint_peer, and only performing the add once we have a valid peer. This requires a bit of re-work that is particular to bridge allocation in the SetupEndpoint case. Signed-off-by: Jeremy Kerr <[email protected]>
The if will return in all paths, no need for the `else`. Signed-off-by: Jeremy Kerr <[email protected]>
This shouldn't be a bitwise test. Signed-off-by: Jeremy Kerr <[email protected]>
Guarantee that we're not altering ctx/net data. Signed-off-by: Jeremy Kerr <[email protected]>
`net_learn` only specifies the usage scenario, not the behaviour. Signed-off-by: Jeremy Kerr <[email protected]>
Currently, we attempt to allocate bridge pool ranges of the size of our max pool configuration setting, and then trim after we know the requested pool size. If the max allocation is not available, we do not provide *any* EID range to the requesting bridge. However, it's entirely likely that the bridge will request a pool that is smaller than our maximum. We should not reject that allocation, as there is space available. Instead of insisting on allocating the max, just pre-allocate the largest space up to the max. When we then learn the bridge pool size, offer the allocation that we made. If this is smaller than the preallocation, we trim. If it is larger, we just offer what is allocated. This changes a failure case, where the tests expect the less-than-max allocation to fail. No need to preserve this behaviour, as we can actually offer a workable pool at this point. Signed-off-by: Jeremy Kerr <[email protected]>
Signed-off-by: Jeremy Kerr <[email protected]>
Signed-off-by: Jeremy Kerr <[email protected]>
5a90891
to
f4870bc
Compare
A few minor fixes based on @mkj's in-office review. |
Thank you for rebasing the PR with your br :)
Ack, would be better this way. |
No problem - we needed the rebase in order to resolve conflicts with 9711f0b. Next step is to do a little testing on hardware we have here, then we should be good to merge. Let me know (soon!) if you have any further alterations beforehand. |
Merged 🎉 - thank you for the contributions! |
Thank you for your help and guidance/reviews. |
This PR aims to introduce ALLOCATE_ENDPOINT_ID message support along with MCTP Bridge endpoint into the existing peer structure.