There have been many recent debates about centralized control planes vs central control. Our team has been working on abstracting stock Linux managed centralized control plane abstracted on Openflow for sometime and hence this topic is that much more interesting to us.
As Ivan rightly pointed out, implementing data-plane functionality like LLDP, ARP, BFD or STP on top of controller is not so practical. In this post, I would analyze each of these protocols and what makes it easy/hard for such implementations:
1) LLDP
Traditionally, LLDP is implemented in Data-plane and normally contains the info about MAC/PHY information, port/system names of neigbours and so on. This information is stored in Data-base and queried using SNMP or other management protocols to discover topology. There were no evident scalability issues. SDN controllers started using LLDP with Openflow packet-in/packet-out exchanges to find out topology and hence calculate routes between nodes. Why can’t such a scheme scale ? For a standard switch with 48-ports, we would need to process 48 LLDP packets per sec (if interval is 1s per LLDP packet). So for a very sizeable 100 switch topology we would have 4800 packets per second. Is that really a big deal for modern multi-threaded controller software running of top-end Intel boxes ? Our experience says “NO”. In our lab, the root-cause of scalability was that almost all available Openflow switches won’t process slow-path packet-in/outs well while doing TCAM flow modification, stats update etc. And this lets all hell break lose at controller’s side.
Winding back, controllers started using LLDP at the first place because there was no standard Openflow way to plumb LLDP topology information from the switches. But even with that, switch software will need to handle slow path messages as well as TCAM updates and what not in an efficient manner.
Bottom line – Well designed switch software along with Openflow spec modification would really solve this issue once and for all.
2) ARP
ARP is a tricky area and difficult to label as a DP only entity. Think what would happen if we ask the already over-burdened OF switches to do distributed ARP for implementing centralized VLANs spanning multi-switches. Not going to happen soon. AFAIK, ARP has been implemented as control-plane as well data-plane entity by different vendors. But it always requires close co-ordination between the two.
So, what exactly is the scalability concern of having ARP on a central controller ? For a conservative estimate of around 8k next-hops we would need to handle few thousand ARP request/responses per ARP timeout (which is normally ~ 60s range). This again is not a problem in itself.
So, what happens when switch (temporarily) stops sending ARP packet-in/outs due to issues mentioned in 1 ? Again no big problem, if you have segregated L3 and next-hop flows in separate flow-tables such that ARP changes only reflect in the next-hop table and there is no touching L3 (routes) flow table.
Finally there is one more concern : If next-hop is not resolved does the switch send line-rate traffic to controller ? Nope, the controller can always direct such misses to itself using a meter (OF1.3 feature). Per (unresolved) next-hop meters for send-to controller flows solved all scalability issues for us.
3) BFD
BFD has always been a data-plane entity. Many vendors also have ASIC support for BFD these days. If ONF can improve the OF spec to programmatically enable BFD on a given object of interest (port/tunnel) and send periodic updates on its status to controller, there should be no scaling issues.
4) STP
Firstly, anybody should be my guest if they are still using STP based design in their networks. Secondly, even if it is needed, Openflow has it covered by having different port-states which can be modified based on DP’s STP port-state and having it notified to controller.
In the next blog, we will try to analyze the scaling issues of centralized control plane from routing protocols perspective and what can be done with Openflow alone. We already know NSX-DLR and NSX-edge implement such functionality for sometime now.
Shout out to ONF – Please solve ARP, LLDP & BFD in upcoming Openflow spec releases.
OpenMUL Updates 