CPU Overload on ZTE 5900E Series?

Recently we have encountered several customer cases, where we experienced CPU Overload on ZTE 5900E Series or other words –  ZTE 5900E series switch CPU vulnerability to protocol packet storms. Complains have been based on facts that ZTE 5900E  switch CPU is easily loaded with unnecessary protocol packets, which leads to CPU resources extinction and impact to normal services.

Normally we can differentiate two ZTE 5900E switch working modes. First one is pure Layer 2 mode. Second is Layer 3 routing mode. In layer 2 working mode, the switch does not have any interfaces with or without any IP addresses configured. This means that there is no need to involve control plane in processing of protocol packets, or routing other IP packets. Using these series switches as only layer 2 devices does not arise any problems and performance is as expected. To check L2 packet rate information, the following command must be issued.


ZXR10#show interface gei_1/5
gei_1/5 is down,  line protocol is down
  Description is none
  The port is optical
  Duplex full
  MTU 1500 bytes    BW 1000000 Kbits
  Last clearing of "show interface" counters never
     20 seconds input rate :                  0 Bps,                0 pps
     20 seconds output rate:                  0 Bps,                0 pps
  Interface peak rate  : 
    input                    0 Bps, output                    0 Bps 
  Interface utilization: input       0%,     output       0%
  Input:
    Packets      : 0                        Bytes     : 0                   
    Unicasts     : 0                        Multicasts: 0                    
    Broadcasts   : 0                        Undersize : 0                    
    Oversize     : 0                        CRC-ERROR : 0                   
    Dropped      : 0                        Fragments : 0                    
    Jabber       : 0                        MacRxErr  : 0                    
  Output:
    Packets      : 0                        Bytes     : 0                   
    Unicasts     : 0                        Multicasts: 0                    
    Broadcasts   : 0                        Collision : 0                   
    LateCollision: 0                   

Layer 3 routing mode is achieved when any VLAN interface is created. This means switch’s L3 protection mechanisms are engaged and Control Plane Security starts to monitor protocol packet sending rates. In this way, there are possibilities to control protocol packet rates in interfaces by setting protocol packets average or peak rates. If the protocol packets speed exceeds the peak rate, the switch will start to drop those packets – it costs some CPU to process. For example to check protocol packet rates the show protocol-protect token-buckets command could be issued.


ZXR10#show protocol-protect token-buckets gei_1/1
Device_num:  0    Port_num:      1
ProtocolName   CurTokens BktCap TokenSpeed PassedPkts RejectedPkts
--------------------------------------------------------------------------------
igmp            300       300    100        0          0             
icmp            300       300    100        0          0             
v6-na           300       300    100        0          0             
v6-ns           300       300    100        0          0             
v6-ra           300       300    100        0          0             
v6-rs           300       300    100        0          0             
v6-mld          300       300    100        0          0             
dhcp            300       300    100        0          0             
snmp            300       300    100        0          0             
arp             300       300    100        0          0             
udld            300       300    100        0          0             
vbas            300       300    100        0          0             
802.1X          300       300    100        0          0             
cfm             300       300    100        0          0             
lldp            300       300    100        0          0             
group mng       300       300    100        0          0             
zesr hello      300       300    100        0          0             
zesr flush      300       300    100        0          0             
bpdu            300       300    100        0          0  

The command protocol-packet-protect enable/disable is designed to globally turn on or turn off any protocol packet processing in switch. This means if disabled, then on any physical port incoming protocol packets will be not processed and the token-bucket counter is not counting packets destined to this switch. This could lead to problems if not taken seriously account. For example ARP messages will not build ARP table in switch and PC`s on access switch ports will not be able to reach switch’s gateway.

On the first sight Layer 3 working mode comes with a little bit of confusion. Creating any VLAN interface automatically all incoming traffic from all VLANS is being processed by CPU. This means even traffic in VLANS who does not have interface is processed in CPU. In other words, layer 2 VLAN traffic is not being transmitted transparently trough forwarding plane, but is instead processed by control plane – CPU.

To prevent this layer 2 protocol packet processing in CPU we need to configure Access Control List. This ACL contains rules that tell CPU what VLAN is truly with L3 Interface and needs processing in Control Plane, and rules that tell that in all other VLANS these specific protocol packets should be transmitted transparently – not sent to CPU. The following ACL demonstrates full protocol range from any VLAN being not sent to CPU (including ARP) but transparently forwarded in layer 2. If not configured all packets from VLAN 2 must be processed in Control Plane & CPU.


acl hybrid number 348
rule 1 permit ip any any any doutervlan 2 ingress any egress any 
rule 2 permit ip any any arp ingress any egress any switch-cpu-cancel 
rule 3 permit 0 any any ip ingress any egress any copy-cpu-cancel 
rule 4 permit 1 any any ip ingress any egress any copy-cpu-cancel 
rule 5 permit 2 any any ip ingress any egress any copy-cpu-cancel
rule 6 permit 3 any any ip ingress any egress any copy-cpu-cancel
rule 7 permit 4 any any ip ingress any egress any copy-cpu-cancel
rule 8 permit 5 any any ip ingress any egress any copy-cpu-cancel
rule 9 permit 6 any any ip ingress any egress any copy-cpu-cancel
rule 10 permit 7 any any ip ingress any egress any copy-cpu-cancel
rule 11 permit 8 any any ip ingress any egress any copy-cpu-cancel
rule 12 permit 9 any any ip ingress any egress any copy-cpu-cancel
rule 13 permit 10 any any ip ingress any egress any copy-cpu-cancel
rule 14 permit 11 any any ip ingress any egress any copy-cpu-cancel
rule 15 permit 12 any any ip ingress any egress any copy-cpu-cancel
rule 16 permit 13 any any ip ingress any egress any copy-cpu-cancel
rule 17 permit 14 any any ip ingress any egress any copy-cpu-cancel
rule 18 permit 15 any any ip ingress any egress any copy-cpu-cancel
rule 19 permit 16 any any ip ingress any egress any copy-cpu-cancel
rule 20 permit 17 any any ip ingress any egress any copy-cpu-cancel
rule 21 permit 18 any any ip ingress any egress any copy-cpu-cancel
rule 22 permit 19 any any ip ingress any egress any copy-cpu-cancel
rule 23 permit 20 any any ip ingress any egress any copy-cpu-cancel
rule 24 permit 21 any any ip ingress any egress any copy-cpu-cancel
rule 25 permit 22 any any ip ingress any egress any copy-cpu-cancel
rule 26 permit 23 any any ip ingress any egress any copy-cpu-cancel
rule 27 permit 24 any any ip ingress any egress any copy-cpu-cancel
rule 28 permit 25 any any ip ingress any egress any copy-cpu-cancel
rule 29 permit 26 any any ip ingress any egress any copy-cpu-cancel
rule 30 permit 27 any any ip ingress any egress any copy-cpu-cancel
rule 31 permit 28 any any ip ingress any egress any copy-cpu-cancel
rule 32 permit 29 any any ip ingress any egress any copy-cpu-cancel
rule 33 permit 30 any any ip ingress any egress any copy-cpu-cancel
rule 34 permit 31 any any ip ingress any egress any copy-cpu-cancel
rule 35 permit 32 any any ip ingress any egress any copy-cpu-cancel
rule 36 permit 33 any any ip ingress any egress any copy-cpu-cancel
rule 37 permit 34 any any ip ingress any egress any copy-cpu-cancel
rule 38 permit 35 any any ip ingress any egress any copy-cpu-cancel
rule 39 permit 36 any any ip ingress any egress any copy-cpu-cancel
rule 40 permit 37 any any ip ingress any egress any copy-cpu-cancel
rule 41 permit 38 any any ip ingress any egress any copy-cpu-cancel
rule 42 permit 39 any any ip ingress any egress any copy-cpu-cancel
rule 43 permit 40 any any ip ingress any egress any copy-cpu-cancel
rule 44 permit 41 any any ip ingress any egress any copy-cpu-cancel
rule 45 permit 42 any any ip ingress any egress any copy-cpu-cancel
rule 46 permit 43 any any ip ingress any egress any copy-cpu-cancel
rule 47 permit 44 any any ip ingress any egress any copy-cpu-cancel
rule 48 permit 45 any any ip ingress any egress any copy-cpu-cancel
rule 49 permit 46 any any ip ingress any egress any copy-cpu-cancel
rule 50 permit 47 any any ip ingress any egress any copy-cpu-cancel
rule 51 permit 48 any any ip ingress any egress any copy-cpu-cancel
rule 52 permit 49 any any ip ingress any egress any copy-cpu-cancel
rule 53 permit 50 any any ip ingress any egress any copy-cpu-cancel
rule 54 permit 51 any any ip ingress any egress any copy-cpu-cancel
rule 55 permit 52 any any ip ingress any egress any copy-cpu-cancel
rule 56 permit 53 any any ip ingress any egress any copy-cpu-cancel
rule 57 permit 54 any any ip ingress any egress any copy-cpu-cancel
rule 58 permit 55 any any ip ingress any egress any copy-cpu-cancel
rule 59 permit 56 any any ip ingress any egress any copy-cpu-cancel
rule 60 permit 57 any any ip ingress any egress any copy-cpu-cancel
rule 61 permit 58 any any ip ingress any egress any copy-cpu-cancel
rule 62 permit 59 any any ip ingress any egress any copy-cpu-cancel
rule 63 permit 60 any any ip ingress any egress any copy-cpu-cancel
rule 64 permit 61 any any ip ingress any egress any copy-cpu-cancel
rule 65 permit 62 any any ip ingress any egress any copy-cpu-cancel
rule 66 permit 63 any any ip ingress any egress any copy-cpu-cancel
rule 67 permit 64 any any ip ingress any egress any copy-cpu-cancel
rule 68 permit 65 any any ip ingress any egress any copy-cpu-cancel
rule 69 permit 66 any any ip ingress any egress any copy-cpu-cancel
rule 70 permit 67 any any ip ingress any egress any copy-cpu-cancel
rule 71 permit 68 any any ip ingress any egress any copy-cpu-cancel
rule 72 permit 69 any any ip ingress any egress any copy-cpu-cancel
rule 73 permit 70 any any ip ingress any egress any copy-cpu-cancel
rule 74 permit 71 any any ip ingress any egress any copy-cpu-cancel
rule 75 permit 72 any any ip ingress any egress any copy-cpu-cancel
rule 76 permit 73 any any ip ingress any egress any copy-cpu-cancel
rule 77 permit 74 any any ip ingress any egress any copy-cpu-cancel
rule 78 permit 75 any any ip ingress any egress any copy-cpu-cancel
rule 79 permit 76 any any ip ingress any egress any copy-cpu-cancel
rule 80 permit 77 any any ip ingress any egress any copy-cpu-cancel
rule 81 permit 78 any any ip ingress any egress any copy-cpu-cancel
rule 82 permit 79 any any ip ingress any egress any copy-cpu-cancel
rule 83 permit 80 any any ip ingress any egress any copy-cpu-cancel
rule 84 permit 81 any any ip ingress any egress any copy-cpu-cancel
rule 85 permit 82 any any ip ingress any egress any copy-cpu-cancel
rule 86 permit 83 any any ip ingress any egress any copy-cpu-cancel
rule 87 permit 84 any any ip ingress any egress any copy-cpu-cancel
rule 88 permit 85 any any ip ingress any egress any copy-cpu-cancel
rule 89 permit 86 any any ip ingress any egress any copy-cpu-cancel
rule 90 permit 87 any any ip ingress any egress any copy-cpu-cancel
rule 91 permit 88 any any ip ingress any egress any copy-cpu-cancel
rule 92 permit 89 any any ip ingress any egress any copy-cpu-cancel
rule 93 permit 90 any any ip ingress any egress any copy-cpu-cancel
rule 94 permit 91 any any ip ingress any egress any copy-cpu-cancel
rule 95 permit 92 any any ip ingress any egress any copy-cpu-cancel
rule 96 permit 93 any any ip ingress any egress any copy-cpu-cancel
rule 97 permit 94 any any ip ingress any egress any copy-cpu-cancel
rule 98 permit 95 any any ip ingress any egress any copy-cpu-cancel
rule 99 permit 96 any any ip ingress any egress any copy-cpu-cancel
rule 100 permit 97 any any ip ingress any egress any copy-cpu-cancel
rule 101 permit 98 any any ip ingress any egress any copy-cpu-cancel
rule 102 permit 99 any any ip ingress any egress any copy-cpu-cancel
rule 103 permit 100 any any ip ingress any egress any copy-cpu-cancel
rule 104 permit 101 any any ip ingress any egress any copy-cpu-cancel
rule 105 permit 102 any any ip ingress any egress any copy-cpu-cancel
rule 106 permit 103 any any ip ingress any egress any copy-cpu-cancel
rule 107 permit 104 any any ip ingress any egress any copy-cpu-cancel
rule 108 permit 105 any any ip ingress any egress any copy-cpu-cancel
rule 109 permit 106 any any ip ingress any egress any copy-cpu-cancel
rule 110 permit 107 any any ip ingress any egress any copy-cpu-cancel
rule 111 permit 108 any any ip ingress any egress any copy-cpu-cancel
rule 112 permit 109 any any ip ingress any egress any copy-cpu-cancel
rule 113 permit 110 any any ip ingress any egress any copy-cpu-cancel
rule 114 permit 111 any any ip ingress any egress any copy-cpu-cancel
rule 115 permit 112 any any ip ingress any egress any copy-cpu-cancel
rule 116 permit 113 any any ip ingress any egress any copy-cpu-cancel
rule 117 permit 114 any any ip ingress any egress any copy-cpu-cancel
rule 118 permit 115 any any ip ingress any egress any copy-cpu-cancel
rule 119 permit 116 any any ip ingress any egress any copy-cpu-cancel
rule 120 permit 117 any any ip ingress any egress any copy-cpu-cancel
rule 121 permit 118 any any ip ingress any egress any copy-cpu-cancel
rule 122 permit 119 any any ip ingress any egress any copy-cpu-cancel
rule 123 permit 120 any any ip ingress any egress any copy-cpu-cancel
rule 124 permit 121 any any ip ingress any egress any copy-cpu-cancel
rule 125 permit 122 any any ip ingress any egress any copy-cpu-cancel
rule 126 permit 123 any any ip ingress any egress any copy-cpu-cancel
rule 127 permit 124 any any ip ingress any egress any copy-cpu-cancel
rule 128 permit 125 any any ip ingress any egress any copy-cpu-cancel
rule 129 permit 126 any any ip ingress any egress any copy-cpu-cancel
rule 130 permit 127 any any ip ingress any egress any copy-cpu-cancel
rule 131 permit 128 any any ip ingress any egress any copy-cpu-cancel
rule 132 permit 129 any any ip ingress any egress any copy-cpu-cancel
rule 133 permit 130 any any ip ingress any egress any copy-cpu-cancel
rule 134 permit 131 any any ip ingress any egress any copy-cpu-cancel
rule 135 permit 132 any any ip ingress any egress any copy-cpu-cancel
rule 136 permit 133 any any ip ingress any egress any copy-cpu-cancel
rule 137 permit 134 any any ip ingress any egress any copy-cpu-cancel
rule 138 permit 135 any any ip ingress any egress any copy-cpu-cancel
rule 139 permit 136 any any ip ingress any egress any copy-cpu-cancel
rule 140 permit 137 any any ip ingress any egress any copy-cpu-cancel
rule 141 permit 138 any any ip ingress any egress any copy-cpu-cancel
rule 142 permit 139 any any ip ingress any egress any copy-cpu-cancel
rule 143 permit 140 any any ip ingress any egress any copy-cpu-cancel
rule 144 permit 141 any any ip ingress any egress any copy-cpu-cancel
rule 145 permit 142 any any ip ingress any egress any copy-cpu-cancel
rule 146 permit ip any any any ingress any egress any  

Posted in: