rXg LAGG configuration
January 27, 2025
Introduction
In many scenarios, a single physical link provides insufficient capacity to carry all the user traffic. Managing multiple separate physical links becomes quickly complicated, and even technically challenging, when multiple VLANs, prefixes, etc. need to be accessible across several physical interfaces. In such situations, the use of link aggregation (see https://en.wikipedia.org/wiki/Link_aggregation) comes in handy, providing the ability to add capacity to the system without affecting the logical configuration associated with the interface. The capacity increase is observed at the system level, whereby traffic from any single host / application is always hashed (directed) to one of the member links in the link aggregation interface, limiting the effective throughput for that host to whatever the throughput of the given physical link is. The member link selection mechanism is usually vendor-specific, but from the end-user perspective, the link aggregation interfaces guarantee packets within the given data stream are not re-ordered.
The added bonus? Link aggregation does provide inherent redundancy, whereby the failure of one of physical links does not affect the overall status of the given link, even though it does decrease its aggregate capacity. In the failure scenario, the link aggregation interface is self-healing and traffic is shifted away from the failed member link into one of the operating member links, providing the aforementioned redundancy.
Link aggregation configuration on rXg
The test rXg unit is running code 15.812 (latest stable build at the time of writing) and has an Intel X520 NIC with two 10GE ports, as shown below. Ports ix0 and ix1 will be used to build a link aggregation interface (LAGG) with IEEE 802.1AX Link Aggregation Control Protocol (LACP) enabled for load balancing and redundancy control. The use of LACP also provides much better interoperability with a variety of different switching / routing platforms.
rxg# pciconf -lv | grep -A1 -B3 network
em0@pci0:0:25:0: class=0x020000 rev=0x05 hdr=0x00 vendor=0x8086 device=0x153a subvendor=0x1043 subdevice=0x8535
vendor = 'Intel Corporation'
device = 'Ethernet Connection I217-LM'
class = network
subclass = ethernet
--
ix0@pci0:3:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x154d subvendor=0x8086 subdevice=0x7b11
vendor = 'Intel Corporation'
device = 'Ethernet 10G 2P X520 Adapter'
class = network
subclass = ethernet
ix1@pci0:3:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x154d subvendor=0x8086 subdevice=0x7b11
vendor = 'Intel Corporation'
device = 'Ethernet 10G 2P X520 Adapter'
class = network
subclass = ethernet
In this example, a Juniper QFX5100-48S-4C switch is used, running Junos 21.2X4.2 flex, providing a relatively new firmware baseline. The features used in this KB document have been in the code base for a long time and no functional changes are expected.
This switch is configured with two physical 10GE interfaces and an aggregated Ethernet (Juniper-speak for link aggregation) interface AE20 built with LACP (fast rate) enabled, as shown below. Port descriptions and selection of the ae interface number are arbitrary.
SW-QFX5100# show interfaces xe-0/0/32 | display set
set interfaces xe-0/0/32 description LINK-2-RXG-HOME-IXL0
set interfaces xe-0/0/32 ether-options 802.3ad ae20
{master:0}[edit]
SW-QFX5100# show interfaces xe-0/0/33 | display set
set interfaces xe-0/0/33 description LINK-2-RXG-HOME-IXL1
set interfaces xe-0/0/33 ether-options 802.3ad ae20
{master:0}[edit]
SW-QFX5100# show interfaces ae20 | display set
set interfaces ae20 description LINK-2-RXG-HOME
set interfaces ae20 mtu 9200
set interfaces ae20 aggregated-ether-options minimum-links 1
set interfaces ae20 aggregated-ether-options link-speed 10g
set interfaces ae20 aggregated-ether-options lacp active
set interfaces ae20 aggregated-ether-options lacp periodic fast
{master:0}[edit]
The rXg side configuration involves checking the status of both member interfaces (ix0 and ix1) to make sure they are up and connected, using the ‘Network::LAN::Ethernet Interfaces’ scaffold, as shown below:
With both ix0 and ix1 interfaces confirmed to be up and connected, LAGG instance is created using the ‘Network::LAN::Pseudo Interfaces’ scaffold, as shown below. Interfaces ix0 and ix1 are both tied into the newly created LAGG interface. LACP protocol is selected to maximize interoperability with a variety of different switching / routing platforms. IP configuration may be left unselected, if the LAGG interface is created as a L2 VLAN trunk.
Once created, the LAGG interface becomes displayed under the ‘Network::LAN::Pseudo Interfaces’ scaffold, as shown below. LAGG interface numbering is controlled by rXg and cannot be modified from the frontend.
The LAGG interface status can be also displayed from the rXg shell, as shown below.
rxg# ifconfig lagg1
lagg1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether a0:36:9f:13:9b:c2
hwaddr 00:00:00:00:00:00
inet 192.168.5.1 netmask 0xffffff00 broadcast 192.168.5.255
laggproto lacp lagghash l2,l3,l4
laggport: ix0 flags=0<>
laggport: ix1 flags=0<>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
On the Juniper QFX5100 side, ae20 is up, member interfaces are up, as shown below:
SW-QFX5100# run show interfaces descriptions | match RXG-HOME
xe-0/0/32 up up LINK-2-RXG-HOME-IXL0
xe-0/0/33 up up LINK-2-RXG-HOME-IXL1
ae20 up up LINK-2-RXG-HOME
{master:0}[edit]
SW-QFX5100# run show interfaces xe-0/0/32
Physical interface: xe-0/0/32, Enabled, Physical link is Up
Interface index: 679, SNMP ifIndex: 549
Description: LINK-2-RXG-HOME-IXL0
Link-level type: Ethernet, MTU: 9200, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None,
Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Media type: Fiber
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 12 supported, 12 maximum usable queues
Current address: e4:f2:7c:1a:49:e3, Hardware address: e4:f2:7c:1a:49:e3
Last flapped : 2025-01-20 20:54:08 MST (12:28:07 ago)
Input rate : 624 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
PCS statistics Seconds
Bit errors 0
Errored blocks 0
Ethernet FEC statistics Errors
FEC Corrected Errors 0
FEC Uncorrected Errors 0
FEC Corrected Errors Rate 0
FEC Uncorrected Errors Rate 0
PRBS Mode : Disabled
Interface transmit statistics: Disabled
{master:0}[edit]
SW-QFX5100# run show interfaces xe-0/0/33
Physical interface: xe-0/0/33, Enabled, Physical link is Up
Interface index: 680, SNMP ifIndex: 550
Description: LINK-2-RXG-HOME-IXL1
Link-level type: Ethernet, MTU: 9200, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None,
Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Media type: Fiber
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 12 supported, 12 maximum usable queues
Current address: e4:f2:7c:1a:49:e4, Hardware address: e4:f2:7c:1a:49:e4
Last flapped : 2025-01-20 18:36:32 MST (14:45:45 ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : None
Active defects : None
PCS statistics Seconds
Bit errors 0
Errored blocks 0
Ethernet FEC statistics Errors
FEC Corrected Errors 0
FEC Uncorrected Errors 0
FEC Corrected Errors Rate 0
FEC Uncorrected Errors Rate 0
PRBS Mode : Disabled
Interface transmit statistics: Disabled
{master:0}[edit]
SW-QFX5100# run show interfaces ae20
Physical interface: ae20, Enabled, Physical link is Up
Interface index: 681, SNMP ifIndex: 553
Description: LINK-2-RXG-HOME
Link-level type: Ethernet, MTU: 9200, Speed: 20Gbps, BPDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 1bps
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Current address: e4:f2:7c:1a:4a:86, Hardware address: e4:f2:7c:1a:4a:86
Last flapped : 2025-01-22 10:34:49 MST (00:10:37 ago)
Input rate : 1248 bps (0 pps)
Output rate : 4704 bps (5 pps)
Logical interface ae20.0 (Index 583) (SNMP ifIndex 554)
Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
Statistics Packets pps Bytes bps
Bundle:
Input : 0 0 0 0
Output: 1 0 46 0
Adaptive Statistics:
Adaptive Adjusts: 0
Adaptive Scans : 0
Adaptive Updates: 0
Protocol eth-switch, MTU: 9200
Flags: Trunk-Mode
{master:0}[edit]The operation of LACP between the switch and rXg can be further confirmed as follows
SW-QFX5100# run show lacp interfaces ae20
Aggregated interface: ae20
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/32 Actor No No Yes Yes Yes Yes Fast Active
xe-0/0/32 Partner No No Yes Yes Yes Yes Slow Active
xe-0/0/33 Actor No No Yes Yes Yes Yes Fast Active
xe-0/0/33 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/32 Current Slow periodic Collecting distributing
xe-0/0/33 Current Slow periodic Collecting distributing
{master:0}[edit]Note that there is no way to control the LACP packet rate from the rXg frontend, hence the “Slow” value in the “Timeout” column for the link Partner.
Word of a warning
There is also a way to create LAGG type interfaces directly from rXg shell, though this method is not recommended outside of troubleshooting activities, since the resulting interface entries are not properly propagated within the frontend database.
