Interaction between ECN and anycast routing

 Interesting post on the NANOG list about ECN and anycast routing.  The summary is that the provider was using ECMP with the ToS byte which means the path selected after connection establishment was different than before because during connection establishment ECT is not set while after connection establishment, ECT is set and CE may be used.  

This seems like a big mistake for the provide that configured the entropy calculation for ECMP.  They should never have used the ECN bits portion of the ToS byte.  Most implementations that I'm aware of use 5-tuple only (IPDA, IPSA, IP Protocol, Src Port, Dst Port).

OSPF vs IS-IS

A recent discussion on the nanog mailing list brought up some issues on the OSPF vs IS-IS, some which I am aware of but some that I hadn't seen before.

• OSPF runs directly over IP.  IS-IS uses CLNS (connectionless mode network service) over the link layer.
• OSPFv3 must be used for IPv6, but IS-IS can be used for both v4 and v6.
• OSPF has a hierarchy of 2 (area, AS) while IS-IS is being extended in the IETF for up to 8 levels of hierarchy (levels 1 to 8).
• Those that can route CLNS is dying out (not needed in most cases, though).
• Most switches don't have a way to protect IS-IS making it easy for an attacker to kill the box.
• IS-IS frames don't have a DSCP for QoS.
• Most implementation of IS-IS do not contain a proper CLNS implementation, but just a bunch of hacks to get the CLNS header and 802.3 header on the packet.
• No standard way to carry jumbo frames in 802.3, so vendors don't interoperate.
• Due to lack of LSP rollover, ISIS is subject to an attack vector which is very difficult to troubleshoot and resolve.

Secondary addresses and multinetting

Secondary addresses and multinetting are often confused with one another.  Some vendors even have CLI commands for multinetting that use the term secondary address.

Multinetting

Multinetting refers to the configuration of multiple subnets on a given IP interface.  In days prior to VLANs and the use of VLSM (variable length subnet mask) this was used as a way to configure multiple Class C subnets on the same IP interface so that one could have > 254 hosts reachable from that router port.  This functionality continues to be used for legacy reasons, but in most modern networks multinetting has more or less been superceded by the use of classes addressing via VLSM and VLANs, each with its own subnet.

Secondary addresses

Secondary addresses refers to the configuration of multiple IP addresses from the same subnet on a given IP interface. 

This is used by some standards-based protocols such as VRRP (RFC 5798) where we have a virtual IP and an interface IP address on a given interface.  The virtual IP address is the one configured as the default gateway on hosts and the VRRP master responds to ARP requests for that address.  On the other hand, the interface IP address typically is owned just by a given router and is used to reach the router when using protocols such as ping and ssh.  If one wants the router to receive protocol packets on the virtual IP address, it needs to be configured as a secondary address.  Additionally, a router will respond to any other addresses configured as secondary (that it doesn't own) when it becomes master.

In some legacy networks, hosts may be configured with different default gateway addresses within the same subnet.  In that case, secondary addresses are required even without VRRP.

A collection of networking resources

Forwarding ASICs

• P4 and P4runtime
• NPL and SDK-LT
• Buffering in various products
• Switch comparison table

SDN

• Netconf and YANG
• Differences between NSX and ACI @ NetworkWorld
• SDN and network virtualization: a reality check @ NetworkWorld

Network monitoring/telemetry

• Network monitoring tools
• sflow, NetFlow, JFlow, IPFIX
• INT, IOAM
• Sonata -- query driven telemetry using P4
• sflow testing
• sflow test suite
• source code for the tests
• running the tests
• Telemetry at IETF 104 March 2019

Machine learning for networking

• Packet Pushers article

Traffic engineering

• National backbone with latency metric

BGP EVPN

• Cisco Live presentation -- covers unicast, multicast, vpc

Cloud

• QinQ and Microsoft Azure

Buffering

• Buffer sizing workshop

Books

• Computer Networking: A Systems Approach by Larry Peterson & Bruce Davie
• Software Defined Networks: A Systems Approach by Peterson et al.
• 5G Mobile Networks: A Systems Approach by Larry Peterson and Oguz Sunay
• High Performance Browser Networking by Ilya Grigorik

Container networking

• Project Calico datasheet
• Why Calico?
• 5 ways to design your container network
• Kubernetes for network engineers
• Arista EOS and Tigera Calico Technical brief
• Determine the best networking option with Calico
• Comparing Flannel, Calico, Canal, Weave

SmartNICs

• Azure SmartNIC

Time Synchronization

• IEEE tutorial on time sync in TSN (2021)

Transport congestion control and AQM

TCP

• Tahoe
• Reno
• NewReno
• Vegas
• SACK
• ECN
• BIC
• CUBIC
• DC-TCP
• BBR (overview)

The default window size in TCP is 64 KB (2^16).  Newer versions of TCP have a window scale option that allows the window to be scaled by 2^14, so the total window size is 2^30 or 1 GB.  However, some implementations only allow a limited window size.

RoCEv2

• Mellanox recommended configuration
• DCQCN (data center quantized congestion notification)
• Note that end station algorithm for reacting to congestion is not standardized
• PCN (photonic congestion notification)

QUIC

AQM 

• RED, WRED, RIO
• Drop/mark from front
• PIE, Fq-PIE (Proportional Integrated controller Enhanced)
• CoDel, Fq-CoDel (Controlled Delay)
• AFD (Approximate Fair Dropping) [Used with elephant trap and computing the fair rate for each elephant]

Ethernet frame encapsulations

Ethernet packets are referred to as frames.  There are 4 types of Ethernet encapsulations:

• Ethernet II: This is the version that came out of the original Ethernet specification from Digital-Intel-Xerox, commonly referred to as DIX.  This is the one that uses an Ethertype and is the most common format in use today.  [MAC DA | MAC SA | Etype | Payload | FCS] 
• IEEE 802.3 Raw: This was defined by IEEE 802.3.  It was most widely used by Novell IPX with the first two bytes of the payload being 0xFFFF. [MAC DA | MAC SA | Length | Payload | FCS]
• IEEE 802.3 LLC: This version used the service access points (DSAP/SSAP) to indicate the protocol in the frame. [MAC DA | MAC SA | Length | DSAP | SSAP | Control | Payload | FCS]
• IEEE 802.3 SNAP: This version was use by SNA and some IP implementations. [MAC DA | MAC SA | Length | DSAP=0xAA SSAP=0xAA Control=0x03 | OUI=00 00 00 | Type=Ethertype | Payload | FCS]

Frame size

The minimum frame size in Ethernet is 64 bytes (from the start of the MAC DA to the end of the FCS).

For length-encapsulated frames, the length field indicates the actual amount of data following the length field excluding any pad.  Valid values of the length field are 0 to 1500 bytes.  If the length is less than 46 bytes, then one or more bytes of pad are added to the frame to bring it up to 64 bytes.  A received frame that is smaller than 64 bytes is termed a runt.  Runts are usually discarded by the receiver.  Runts were more common in the days when Ethernet deployments used CSMA/CD, collisions being the main cause.

If the 2-byte following the MAC SA is 1536 or larger, then it is interpreted as an Ethertype.  Frames for which these 2 bytes are greater than 1500 but less than 1536 are non-standard frames and may be discarded by some switches.

The maximum payload size in IEEE 802.3 is 1500 bytes.  This is what follows the type/length field (and includes the DSAP, SSAP, etc. if used).  This makes the max frame size in Ethernet 1518 bytes (from start of MAC DA to end of FCS).  IEEE 802.3ac increase the max frame size to 1522 bytes when a VLAN tag is present; without the VLAN tag the max frame size is still 1518 bytes.  IEEE 802.3as increased the max frame size to 2000 bytes but specifies that the payload size is still 1500 bytes.  The additional bytes (500 of them in all) are used to accommodate headers define by 802 (e.g. MACsec, Provider Backbone Bridge aka MAC-in-MAC, etc.).

A frame that is larger than 1518 (or 1522 with a VLAN tag) is term a jumbo frame.  These are non-standard frames, but they are supported by almost all vendors of commercial networking equipment.  Jumbo frames are sometimes also referred to as giants.  Frames that are slightly over 1518/1522 bytes are sometimes called as baby giants.  Commonly supported jumbo frame sizes are 9 KB (9216 bytes) and 12 KB.  There are slight differences in the jumbo frame sizes supported by different vendors.

There was an attempt to create a project to standardize jumbo frames in IEEE 802.3, but the effort failed.  This is documented in draft-ietf-isis-ext-eth-01 (see the appendices at the end of the document).

On the wire frames also include an 8-byte-preamble, and 12-bytes of inter-frame gap.

Useful references

• A listing of Ethertypes.  Ethertypes are assigned by the IEEE RAC (Registration Authority Committee).
• A listing of OUIs.  OUIs are also assigned by the IEEE RAC.

Open source routing protocols, white boxes, and open source OSes

White boxes

• Accton/Edgecore
• Delta Networks
• Quanta Cloud Technology

Brite boxes

• Dell

OSes

• Amazon Dent OS
• AT&T dNOS (whitepaper)
• Big Switch Switch Light NOS (builds on Open Network Linux)
• Cumulus Linux
• Dell OS10 Base / OpenSwitch
• Facebook FBOSS
• Gigamon GigaVUE-OS
• IP Infusion OcNOS
• Indigo
• Juniper Junos
• Microsoft SONiC
• Open Network Linux
• Pica8 PicOS
• SnapRoute FlexSwitch
• Stratum

Routing protocols

• BIRD
• exaBGP
• FRR - Free Range Routing
• GoBGP
• OpenBGPD
• Quagga
• XORP

This presentation contains a nice overview of the various options.

In the early days there were a couple of other choices:

• GateD
• ZebOS

Useful references

• http://packetpushers.net/virtual-toolbox/list-network-operating-systems/
• https://www.networkworld.com/article/2919599/cisco-subnet/clearing-the-fog-around-open-switching-terminology.html

Resources for IEEE 802

The following is a collection of resources for IEEE 802 and related organizations.

• 802tsn.org - IEEE 802.1 TSN unofficial page.
• AVnu Alliance

Tutorial and other useful presentations

• 802.1AS
• Cloud scale data center design by Microsoft (contains cable length information)

IEEE 802.1 projects

The following is a list of IEEE 802 and IEEE 802.1 projects.  At some point I hope to add a few lines providing an executive summary of each project.  This list doesn't include projects that are revisions (-REV) and projects that are corrigenda (-Cor-x), except in cases where the revisions were actually adding significant new functionality.

IEEE 802 projects

• 802 - Overview and Architecture
• 802a - Playpen Ethertypes
• 802b - Registration of object identifiers (OIDs)
• 802c - Local MAC address usage
• 802d - Allocation of uniform resource name (URN) values
• 802E - Privacy considerations for 802 technologies

IEEE 802.1 projects

• 802.1B - LAN/MAN management
• 802.1D - MAC bridges
• 802.1E - System load protocol
• 802.1F - Common definitions and procedures for IEEE 802 management information
• 802.1G - Remote MAC bridging
• 802.1H - Bridging of Ethernet v2 to IEEE 802 LANs
• 802.1j - Managed objects for MAC bridges
• 802.1p - Expedited traffic classes and dynamic multicast filtering
• 802.1Q - MAC bridges and virtual bridged LANs (formerly virtual bridged LANs)
• See also IEEE 802.3ac
• 802.1s - Multiple spanning trees
• 802.1t - 802.1D maintenance
• 802.1u - 802.1Q maintenance
• 802.1v - VLAN classification by protocol and port
• 802.1w - Rapid reconfiguration (RSTP)
• 802.1X- Port-based network access control
• 802.1y - 802.1D maintenance
• 802.1z - 802.1Q maintenance
• 802.1aa - 802.1X maintenance
• 802.1AB - Link layer discovery protocol (LLDP)
• 802.1AC - MAC service definition
• 802.1ad - Provider bridging
• 802.1AE - MACsec
• 802.1af - MACsec key management
• 802.1ag - Connectivity fault management
• 802.1ah - Provider backbone bridging
• 802.1aj - Two-port MAC relay
• 802.1ak - Multiple registration protocol (MRP)
• 802.1ap - MIB definition for VLAN bridges
• 802.1aq - Shortest path bridging
• 802.1AR - Secure Device Identifier
• 802.1AS - Time Synchronization
• 802.1Qat - Stream reservation protocol (SRP)
• 802.1Qau - Congestion notification
• 802.1Qav - Credit-based shaper
• 802.1Qaw - Data driven connectivity fault management
• 802.1AX - Link aggregation
• 802.1AX-REV - Distributed resilient network interconnect (DRNI)
• 802.1Qay - Provider backbone bridging - traffic engineering
• 802.1Qaz - Enhanced transmission selection (ETS) and DCBX
• 802.1BA - AVB systems
• 802.1Qbb - Priority-based flow control (PFC)
• See also 802.3bd
• 802.1Qbc - Remote customer service interfaces (RCSI)
• 802.3bd - MAC control frame for PFC
• 802.1Qbe - Multiple I-SID registration protocol (MIRP)
• 802.1Qbf - PBB-TE segment protection
• 802.1Qbg - Edge virtual bridging (EVB)
• 802.1Qbh - Bridge port extenders (replaced by 802.1BR)
• 802.1AXbk - Link aggregation protocol addressing
• 802.1AEbn - 256-bit encryption
• 802.1Qbp - Equal cost multiple paths (ECMP)
• 802.1AXbq - Distributed resilient network interconnect (replaced by AX-REV)
• 802.1BR - Bridge port extenders
• 802.1ASbt - Time synchronization - Enhancement and performance improvements
• 802.1Qbu - Frame preemption
• See also 802.3br - Interspersed express traffic (IET)
• 802.1Qbv - Enhancements for scheduled traffic -- time-aware shaper
• 802.1AEbw - MACsec extended packet numbering
• 802.1Xbx- MACsec key agreement extensions
• 802.1ACby - Ethernet over MOST
• 802.1Qbz - Enhancements to bridging of 802.11
• 802.1Qca - Path control and reservation
• 802.1CB - Frame replication and elimination for reliability (FRER)
• 802.1Qcc - SRP enhancements and performance improvements
• 802.1Qcd - Application VLAN TLV
• 802.1ARce - Secure device identity - SHA-384 and P-384 elliptic curve
• 802.1CF - OmniRAN network reference model and functional description of access network
• 802.1AEcg - Ethernet data encryption devices
• 802.1Qch - Cyclic queueing and forwarding
• 802.1Qci - Per-stream filtering and policing
• 802.1Qcj - Automatic attachment to PBB services
• 802.1Xck - 802.1X YANG data model
• 802.1CM - TSN profile for fronthaul
• 802.1Qcn - VSI/VDP extensions for NVO3
• 802.1Qcp - 802.1Q YANG data model
• 802.1CQ - Multicast and local address assignment protocol
• 802.1Qcr - Asynchronous traffic shaping
• 802.1CS - Link-local registration protocol (LRP)
• 802.1AR-REV - SHA-384 and P-384 enhancements
• 802.1ACct - Support for 802.15.3
• 802.1ABcu - LLDP YANG
• 802.1CBcv - FRER YANG & MIB
• 802.1Qcw - TSN YANG for Qbu, Qcv, Qci
• 802.1Qcx - CFM YANG
• 802.1Qcy - EVB VDP extensions for NVO
• 802.1Qcz - Congestion isolation
• 802.1DA (60802) - Profile for industrial automation
• 802.1CBdb - Extended stream identification
• 802.1DC - QoS provisioning for non-bridge systems
• 802.1Qdd - Resource allocation protocol
• 802.1CMde - Enhancements for fronthaul interface
• 802.1DF - TSN profile for service provider networks
• 802.1DG - TSN profile for automotive networks
• 802.1ABdh - LLDPv2 - Support for Multiframe Protocol Data Units
• 802.1Qdj - TSN configuration enhancements
• 802.1AEdk - Privacy enhancements for use with MACsec
• 802f - Ethertype YANG
• 802.1ASdm - Hot standby
• 802.1ASdn - Time Synch YANG
• 802.1DP - TSN profile for aerospace
• 802.1Qdq - Tspec - Shaper parameter settings for bursty traffic requiring bounded latency
• 802.1ASdr - Time Synch inclusive terminology
• 802.1ASds - Time Synch support for half duplex MAC
• 802.1Qdt - PFC enhancements
• 802.1DU - Cut-through forwarding
• 802.1Qdv - Cyclic queueing and forwarding (CQF) enhancements
• 802.1Qdw - Source flow control
• 802.1Qdx - Credit shaper YANG
• 802.1Qdy - MSTP YANG

Networking cheat sheet

The following are some useful rules of thumb:

• Forwarding rate calculations for Ethernet
• Packets per second (pps) = link speed in bps / ((size+20)*8)
• Preamble + IFG is 20 bytes.
• Gbps x 1.5 = Mpps @ 64 bytes (actual value is 1.488).
• 1.5 Mpps @ 1 Gbps
• 15 Mpps @ 10 Gbps
• Latency
• 5 usec per km (one way).
• 10 usec per km (RTT).
• Speed of light in medium = 2x10^8 m/sec, i.e. 0.67 x c.
• 67.2 nsec @ 10G for 64 bytes (including preamble + IFG).
• Data in flight
• ~1300 bytes / 100 m @ 10 Gbps (RTT).
• Recommended default sampling rate for sflow
• 1 in (link speed in Gbps x 1000).
• 10G, 1 in 10,000
• 100G, 1 in 100,000
• 400G, 1 in 400,000
• Can use higher sampling rate if utilization is low.
• Backoff in sampling is not recommended.  Instead keep track of discards and report them.

Resources from standards organizations

The following are resources from various standards I have followed or participated in over the years.

• IEEE 802 
• IETF
• OCP wiki
• T11

Encapsulations for NVO3

There are several encapsulations that have been proposed for use with NVO3.  In no particular order, they are:

• NVGRE
• VXLAN
• STT
• VXLAN-GPE
• Geneve
• LISP (and its Layer 2 counterpart which is similar to VXLAN)
• GUE
• L2-over-MPLS (over GRE) -- L2VPN
• L3-over-MPLS (over GRE) -- L3VPN
• L2TPv3

Which ones survive is to be seen.

Decoding an IEEE 802 project number

IEEE SA has a number of standards committees, e.g. 802, 1588, etc.  IEEE 802 is one of the standards committees that is focused on the development of standards for LANs and MANs (local and metropolitan area networks).  Within IEEE 802,  802.1 is the HILI (higher layer interface) working group.  It deals with standards that are MAC layer independent.  In contrast, other working groups in the IEEE 802 work on MAC layer specific standards.  For example, 802.3 work on standards for Ethernet, 802.11 works on standards for WiFi, and 802.16 works on standards for WiMax.  IEEE 802.1 is also responsible for standards designated 802, indicating that they apply to all IEEE 802.1 working groups.  An example is the IEEE 802 Overview & Architecture standard.

In IEEE 802, projects and the subsequent standard are typically denoted IEEE 802.NXy.

• IEEE indicates that it is an IEEE standard.
• 802 indicates it was developed by the LAN/MAN Standards Committee.
• N is numeric and denotes the working group in which the standard was developed.  For instance, 802.1 indicates a standard developed in the "dot 1" working group which deals with MAC layer independent standards.  N is omitted only for standards that span all working groups which are typically architecture documents.
• X is one or more uppercase letters.  It is optional.  If present, it indicates the base standard.  Some working groups, e.g. IEEE 802.3, have just one big document.  In that case, the uppercase letters are not needed and are omitted.  In the case of IEEE 802.1, there are multiple base standards, e.g. IEEE 802.1D, IEEE 8021.Q, IEEE 802.1AB, etc.
• y is one or more lowercase letters.  Lowercase letters indicate that this project is an amendment to some base standard.  Prior to circa 2006, the base standard of the project was not clear because the concept of using X along with y was not in use; either X or y was used.  So, earlier, we had standards like 802.1p and 802.1ah (which indicated they were amendments to a base standard, but not the base standard to which they applied, which in these cases were 802.1D and 802.1Q respectively).  Later, X and y were used together so we now have projects such as 802.1Qbg and 802.1ASbt (which are amendments to 802.1Q and 802.1AS respectively).

Sometimes,  a project or a standard will simply be called IEEE 802.NX-REV, indicating that the project is a revision of a standard IEEE 802.1NX previously developed.  For example, we have projects such as 802.1Q-REV and 802.1AX-REV.

Finally, there are corrigenda projects which are called IEEE 802.1NX-Cor-M.  The 'Cor' indicates that it is a corrigendum.  The '-M', M being numeric, indicates the number of the corrigendum in case more than one is needed.

If a project is a new base document, it is published independently and remains an independent document forever.  If the project is an amendment, it is initially published as an independent document and then subsequently rolled into the base document when it is revised.  IEEE requires that revisions be done at a certain interval before new amendment projects are authorized.

Where to get IEEE 802 standards?

This post is specifically about standards by the IEEE 802 LAN/MAN Standards Committee.

All published IEEE 802 standards are available for free 6 months after they have been published via the IEEE Get Program.  To get access them within that 6 month period, they would have to be purchased.  One place to get them is Techstreet.  Techstreet also sells IEEE standards that were not developed by IEEE 802, e.g. IEEE 1588.

A CD/DVD containing all of the standards is provided to all attendees at IEEE 802 meetings for their personal use.

For unpublished standards and works in progress, you would have to go to the working group of interest which you can get to from the main IEEE website.  Unfortunately, accessing these will usually require a userid/password.  If you don't attend the meetings, you might be able to get that information from the chair of the working group, provided a reasonable justification is made.

Tutorials presented at the plenary meetings are archived here.

About this blog

The blog is a collection of information related to computer networking.  Specifically, it is information about various topics that I have been exposed to during my career.  I will also have posts that serve as a reference desk or collection of links to various pieces of information.

You can send me feedback either by leaving a comment or by emailing me.  My email address is available in my profile.