p4-apps@lists.p4.org

P4 Applications Working Group

View all threads

Telemetry Report 2.0 options

MS
Mickey Spiegel
Wed, Mar 11, 2020 9:12 PM

The new requirement from Cisco and CableLabs is the ability to coalesce
multiple reports into one packet. In addition we need to add Domain
Specific ID, Domain Specific Md Bits, and an extension mechanism for
arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be
restructured, separating fields common to all reports in the packet from
fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own
packet fragment or flow identifying information, the report header needs to
change from a small header that is followed by a packet fragment, to a
report header that encapsulates a packet fragment. With this in mind, I
suggest that we don't need a next protocol. All the contents are within the
reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

  1. The more compact format uses an "InnerProtocol" field to identify
    whether a packet fragment or domain specific extensions are included in the
    report. In order to get to those fields, the receiver of the report has to
    parse through variable optional metadata based on RepMdBits and Domain
    Specific Md Bits. If it does not understand how to parse those bits, then
    it cannot figure out where the packet fragment or domain specific
    extensions begin. This format adds only 4 bytes, compared to the Telemetry
    Report v1.0 format.
  2. Nested TLV structure. An implementation that does not support
    coalescing needs to generate two levels of TLVs, each with only one TLV,
    with two different lengths values. This format adds 8 bytes compared to the
    Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with
    the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order
    to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable
    Optional Metadata.
  4. Length of each report is limited to 1024 bytes.
  5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits
    each.
  6. Domain Specific Md Status has been dropped. Use Domain Specific
    Extensions if necessary.
  7. Inner Protocol:

0: None

1: Domain Specific Extensions
2: Ethernet Packet Fragment

3: IPv4 Packet Fragment

4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with
    the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order
    to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable
    Optional Metadata.
  4. Length of each report is limited to 1024 bytes. If we want to expand
    that to 16k, then DQFI would have to move down, squeezing RepMdBits and
    Domain Specific Md Bits to 28 bits, and either burn a separate type for
    each packet fragment protocol, or move Protocol to the 16 Reserved bits on
    the right.
  5. Domain Specific Md Status has been moved to Domain Specific
    Extensions.
  6. Type (inner)
    1. Packet Fragment
    2. Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information. In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet. In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports. There are two high level options for the packet structure: 1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format. 2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format. Compact Telemetry Report v2.0 proposal +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |InProt | Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || |D|Q|F|I| RepMdBits |Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment or Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each. 6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary. 7. Inner Protocol: 0: None 1: Domain Specific Extensions 2: Ethernet Packet Fragment 3: IPv4 Packet Fragment 4: IPv6 Packet Fragment Nested TLV Telemetry Report v2.0 proposal: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |D|Q|F|I| Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | RepMdBits | Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Prot | Length | Reserved | Report +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Rsvd | Length | Domain Specific Md Status | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right. 5. Domain Specific Md Status has been moved to Domain Specific Extensions. 6. Type (inner) 1. Packet Fragment 2. Domain Specific Extensions Any preferences? questions? comments? suggestions? Mickey
LJ
Lee, Jeongkeun
Thu, Mar 12, 2020 12:09 AM

Hi Mickey,

Thanks for suggesting the interesting options.
I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios.

hw_id and seq # fields are sharing 24 bits.
Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)?

Thanks,
JK


From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Sent: Wednesday, March 11, 2020 2:12 PM
To: p4-apps@lists.p4.org p4-apps@lists.p4.org
Subject: [P4-Apps] Telemetry Report 2.0 options

The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

  1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
  2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes.
  5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
  6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
  7. Inner Protocol:

0: None
1: Domain Specific Extensions
2: Ethernet Packet Fragment
3: IPv4 Packet Fragment
4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
  5. Domain Specific Md Status has been moved to Domain Specific Extensions.
  6. Type (inner)
    *  Packet Fragment
    *  Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Hi Mickey, Thanks for suggesting the interesting options. I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios. hw_id and seq # fields are sharing 24 bits. Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)? Thanks, JK ________________________________ From: P4-apps <p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.com> Sent: Wednesday, March 11, 2020 2:12 PM To: p4-apps@lists.p4.org <p4-apps@lists.p4.org> Subject: [P4-Apps] Telemetry Report 2.0 options The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information. In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet. In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports. There are two high level options for the packet structure: 1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format. 2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format. Compact Telemetry Report v2.0 proposal +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |InProt | Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || |D|Q|F|I| RepMdBits |Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment or Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each. 6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary. 7. Inner Protocol: 0: None 1: Domain Specific Extensions 2: Ethernet Packet Fragment 3: IPv4 Packet Fragment 4: IPv6 Packet Fragment Nested TLV Telemetry Report v2.0 proposal: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |D|Q|F|I| Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | RepMdBits | Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Prot | Length | Reserved | Report +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Rsvd | Length | Domain Specific Md Status | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right. 5. Domain Specific Md Status has been moved to Domain Specific Extensions. 6. Type (inner) * Packet Fragment * Domain Specific Extensions Any preferences? questions? comments? suggestions? Mickey
MS
Mickey Spiegel
Thu, Mar 12, 2020 12:49 AM

On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun jk.lee@intel.com wrote:

Hi Mickey,

Thanks for suggesting the interesting options.
I see that the major difference between the two lies in the co-existence
of Packet Fragment and Domain Specific Extensions. The compact option
treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh
and Randy comment on this based on their use case scenarios.

hw_id and seq # fields are sharing 24 bits.
Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)?

6 bits accommodates 16 line cards or stackable units * 4 hw_id values for
each. That is probably enough.  I only rounded up from 6 to 8 bits because
I did not think 20 bits vs 22 bits of sequence number is significant. We
can go either way.

Mickey

Thanks,
JK


From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey
Spiegel mspiegel@barefootnetworks.com
Sent: Wednesday, March 11, 2020 2:12 PM
To: p4-apps@lists.p4.org p4-apps@lists.p4.org
Subject: [P4-Apps] Telemetry Report 2.0 options

The new requirement from Cisco and CableLabs is the ability to coalesce
multiple reports into one packet. In addition we need to add Domain
Specific ID, Domain Specific Md Bits, and an extension mechanism for
arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be
restructured, separating fields common to all reports in the packet from
fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own
packet fragment or flow identifying information, the report header needs to
change from a small header that is followed by a packet fragment, to a
report header that encapsulates a packet fragment. With this in mind, I
suggest that we don't need a next protocol. All the contents are within the
reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

1. The more compact format uses an "InnerProtocol" field to identify
whether a packet fragment or domain specific extensions are included in the
report. In order to get to those fields, the receiver of the report has to
parse through variable optional metadata based on RepMdBits and Domain
Specific Md Bits. If it does not understand how to parse those bits, then
it cannot figure out where the packet fragment or domain specific
extensions begin. This format adds only 4 bytes, compared to the Telemetry
Report v1.0 format.
2. Nested TLV structure. An implementation that does not support
coalescing needs to generate two levels of TLVs, each with only one TLV,
with two different lengths values. This format adds 8 bytes compared to the
Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant
with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in
order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence
Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable
Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14
bits each.
6. Domain Specific Md Status has been dropped. Use Domain Specific
Extensions if necessary.
7. Inner Protocol:

0: None

1: Domain Specific Extensions
2: Ethernet Packet Fragment

3: IPv4 Packet Fragment

4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant
with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in
order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence
Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable
Optional Metadata.
4. Length of each report is limited to 1024 bytes. If we want to
expand that to 16k, then DQFI would have to move down, squeezing RepMdBits
and Domain Specific Md Bits to 28 bits, and either burn a separate type for
each packet fragment protocol, or move Protocol to the 16 Reserved bits on
the right.
5. Domain Specific Md Status has been moved to Domain Specific
Extensions.
6. Type (inner)
   1. Packet Fragment
   2. Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun <jk.lee@intel.com> wrote: > Hi Mickey, > > Thanks for suggesting the interesting options. > I see that the major difference between the two lies in the co-existence > of Packet Fragment and Domain Specific Extensions. The compact option > treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh > and Randy comment on this based on their use case scenarios. > > hw_id and seq # fields are sharing 24 bits. > Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)? > 6 bits accommodates 16 line cards or stackable units * 4 hw_id values for each. That is probably enough. I only rounded up from 6 to 8 bits because I did not think 20 bits vs 22 bits of sequence number is significant. We can go either way. Mickey > > Thanks, > JK > > ------------------------------ > *From:* P4-apps <p4-apps-bounces@lists.p4.org> on behalf of Mickey > Spiegel <mspiegel@barefootnetworks.com> > *Sent:* Wednesday, March 11, 2020 2:12 PM > *To:* p4-apps@lists.p4.org <p4-apps@lists.p4.org> > *Subject:* [P4-Apps] Telemetry Report 2.0 options > > The new requirement from Cisco and CableLabs is the ability to coalesce > multiple reports into one packet. In addition we need to add Domain > Specific ID, Domain Specific Md Bits, and an extension mechanism for > arbitrary domain specific information. > > In order to meet these requirements, the packet format needs to be > restructured, separating fields common to all reports in the packet from > fields that are specific to each report in the packet. > > In order to allow for multiple reports in a packet, each with its own > packet fragment or flow identifying information, the report header needs to > change from a small header that is followed by a packet fragment, to a > report header that encapsulates a packet fragment. With this in mind, I > suggest that we don't need a next protocol. All the contents are within the > reports. There is nothing that needs to come after the reports. > > There are two high level options for the packet structure: > > 1. The more compact format uses an "InnerProtocol" field to identify > whether a packet fragment or domain specific extensions are included in the > report. In order to get to those fields, the receiver of the report has to > parse through variable optional metadata based on RepMdBits and Domain > Specific Md Bits. If it does not understand how to parse those bits, then > it cannot figure out where the packet fragment or domain specific > extensions begin. This format adds only 4 bytes, compared to the Telemetry > Report v1.0 format. > 2. Nested TLV structure. An implementation that does not support > coalescing needs to generate two levels of TLVs, each with only one TLV, > with two different lengths values. This format adds 8 bytes compared to the > Telemetry Report v1.0 format. > > Compact Telemetry Report v2.0 proposal > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Ver | hw_id | Sequence Number | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Switch id | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > | Type |InProt | Length | Domain Specific ID | /\ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > |D|Q|F|I| RepMdBits |Reserved | Domain Specific Md Bits | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report > > | Variable Optional Metadata | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Packet Fragment or Domain Specific Extensions | \/ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > Notes: > > 1. There is no outer "length" field, since it is largely redundant > with the UDP and IP length fields. > 2. The Sequence Number has been reduced from 32 bits to 20 bits in > order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence > Number. > 3. Ingress Timestamp gets a RepMdBit value and moves into Variable > Optional Metadata. > 4. Length of each report is limited to 1024 bytes. > 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 > bits each. > 6. Domain Specific Md Status has been dropped. Use Domain Specific > Extensions if necessary. > 7. Inner Protocol: > > 0: None > > 1: Domain Specific Extensions > 2: Ethernet Packet Fragment > > 3: IPv4 Packet Fragment > > 4: IPv6 Packet Fragment > > > Nested TLV Telemetry Report v2.0 proposal: > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Ver | hw_id | Sequence Number | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Switch id | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > | Type |D|Q|F|I| Length | Domain Specific ID | /\ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | RepMdBits | Reserved | Domain Specific Md Bits | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Variable Optional Metadata | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Type | Prot | Length | Reserved | Report > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Packet Fragment | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Type | Rsvd | Length | Domain Specific Md Status | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Domain Specific Extensions | \/ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > > Notes: > > > 1. There is no outer "length" field, since it is largely redundant > with the UDP and IP length fields. > 2. The Sequence Number has been reduced from 32 bits to 20 bits in > order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence > Number. > 3. Ingress Timestamp gets a RepMdBit value and moves into Variable > Optional Metadata. > 4. Length of each report is limited to 1024 bytes. If we want to > expand that to 16k, then DQFI would have to move down, squeezing RepMdBits > and Domain Specific Md Bits to 28 bits, and either burn a separate type for > each packet fragment protocol, or move Protocol to the 16 Reserved bits on > the right. > 5. Domain Specific Md Status has been moved to Domain Specific > Extensions. > 6. Type (inner) > 1. Packet Fragment > 2. Domain Specific Extensions > > Any preferences? questions? comments? suggestions? > > Mickey > >
RS
Ramesh Sivakolundu (sramesh)
Thu, Mar 12, 2020 4:31 PM

Thanks Mickey for coming up with the format options.

From my perspective, Nested TLV is flexible and address all know use cases. The only comment I have is that Instruction Bitmap in the INT Header is 16 bits and Domain Specific Instruction is 16 bits. RepMD bits and Domain Specific MD bits add up to a max of 28 bits (without any reserved bits) in Compact format and 32-bits in Nested TLV format. Does it make sense to move the Domain Specific MD bits to Domain Specific extensions, similar to Domain Specific Md Status.

Thoughts?

-Ramesh

From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Date: Wednesday, March 11, 2020 at 5:50 PM
To: "Lee, Jeongkeun" jk.lee@intel.com
Cc: "p4-apps@lists.p4.org" p4-apps@lists.p4.org, "Alvarez, Daniel A" daniel.a.alvarez@intel.com
Subject: Re: [P4-Apps] Telemetry Report 2.0 options

On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun <jk.lee@intel.commailto:jk.lee@intel.com> wrote:
Hi Mickey,

Thanks for suggesting the interesting options.
I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios.

hw_id and seq # fields are sharing 24 bits.
Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)?

6 bits accommodates 16 line cards or stackable units * 4 hw_id values for each. That is probably enough.  I only rounded up from 6 to 8 bits because I did not think 20 bits vs 22 bits of sequence number is significant. We can go either way.

Mickey

Thanks,
JK


From: P4-apps <p4-apps-bounces@lists.p4.orgmailto:p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.commailto:mspiegel@barefootnetworks.com>
Sent: Wednesday, March 11, 2020 2:12 PM
To: p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org <p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org>
Subject: [P4-Apps] Telemetry Report 2.0 options

The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

  1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
  2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.
    Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes.
  5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
  6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
  7. Inner Protocol:
    0: None
    1: Domain Specific Extensions
    2: Ethernet Packet Fragment
    3: IPv4 Packet Fragment
    4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
  5. Domain Specific Md Status has been moved to Domain Specific Extensions.
  6. Type (inner)
 *   Packet Fragment
 *   Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Thanks Mickey for coming up with the format options. From my perspective, Nested TLV is flexible and address all know use cases. The only comment I have is that Instruction Bitmap in the INT Header is 16 bits and Domain Specific Instruction is 16 bits. RepMD bits and Domain Specific MD bits add up to a max of 28 bits (without any reserved bits) in Compact format and 32-bits in Nested TLV format. Does it make sense to move the Domain Specific MD bits to Domain Specific extensions, similar to Domain Specific Md Status. Thoughts? -Ramesh From: P4-apps <p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.com> Date: Wednesday, March 11, 2020 at 5:50 PM To: "Lee, Jeongkeun" <jk.lee@intel.com> Cc: "p4-apps@lists.p4.org" <p4-apps@lists.p4.org>, "Alvarez, Daniel A" <daniel.a.alvarez@intel.com> Subject: Re: [P4-Apps] Telemetry Report 2.0 options On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun <jk.lee@intel.com<mailto:jk.lee@intel.com>> wrote: Hi Mickey, Thanks for suggesting the interesting options. I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios. hw_id and seq # fields are sharing 24 bits. Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)? 6 bits accommodates 16 line cards or stackable units * 4 hw_id values for each. That is probably enough. I only rounded up from 6 to 8 bits because I did not think 20 bits vs 22 bits of sequence number is significant. We can go either way. Mickey Thanks, JK ________________________________ From: P4-apps <p4-apps-bounces@lists.p4.org<mailto:p4-apps-bounces@lists.p4.org>> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.com<mailto:mspiegel@barefootnetworks.com>> Sent: Wednesday, March 11, 2020 2:12 PM To: p4-apps@lists.p4.org<mailto:p4-apps@lists.p4.org> <p4-apps@lists.p4.org<mailto:p4-apps@lists.p4.org>> Subject: [P4-Apps] Telemetry Report 2.0 options The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information. In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet. In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports. There are two high level options for the packet structure: 1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format. 2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format. Compact Telemetry Report v2.0 proposal +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |InProt | Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || |D|Q|F|I| RepMdBits |Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment or Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each. 6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary. 7. Inner Protocol: 0: None 1: Domain Specific Extensions 2: Ethernet Packet Fragment 3: IPv4 Packet Fragment 4: IPv6 Packet Fragment Nested TLV Telemetry Report v2.0 proposal: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |D|Q|F|I| Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | RepMdBits | Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Prot | Length | Reserved | Report +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Rsvd | Length | Domain Specific Md Status | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right. 5. Domain Specific Md Status has been moved to Domain Specific Extensions. 6. Type (inner) * Packet Fragment * Domain Specific Extensions Any preferences? questions? comments? suggestions? Mickey
RL
Randy Levensalor
Thu, Mar 12, 2020 5:12 PM

Hi Mickey,

Thanks for pulling together these two proposals.

It looks like both options will address our immediate need to collate multiple packet fragments into a single report.  The 1024 size limitation would not be an issue for the use cases which we are exploring.

While we are not currently sending any domain specific TR data, just domain specific data in the INT report.  I do see option where we would send some additional domain specific data from the switch/sink generating the TR.  For instance, we may leverage some counters or include the current BGP flowspec rules.  If this domain specific data doesn’t need to be repeated on each report within the packet, so both option will work.

I like the versatility of the Nested TLV Telemetry Report, since it allows for both domain specific metadata and the packet fragment to be in one report.  I don’t have a compelling use for both domain specific and packet fragment info per report at this time.

Thanks,
Randy Levensalor

From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Date: Wednesday, March 11, 2020 at 6:50 PM
To: "Lee, Jeongkeun" jk.lee@intel.com
Cc: "p4-apps@lists.p4.org" p4-apps@lists.p4.org, "Alvarez, Daniel A" daniel.a.alvarez@intel.com
Subject: Re: [P4-Apps] Telemetry Report 2.0 options

CableLabs WARNING: The sender of this email could not be validated and may not match the person in the "From" field.

On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun <jk.lee@intel.commailto:jk.lee@intel.com> wrote:
Hi Mickey,

Thanks for suggesting the interesting options.
I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios.

hw_id and seq # fields are sharing 24 bits.
Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)?

6 bits accommodates 16 line cards or stackable units * 4 hw_id values for each. That is probably enough.  I only rounded up from 6 to 8 bits because I did not think 20 bits vs 22 bits of sequence number is significant. We can go either way.

Mickey

Thanks,
JK


From: P4-apps <p4-apps-bounces@lists.p4.orgmailto:p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.commailto:mspiegel@barefootnetworks.com>
Sent: Wednesday, March 11, 2020 2:12 PM
To: p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org <p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org>
Subject: [P4-Apps] Telemetry Report 2.0 options

The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

  1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
  2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.
    Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes.
  5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
  6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
  7. Inner Protocol:
    0: None
    1: Domain Specific Extensions
    2: Ethernet Packet Fragment
    3: IPv4 Packet Fragment
    4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
  5. Domain Specific Md Status has been moved to Domain Specific Extensions.
  6. Type (inner)
 *   Packet Fragment
 *   Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Hi Mickey, Thanks for pulling together these two proposals. It looks like both options will address our immediate need to collate multiple packet fragments into a single report. The 1024 size limitation would not be an issue for the use cases which we are exploring. While we are not currently sending any domain specific TR data, just domain specific data in the INT report. I do see option where we would send some additional domain specific data from the switch/sink generating the TR. For instance, we may leverage some counters or include the current BGP flowspec rules. If this domain specific data doesn’t need to be repeated on each report within the packet, so both option will work. I like the versatility of the Nested TLV Telemetry Report, since it allows for both domain specific metadata and the packet fragment to be in one report. I don’t have a compelling use for both domain specific and packet fragment info per report at this time. Thanks, Randy Levensalor From: P4-apps <p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.com> Date: Wednesday, March 11, 2020 at 6:50 PM To: "Lee, Jeongkeun" <jk.lee@intel.com> Cc: "p4-apps@lists.p4.org" <p4-apps@lists.p4.org>, "Alvarez, Daniel A" <daniel.a.alvarez@intel.com> Subject: Re: [P4-Apps] Telemetry Report 2.0 options CableLabs WARNING: The sender of this email could not be validated and may not match the person in the "From" field. On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun <jk.lee@intel.com<mailto:jk.lee@intel.com>> wrote: Hi Mickey, Thanks for suggesting the interesting options. I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios. hw_id and seq # fields are sharing 24 bits. Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)? 6 bits accommodates 16 line cards or stackable units * 4 hw_id values for each. That is probably enough. I only rounded up from 6 to 8 bits because I did not think 20 bits vs 22 bits of sequence number is significant. We can go either way. Mickey Thanks, JK ________________________________ From: P4-apps <p4-apps-bounces@lists.p4.org<mailto:p4-apps-bounces@lists.p4.org>> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.com<mailto:mspiegel@barefootnetworks.com>> Sent: Wednesday, March 11, 2020 2:12 PM To: p4-apps@lists.p4.org<mailto:p4-apps@lists.p4.org> <p4-apps@lists.p4.org<mailto:p4-apps@lists.p4.org>> Subject: [P4-Apps] Telemetry Report 2.0 options The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information. In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet. In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports. There are two high level options for the packet structure: 1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format. 2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format. Compact Telemetry Report v2.0 proposal +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |InProt | Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || |D|Q|F|I| RepMdBits |Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment or Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each. 6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary. 7. Inner Protocol: 0: None 1: Domain Specific Extensions 2: Ethernet Packet Fragment 3: IPv4 Packet Fragment 4: IPv6 Packet Fragment Nested TLV Telemetry Report v2.0 proposal: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |D|Q|F|I| Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | RepMdBits | Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Prot | Length | Reserved | Report +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Rsvd | Length | Domain Specific Md Status | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right. 5. Domain Specific Md Status has been moved to Domain Specific Extensions. 6. Type (inner) * Packet Fragment * Domain Specific Extensions Any preferences? questions? comments? suggestions? Mickey
MS
Mickey Spiegel
Fri, Mar 13, 2020 6:00 PM

The outcome of today's P4 apps meeting.

There are two high level options for the packet structure, both will be
specified in Telemetry Report v2.0:

  1. The more compact format uses an "InnerProtocol" field to identify
    whether a packet fragment or domain specific extensions are included in the
    report. In order to get to those fields, the receiver of the report has to
    parse through variable optional metadata based on RepMdBits and Domain
    Specific Md Bits. If it does not understand how to parse those bits, then
    it cannot figure out where the packet fragment or domain specific
    extensions begin. This format adds only 4 bytes, compared to the Telemetry
    Report v1.0 format.
  2. Nested TLV structure. An implementation that does not support
    coalescing needs to generate two levels of TLVs, each with only one TLV,
    with two different lengths values. This format adds 8 bytes compared to the
    Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |  hw_id  |              Sequence Number              |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                            Node id                            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type 1|InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I| Rsvd  |          RepMdBits          |  DS Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with
    the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 22 bits in order
    to save 4 bytes.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable
    Optional Metadata.
  4. Length of each report is limited to 1024 bytes.
  5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits
    each.
  6. Domain Specific Md Status has been dropped. Use Domain Specific
    Extensions if necessary.
  7. Inner Protocol:

0: None

1: Domain Specific Extensions

2: Ethernet Packet Fragment

3: IPv4 Packet Fragment

4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |  hw_id  |              Sequence Number              |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                            Node id                            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+        --

| Type 2|D|Q|F|I| Report Length |      Domain Specific ID      |        /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Rsvd  |    Length    |          RepMdBits          |  /\    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                  Variable Optional Metadata                  |  /    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Prot  |    Length    |          Reserved            |  /
Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                        Packet Fragment                        |  /    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Rsvd  |    Length    |    Domain Specific Md Bits    |  /\    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                  Domain Specific Extensions                  |  /    /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    --

Notes:

  1. There is no outer "length" field, since it is largely redundant with
    the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 22 bits in order
    to save 4 bytes.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable
    Optional Metadata.
  4. Length of each report is limited to 1024 bytes. If we want to expand
    that to 16k, then DQFI would have to move down, squeezing RepMdBits and
    Domain Specific Md Bits to 28 bits, and either burn a separate type for
    each packet fragment protocol, or move Protocol to the 16 Reserved bits on
    the right.
  5. Domain Specific Md Status has been moved to Domain Specific
    Extensions.
  6. Type (inner)
    1. Report Metadata
    2. Packet Fragment
    3. Domain Specific Extensions

On Wed, Mar 11, 2020 at 2:12 PM Mickey Spiegel <
mspiegel@barefootnetworks.com> wrote:

The new requirement from Cisco and CableLabs is the ability to coalesce
multiple reports into one packet. In addition we need to add Domain
Specific ID, Domain Specific Md Bits, and an extension mechanism for
arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be
restructured, separating fields common to all reports in the packet from
fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own
packet fragment or flow identifying information, the report header needs to
change from a small header that is followed by a packet fragment, to a
report header that encapsulates a packet fragment. With this in mind, I
suggest that we don't need a next protocol. All the contents are within the
reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

1. The more compact format uses an "InnerProtocol" field to identify
whether a packet fragment or domain specific extensions are included in the
report. In order to get to those fields, the receiver of the report has to
parse through variable optional metadata based on RepMdBits and Domain
Specific Md Bits. If it does not understand how to parse those bits, then
it cannot figure out where the packet fragment or domain specific
extensions begin. This format adds only 4 bytes, compared to the Telemetry
Report v1.0 format.
2. Nested TLV structure. An implementation that does not support
coalescing needs to generate two levels of TLVs, each with only one TLV,
with two different lengths values. This format adds 8 bytes compared to the
Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant
with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in
order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence
Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable
Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14
bits each.
6. Domain Specific Md Status has been dropped. Use Domain Specific
Extensions if necessary.
7. Inner Protocol:

0: None

1: Domain Specific Extensions
2: Ethernet Packet Fragment

3: IPv4 Packet Fragment

4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant
with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in
order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence
Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable
Optional Metadata.
4. Length of each report is limited to 1024 bytes. If we want to
expand that to 16k, then DQFI would have to move down, squeezing RepMdBits
and Domain Specific Md Bits to 28 bits, and either burn a separate type for
each packet fragment protocol, or move Protocol to the 16 Reserved bits on
the right.
5. Domain Specific Md Status has been moved to Domain Specific
Extensions.
6. Type (inner)
   1. Packet Fragment
   2. Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

The outcome of today's P4 apps meeting. There are two high level options for the packet structure, both will be specified in Telemetry Report v2.0: 1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format. 2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format. Compact Telemetry Report v2.0 proposal +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Node id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type 1|InProt | Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || |D|Q|F|I| Rsvd | RepMdBits | DS Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment or Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each. 6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary. 7. Inner Protocol: 0: None 1: Domain Specific Extensions 2: Ethernet Packet Fragment 3: IPv4 Packet Fragment 4: IPv6 Packet Fragment Nested TLV Telemetry Report v2.0 proposal: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Node id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type 2|D|Q|F|I| Report Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- || | Type | Rsvd | Length | RepMdBits | /\ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || || | Variable Optional Metadata | \/ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- || | Type | Prot | Length | Reserved | /\ Report +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || || | Packet Fragment | \/ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- || | Type | Rsvd | Length | Domain Specific Md Bits | /\ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || || | Domain Specific Extensions | \/ \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right. 5. Domain Specific Md Status has been moved to Domain Specific Extensions. 6. Type (inner) 1. Report Metadata 2. Packet Fragment 3. Domain Specific Extensions On Wed, Mar 11, 2020 at 2:12 PM Mickey Spiegel < mspiegel@barefootnetworks.com> wrote: > The new requirement from Cisco and CableLabs is the ability to coalesce > multiple reports into one packet. In addition we need to add Domain > Specific ID, Domain Specific Md Bits, and an extension mechanism for > arbitrary domain specific information. > > In order to meet these requirements, the packet format needs to be > restructured, separating fields common to all reports in the packet from > fields that are specific to each report in the packet. > > In order to allow for multiple reports in a packet, each with its own > packet fragment or flow identifying information, the report header needs to > change from a small header that is followed by a packet fragment, to a > report header that encapsulates a packet fragment. With this in mind, I > suggest that we don't need a next protocol. All the contents are within the > reports. There is nothing that needs to come after the reports. > > There are two high level options for the packet structure: > > 1. The more compact format uses an "InnerProtocol" field to identify > whether a packet fragment or domain specific extensions are included in the > report. In order to get to those fields, the receiver of the report has to > parse through variable optional metadata based on RepMdBits and Domain > Specific Md Bits. If it does not understand how to parse those bits, then > it cannot figure out where the packet fragment or domain specific > extensions begin. This format adds only 4 bytes, compared to the Telemetry > Report v1.0 format. > 2. Nested TLV structure. An implementation that does not support > coalescing needs to generate two levels of TLVs, each with only one TLV, > with two different lengths values. This format adds 8 bytes compared to the > Telemetry Report v1.0 format. > > Compact Telemetry Report v2.0 proposal > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Ver | hw_id | Sequence Number | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Switch id | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > | Type |InProt | Length | Domain Specific ID | /\ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > |D|Q|F|I| RepMdBits |Reserved | Domain Specific Md Bits | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report > > | Variable Optional Metadata | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Packet Fragment or Domain Specific Extensions | \/ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > Notes: > > 1. There is no outer "length" field, since it is largely redundant > with the UDP and IP length fields. > 2. The Sequence Number has been reduced from 32 bits to 20 bits in > order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence > Number. > 3. Ingress Timestamp gets a RepMdBit value and moves into Variable > Optional Metadata. > 4. Length of each report is limited to 1024 bytes. > 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 > bits each. > 6. Domain Specific Md Status has been dropped. Use Domain Specific > Extensions if necessary. > 7. Inner Protocol: > > 0: None > > 1: Domain Specific Extensions > 2: Ethernet Packet Fragment > > 3: IPv4 Packet Fragment > > 4: IPv6 Packet Fragment > > > Nested TLV Telemetry Report v2.0 proposal: > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Ver | hw_id | Sequence Number | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > | Switch id | > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > | Type |D|Q|F|I| Length | Domain Specific ID | /\ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | RepMdBits | Reserved | Domain Specific Md Bits | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Variable Optional Metadata | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Type | Prot | Length | Reserved | Report > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Packet Fragment | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Type | Rsvd | Length | Domain Specific Md Status | || > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || > > | Domain Specific Extensions | \/ > > +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- > > > Notes: > > > 1. There is no outer "length" field, since it is largely redundant > with the UDP and IP length fields. > 2. The Sequence Number has been reduced from 32 bits to 20 bits in > order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence > Number. > 3. Ingress Timestamp gets a RepMdBit value and moves into Variable > Optional Metadata. > 4. Length of each report is limited to 1024 bytes. If we want to > expand that to 16k, then DQFI would have to move down, squeezing RepMdBits > and Domain Specific Md Bits to 28 bits, and either burn a separate type for > each packet fragment protocol, or move Protocol to the 16 Reserved bits on > the right. > 5. Domain Specific Md Status has been moved to Domain Specific > Extensions. > 6. Type (inner) > 1. Packet Fragment > 2. Domain Specific Extensions > > Any preferences? questions? comments? suggestions? > > Mickey > >
RL
Randy Levensalor
Wed, Apr 8, 2020 8:43 PM

Hi Mickey, Et al.,

I haven’t seen any updates adding this to a PR or follow-up meetings.

Is there a plan to close on this any time soon?

Can I help?  Should I add this to my PR for multiple reports per packet or will this be added to another PR?

Many Thanks and looking forward to the release of the 2.0 Telemetry Report.

Randy Levensalor

From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Date: Friday, March 13, 2020 at 12:01 PM
To: "p4-apps@lists.p4.org" p4-apps@lists.p4.org
Subject: Re: [P4-Apps] Telemetry Report 2.0 options

CableLabs WARNING: The sender of this email could not be validated and may not match the person in the "From" field.

The outcome of today's P4 apps meeting.

There are two high level options for the packet structure, both will be specified in Telemetry Report v2.0:

  1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
  2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |  hw_id  |              Sequence Number              |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                            Node id                            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type 1|InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I| Rsvd  |          RepMdBits          |  DS Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes.
  5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
  6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
  7. Inner Protocol:

0: None

1: Domain Specific Extensions

2: Ethernet Packet Fragment

3: IPv4 Packet Fragment

4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |  hw_id  |              Sequence Number              |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                            Node id                            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+        --

| Type 2|D|Q|F|I| Report Length |      Domain Specific ID      |        /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Rsvd  |    Length    |          RepMdBits          |  /\    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                  Variable Optional Metadata                  |  /    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Prot  |    Length    |          Reserved            |  /\  Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                        Packet Fragment                        |  /    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Rsvd  |    Length    |    Domain Specific Md Bits    |  /\    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                  Domain Specific Extensions                  |  /    /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.

  2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes.

  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.

  4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.

  5. Domain Specific Md Status has been moved to Domain Specific Extensions.

  6. Type (inner)

 *   Report Metadata
 *   Packet Fragment
 *   Domain Specific Extensions

On Wed, Mar 11, 2020 at 2:12 PM Mickey Spiegel <mspiegel@barefootnetworks.commailto:mspiegel@barefootnetworks.com> wrote:
The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

  1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
  2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.
    Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes.
  5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
  6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
  7. Inner Protocol:
    0: None
    1: Domain Specific Extensions
    2: Ethernet Packet Fragment
    3: IPv4 Packet Fragment
    4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

  1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
  2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
  3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
  4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
  5. Domain Specific Md Status has been moved to Domain Specific Extensions.
  6. Type (inner)
 *   Packet Fragment
 *   Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Hi Mickey, Et al., I haven’t seen any updates adding this to a PR or follow-up meetings. Is there a plan to close on this any time soon? Can I help? Should I add this to my PR for multiple reports per packet or will this be added to another PR? Many Thanks and looking forward to the release of the 2.0 Telemetry Report. Randy Levensalor From: P4-apps <p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.com> Date: Friday, March 13, 2020 at 12:01 PM To: "p4-apps@lists.p4.org" <p4-apps@lists.p4.org> Subject: Re: [P4-Apps] Telemetry Report 2.0 options CableLabs WARNING: The sender of this email could not be validated and may not match the person in the "From" field. The outcome of today's P4 apps meeting. There are two high level options for the packet structure, both will be specified in Telemetry Report v2.0: 1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format. 2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format. Compact Telemetry Report v2.0 proposal +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Node id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type 1|InProt | Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || |D|Q|F|I| Rsvd | RepMdBits | DS Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment or Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each. 6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary. 7. Inner Protocol: 0: None 1: Domain Specific Extensions 2: Ethernet Packet Fragment 3: IPv4 Packet Fragment 4: IPv6 Packet Fragment Nested TLV Telemetry Report v2.0 proposal: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Node id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type 2|D|Q|F|I| Report Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- || | Type | Rsvd | Length | RepMdBits | /\ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || || | Variable Optional Metadata | \/ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- || | Type | Prot | Length | Reserved | /\ Report +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || || | Packet Fragment | \/ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- || | Type | Rsvd | Length | Domain Specific Md Bits | /\ || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || || | Domain Specific Extensions | \/ \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ —- -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right. 5. Domain Specific Md Status has been moved to Domain Specific Extensions. 6. Type (inner) * Report Metadata * Packet Fragment * Domain Specific Extensions On Wed, Mar 11, 2020 at 2:12 PM Mickey Spiegel <mspiegel@barefootnetworks.com<mailto:mspiegel@barefootnetworks.com>> wrote: The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information. In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet. In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports. There are two high level options for the packet structure: 1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format. 2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format. Compact Telemetry Report v2.0 proposal +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |InProt | Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || |D|Q|F|I| RepMdBits |Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment or Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. 5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each. 6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary. 7. Inner Protocol: 0: None 1: Domain Specific Extensions 2: Ethernet Packet Fragment 3: IPv4 Packet Fragment 4: IPv6 Packet Fragment Nested TLV Telemetry Report v2.0 proposal: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver | hw_id | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Switch id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- | Type |D|Q|F|I| Length | Domain Specific ID | /\ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | RepMdBits | Reserved | Domain Specific Md Bits | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Variable Optional Metadata | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Prot | Length | Reserved | Report +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Packet Fragment | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Type | Rsvd | Length | Domain Specific Md Status | || +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ || | Domain Specific Extensions | \/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -- Notes: 1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields. 2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number. 3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata. 4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right. 5. Domain Specific Md Status has been moved to Domain Specific Extensions. 6. Type (inner) * Packet Fragment * Domain Specific Extensions Any preferences? questions? comments? suggestions? Mickey