The new requirement from Cisco and CableLabs is the ability to coalesce
multiple reports into one packet. In addition we need to add Domain
Specific ID, Domain Specific Md Bits, and an extension mechanism for
arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be
restructured, separating fields common to all reports in the packet from
fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own
packet fragment or flow identifying information, the report header needs to
change from a small header that is followed by a packet fragment, to a
report header that encapsulates a packet fragment. With this in mind, I
suggest that we don't need a next protocol. All the contents are within the
reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

1. The more compact format uses an "InnerProtocol" field to identify
   whether a packet fragment or domain specific extensions are included in the
   report. In order to get to those fields, the receiver of the report has to
   parse through variable optional metadata based on RepMdBits and Domain
   Specific Md Bits. If it does not understand how to parse those bits, then
   it cannot figure out where the packet fragment or domain specific
   extensions begin. This format adds only 4 bytes, compared to the Telemetry
   Report v1.0 format.
2. Nested TLV structure. An implementation that does not support
   coalescing needs to generate two levels of TLVs, each with only one TLV,
   with two different lengths values. This format adds 8 bytes compared to the
   Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with
   the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order
   to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable
   Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits
   each.
6. Domain Specific Md Status has been dropped. Use Domain Specific
   Extensions if necessary.
7. Inner Protocol:

0: None

1: Domain Specific Extensions
2: Ethernet Packet Fragment

3: IPv4 Packet Fragment

4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with
   the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order
   to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable
   Optional Metadata.
4. Length of each report is limited to 1024 bytes. If we want to expand
   that to 16k, then DQFI would have to move down, squeezing RepMdBits and
   Domain Specific Md Bits to 28 bits, and either burn a separate type for
   each packet fragment protocol, or move Protocol to the 16 Reserved bits on
   the right.
5. Domain Specific Md Status has been moved to Domain Specific
   Extensions.
6. Type (inner)
   1. Packet Fragment
   2. Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Hi Mickey,

Thanks for suggesting the interesting options.
I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios.

hw_id and seq # fields are sharing 24 bits.
Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)?

Thanks,
JK

From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Sent: Wednesday, March 11, 2020 2:12 PM
To: p4-apps@lists.p4.org p4-apps@lists.p4.org
Subject: [P4-Apps] Telemetry Report 2.0 options

The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
7. Inner Protocol:

0: None
1: Domain Specific Extensions
2: Ethernet Packet Fragment
3: IPv4 Packet Fragment
4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
5. Domain Specific Md Status has been moved to Domain Specific Extensions.
6. Type (inner)
   *  Packet Fragment
   *  Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Thanks Mickey for coming up with the format options.

From my perspective, Nested TLV is flexible and address all know use cases. The only comment I have is that Instruction Bitmap in the INT Header is 16 bits and Domain Specific Instruction is 16 bits. RepMD bits and Domain Specific MD bits add up to a max of 28 bits (without any reserved bits) in Compact format and 32-bits in Nested TLV format. Does it make sense to move the Domain Specific MD bits to Domain Specific extensions, similar to Domain Specific Md Status.

Thoughts?

-Ramesh

From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Date: Wednesday, March 11, 2020 at 5:50 PM
To: "Lee, Jeongkeun" jk.lee@intel.com
Cc: "p4-apps@lists.p4.org" p4-apps@lists.p4.org, "Alvarez, Daniel A" daniel.a.alvarez@intel.com
Subject: Re: [P4-Apps] Telemetry Report 2.0 options

On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun <jk.lee@intel.commailto:jk.lee@intel.com> wrote:
Hi Mickey,

Thanks for suggesting the interesting options.
I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios.

hw_id and seq # fields are sharing 24 bits.
Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)?

6 bits accommodates 16 line cards or stackable units * 4 hw_id values for each. That is probably enough.  I only rounded up from 6 to 8 bits because I did not think 20 bits vs 22 bits of sequence number is significant. We can go either way.

Mickey

Thanks,
JK

From: P4-apps <p4-apps-bounces@lists.p4.orgmailto:p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.commailto:mspiegel@barefootnetworks.com>
Sent: Wednesday, March 11, 2020 2:12 PM
To: p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org <p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org>
Subject: [P4-Apps] Telemetry Report 2.0 options

The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.
   Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
7. Inner Protocol:
   0: None
   1: Domain Specific Extensions
   2: Ethernet Packet Fragment
   3: IPv4 Packet Fragment
   4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
5. Domain Specific Md Status has been moved to Domain Specific Extensions.
6. Type (inner)

 *   Packet Fragment
 *   Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Hi Mickey,

Thanks for pulling together these two proposals.

It looks like both options will address our immediate need to collate multiple packet fragments into a single report.  The 1024 size limitation would not be an issue for the use cases which we are exploring.

While we are not currently sending any domain specific TR data, just domain specific data in the INT report.  I do see option where we would send some additional domain specific data from the switch/sink generating the TR.  For instance, we may leverage some counters or include the current BGP flowspec rules.  If this domain specific data doesn’t need to be repeated on each report within the packet, so both option will work.

I like the versatility of the Nested TLV Telemetry Report, since it allows for both domain specific metadata and the packet fragment to be in one report.  I don’t have a compelling use for both domain specific and packet fragment info per report at this time.

Thanks,
Randy Levensalor

From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Date: Wednesday, March 11, 2020 at 6:50 PM
To: "Lee, Jeongkeun" jk.lee@intel.com
Cc: "p4-apps@lists.p4.org" p4-apps@lists.p4.org, "Alvarez, Daniel A" daniel.a.alvarez@intel.com
Subject: Re: [P4-Apps] Telemetry Report 2.0 options

CableLabs WARNING: The sender of this email could not be validated and may not match the person in the "From" field.

On Wed, Mar 11, 2020 at 5:09 PM Lee, Jeongkeun <jk.lee@intel.commailto:jk.lee@intel.com> wrote:
Hi Mickey,

Thanks for suggesting the interesting options.
I see that the major difference between the two lies in the co-existence of Packet Fragment and Domain Specific Extensions. The compact option treats them mutually exclusive, the TLV allows co-existence. I'd let Ramesh and Randy comment on this based on their use case scenarios.

hw_id and seq # fields are sharing 24 bits.
Can you remind us the case when we need 8b hw_id (vs 6b in the v1.0 spec)?

6 bits accommodates 16 line cards or stackable units * 4 hw_id values for each. That is probably enough.  I only rounded up from 6 to 8 bits because I did not think 20 bits vs 22 bits of sequence number is significant. We can go either way.

Mickey

Thanks,
JK

From: P4-apps <p4-apps-bounces@lists.p4.orgmailto:p4-apps-bounces@lists.p4.org> on behalf of Mickey Spiegel <mspiegel@barefootnetworks.commailto:mspiegel@barefootnetworks.com>
Sent: Wednesday, March 11, 2020 2:12 PM
To: p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org <p4-apps@lists.p4.orgmailto:p4-apps@lists.p4.org>
Subject: [P4-Apps] Telemetry Report 2.0 options

The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.
   Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
7. Inner Protocol:
   0: None
   1: Domain Specific Extensions
   2: Ethernet Packet Fragment
   3: IPv4 Packet Fragment
   4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
5. Domain Specific Md Status has been moved to Domain Specific Extensions.
6. Type (inner)

 *   Packet Fragment
 *   Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey

Hi Mickey, Et al.,

I haven’t seen any updates adding this to a PR or follow-up meetings.

Is there a plan to close on this any time soon?

Can I help?  Should I add this to my PR for multiple reports per packet or will this be added to another PR?

Many Thanks and looking forward to the release of the 2.0 Telemetry Report.

Randy Levensalor

From: P4-apps p4-apps-bounces@lists.p4.org on behalf of Mickey Spiegel mspiegel@barefootnetworks.com
Date: Friday, March 13, 2020 at 12:01 PM
To: "p4-apps@lists.p4.org" p4-apps@lists.p4.org
Subject: Re: [P4-Apps] Telemetry Report 2.0 options

CableLabs WARNING: The sender of this email could not be validated and may not match the person in the "From" field.

The outcome of today's P4 apps meeting.

There are two high level options for the packet structure, both will be specified in Telemetry Report v2.0:

1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.

Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |  hw_id  |              Sequence Number              |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                            Node id                            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type 1|InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I| Rsvd  |          RepMdBits          |  DS Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
7. Inner Protocol:

0: None

1: Domain Specific Extensions

2: Ethernet Packet Fragment

3: IPv4 Packet Fragment

4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |  hw_id  |              Sequence Number              |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                            Node id                            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+        --

| Type 2|D|Q|F|I| Report Length |      Domain Specific ID      |        /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Rsvd  |    Length    |          RepMdBits          |  /\    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                  Variable Optional Metadata                  |  /    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Prot  |    Length    |          Reserved            |  /\  Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                        Packet Fragment                        |  /    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    ||

| Type  | Rsvd  |    Length    |    Domain Specific Md Bits    |  /\    ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||    ||

|                  Domain Specific Extensions                  |  /    /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  —-    --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.

2. The Sequence Number has been reduced from 32 bits to 22 bits in order to save 4 bytes.

3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.

4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.

5. Domain Specific Md Status has been moved to Domain Specific Extensions.

6. Type (inner)

 *   Report Metadata
 *   Packet Fragment
 *   Domain Specific Extensions

On Wed, Mar 11, 2020 at 2:12 PM Mickey Spiegel <mspiegel@barefootnetworks.commailto:mspiegel@barefootnetworks.com> wrote:
The new requirement from Cisco and CableLabs is the ability to coalesce multiple reports into one packet. In addition we need to add Domain Specific ID, Domain Specific Md Bits, and an extension mechanism for arbitrary domain specific information.

In order to meet these requirements, the packet format needs to be restructured, separating fields common to all reports in the packet from fields that are specific to each report in the packet.

In order to allow for multiple reports in a packet, each with its own packet fragment or flow identifying information, the report header needs to change from a small header that is followed by a packet fragment, to a report header that encapsulates a packet fragment. With this in mind, I suggest that we don't need a next protocol. All the contents are within the reports. There is nothing that needs to come after the reports.

There are two high level options for the packet structure:

1. The more compact format uses an "InnerProtocol" field to identify whether a packet fragment or domain specific extensions are included in the report. In order to get to those fields, the receiver of the report has to parse through variable optional metadata based on RepMdBits and Domain Specific Md Bits. If it does not understand how to parse those bits, then it cannot figure out where the packet fragment or domain specific extensions begin. This format adds only 4 bytes, compared to the Telemetry Report v1.0 format.
2. Nested TLV structure. An implementation that does not support coalescing needs to generate two levels of TLVs, each with only one TLV, with two different lengths values. This format adds 8 bytes compared to the Telemetry Report v1.0 format.
   Compact Telemetry Report v2.0 proposal

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |InProt |    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|D|Q|F|I|    RepMdBits    |Reserved |  Domain Specific Md Bits  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Report

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|        Packet Fragment or Domain Specific Extensions        |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes.
5. RepMdBits and Domain Specific Md Bits share 28 bits. Propose 14 bits each.
6. Domain Specific Md Status has been dropped. Use Domain Specific Extensions if necessary.
7. Inner Protocol:
   0: None
   1: Domain Specific Extensions
   2: Ethernet Packet Fragment
   3: IPv4 Packet Fragment
   4: IPv6 Packet Fragment

Nested TLV Telemetry Report v2.0 proposal:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|  Ver  |    hw_id    |            Sequence Number            |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|                          Switch id                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

| Type  |D|Q|F|I|    Length    |      Domain Specific ID      |  /\

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|    RepMdBits    |  Reserved  |    Domain Specific Md Bits    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Variable Optional Metadata                  |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Prot  |    Length    |          Reserved            | Report

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                        Packet Fragment                        |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

| Type  | Rsvd  |    Length    |  Domain Specific Md Status    |  ||

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  ||

|                  Domain Specific Extensions                  |  /

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  --

Notes:

1. There is no outer "length" field, since it is largely redundant with the UDP and IP length fields.
2. The Sequence Number has been reduced from 32 bits to 20 bits in order to save 4 bytes. We could go with any of 16, 20, or 22 bit Sequence Number.
3. Ingress Timestamp gets a RepMdBit value and moves into Variable Optional Metadata.
4. Length of each report is limited to 1024 bytes. If we want to expand that to 16k, then DQFI would have to move down, squeezing RepMdBits and Domain Specific Md Bits to 28 bits, and either burn a separate type for each packet fragment protocol, or move Protocol to the 16 Reserved bits on the right.
5. Domain Specific Md Status has been moved to Domain Specific Extensions.
6. Type (inner)

 *   Packet Fragment
 *   Domain Specific Extensions

Any preferences? questions? comments? suggestions?

Mickey