p4-discuss@lists.p4.org

discussion of p4 language ideas, tools, questions, etc. -- not related to direct code development

View all threads

Re: [P4-discuss] Huge fluctuations in calculated link latency information

JG
John Grant
Sun, Nov 1, 2020 6:41 PM

It's reasonable in this situation where the incoming packets are being
inspected by software and "ingress" means getting to the front of the
queue for inspection (a bit like queueing for immigration at an
airport). It wouldn't be reasonable if "ingress" meant the packet
arriving at the PHY in the case of a point-to-point wire.

J.

On 31/10/2020 20:29, Bibek Shrestha wrote:

Does that mean the huge latency fluctuations that I am seeing is reasonable?

On Sat, Oct 31, 2020 at 5:20 AM John Grant j@ninetiles.com wrote:

ingress_timestamp is defined as "Time that packet began processing in
IngressParser" which I guess would include time waiting for previous
packets to be parsed; anyway it definitely wouldn't be the time the PHY
received the packet. What you're seeing suggests the parser isn't able
to work at line speed.

--
John Grant
Nine Tiles, Cambridge, England
+44 1223 862599 and +44 1223 511455
http://www.ninetiles.com

On 30/10/2020 21:26, Bibek Shrestha wrote:

Hi,

I am trying to calculate the link latency between the switches. Below
is my bare minimal topology.

(h1)------(s1)------(s2)-----(h2)

Link latency is calculated in p4 as,
link_latency = s2.standard_metadata.ingress_timestamp -
s1.standard_metadata.egress_timestamp

I am using the bmv2 switch and made a change in the code to sync the
time with the system time as suggested. My setup is all done in the
mininet.

My expectation was that the latency would mostly be stable around some
values with some jitter. But what I am seeing is huge fluctuations in
the link latency whenever there is a change in queue occupancy in the
switches. If the switches are congested then the link latencies would
go up by a very high number.

Is my expectation of having relatively stable link latency with some
jitter correct in terms of how real networks work? If so, why am I
seeing such behavior with the bmv2 switch? Or is my way of calculating
the link latency is flawed? In that case, how should I calculate the
link latency?

Thank you

It's reasonable in this situation where the incoming packets are being inspected by software and "ingress" means getting to the front of the queue for inspection (a bit like queueing for immigration at an airport). It wouldn't be reasonable if "ingress" meant the packet arriving at the PHY in the case of a point-to-point wire. J. On 31/10/2020 20:29, Bibek Shrestha wrote: > Does that mean the huge latency fluctuations that I am seeing is reasonable? > > On Sat, Oct 31, 2020 at 5:20 AM John Grant <j@ninetiles.com> wrote: >> ingress_timestamp is defined as "Time that packet began processing in >> IngressParser" which I guess would include time waiting for previous >> packets to be parsed; anyway it definitely wouldn't be the time the PHY >> received the packet. What you're seeing suggests the parser isn't able >> to work at line speed. >> >> -- >> John Grant >> Nine Tiles, Cambridge, England >> +44 1223 862599 and +44 1223 511455 >> http://www.ninetiles.com >> >> On 30/10/2020 21:26, Bibek Shrestha wrote: >>> Hi, >>> >>> I am trying to calculate the link latency between the switches. Below >>> is my bare minimal topology. >>> >>> (h1)------(s1)------(s2)-----(h2) >>> >>> Link latency is calculated in p4 as, >>> link_latency = s2.standard_metadata.ingress_timestamp - >>> s1.standard_metadata.egress_timestamp >>> >>> I am using the bmv2 switch and made a change in the code to sync the >>> time with the system time as suggested. My setup is all done in the >>> mininet. >>> >>> My expectation was that the latency would mostly be stable around some >>> values with some jitter. But what I am seeing is huge fluctuations in >>> the link latency whenever there is a change in queue occupancy in the >>> switches. If the switches are congested then the link latencies would >>> go up by a very high number. >>> >>> Is my expectation of having relatively stable link latency with some >>> jitter correct in terms of how real networks work? If so, why am I >>> seeing such behavior with the bmv2 switch? Or is my way of calculating >>> the link latency is flawed? In that case, how should I calculate the >>> link latency? >>> >>> Thank you
AF
Andy Fingerhut
Sun, Nov 1, 2020 8:18 PM

I think it might be good if there was an article written about the pitfalls
of trying to use mininet and bmv2 simple_switch for performance
simulations, so that we could point people at it whenever they ask
questions like this.

bmv2 simple_switch has several properties in how it queues things
internally, and schedules packets for ingress and egress processing, that
are not good fits for how a switch ASIC does those things in real time.

Similarly, mininet links with maximum bit rates configured on them do not
behave the same as constant bit rate physical Ethernet links.  They allow
packets to be enqueued arbitrarily quickly, and then have a FIFO queue with
packet buffer "inside" of them that only lets them reach the other end of
the link at the configured maximum bit rate.  Physical Ethernet links do
not behave that way -- they only allow packets to be transmitted on them at
the maximum bit rate, and any queues before them must be in the device that
is transmitting on the link.

Andy

On Sun, Nov 1, 2020 at 10:42 AM John Grant j@ninetiles.com wrote:

It's reasonable in this situation where the incoming packets are being
inspected by software and "ingress" means getting to the front of the
queue for inspection (a bit like queueing for immigration at an
airport). It wouldn't be reasonable if "ingress" meant the packet
arriving at the PHY in the case of a point-to-point wire.

J.

On 31/10/2020 20:29, Bibek Shrestha wrote:

Does that mean the huge latency fluctuations that I am seeing is

reasonable?

On Sat, Oct 31, 2020 at 5:20 AM John Grant j@ninetiles.com wrote:

ingress_timestamp is defined as "Time that packet began processing in
IngressParser" which I guess would include time waiting for previous
packets to be parsed; anyway it definitely wouldn't be the time the PHY
received the packet. What you're seeing suggests the parser isn't able
to work at line speed.

--
John Grant
Nine Tiles, Cambridge, England
+44 1223 862599 and +44 1223 511455
http://www.ninetiles.com

On 30/10/2020 21:26, Bibek Shrestha wrote:

Hi,

I am trying to calculate the link latency between the switches. Below
is my bare minimal topology.

(h1)------(s1)------(s2)-----(h2)

Link latency is calculated in p4 as,
link_latency = s2.standard_metadata.ingress_timestamp -
s1.standard_metadata.egress_timestamp

I am using the bmv2 switch and made a change in the code to sync the
time with the system time as suggested. My setup is all done in the
mininet.

My expectation was that the latency would mostly be stable around some
values with some jitter. But what I am seeing is huge fluctuations in
the link latency whenever there is a change in queue occupancy in the
switches. If the switches are congested then the link latencies would
go up by a very high number.

Is my expectation of having relatively stable link latency with some
jitter correct in terms of how real networks work? If so, why am I
seeing such behavior with the bmv2 switch? Or is my way of calculating
the link latency is flawed? In that case, how should I calculate the
link latency?

Thank you

I think it might be good if there was an article written about the pitfalls of trying to use mininet and bmv2 simple_switch for performance simulations, so that we could point people at it whenever they ask questions like this. bmv2 simple_switch has several properties in how it queues things internally, and schedules packets for ingress and egress processing, that are not good fits for how a switch ASIC does those things in real time. Similarly, mininet links with maximum bit rates configured on them do not behave the same as constant bit rate physical Ethernet links. They allow packets to be enqueued arbitrarily quickly, and then have a FIFO queue with packet buffer "inside" of them that only lets them reach the other end of the link at the configured maximum bit rate. Physical Ethernet links do not behave that way -- they only allow packets to be transmitted on them at the maximum bit rate, and any queues before them must be in the device that is transmitting on the link. Andy On Sun, Nov 1, 2020 at 10:42 AM John Grant <j@ninetiles.com> wrote: > It's reasonable in this situation where the incoming packets are being > inspected by software and "ingress" means getting to the front of the > queue for inspection (a bit like queueing for immigration at an > airport). It wouldn't be reasonable if "ingress" meant the packet > arriving at the PHY in the case of a point-to-point wire. > > J. > > On 31/10/2020 20:29, Bibek Shrestha wrote: > > Does that mean the huge latency fluctuations that I am seeing is > reasonable? > > > > On Sat, Oct 31, 2020 at 5:20 AM John Grant <j@ninetiles.com> wrote: > >> ingress_timestamp is defined as "Time that packet began processing in > >> IngressParser" which I guess would include time waiting for previous > >> packets to be parsed; anyway it definitely wouldn't be the time the PHY > >> received the packet. What you're seeing suggests the parser isn't able > >> to work at line speed. > >> > >> -- > >> John Grant > >> Nine Tiles, Cambridge, England > >> +44 1223 862599 and +44 1223 511455 > >> http://www.ninetiles.com > >> > >> On 30/10/2020 21:26, Bibek Shrestha wrote: > >>> Hi, > >>> > >>> I am trying to calculate the link latency between the switches. Below > >>> is my bare minimal topology. > >>> > >>> (h1)------(s1)------(s2)-----(h2) > >>> > >>> Link latency is calculated in p4 as, > >>> link_latency = s2.standard_metadata.ingress_timestamp - > >>> s1.standard_metadata.egress_timestamp > >>> > >>> I am using the bmv2 switch and made a change in the code to sync the > >>> time with the system time as suggested. My setup is all done in the > >>> mininet. > >>> > >>> My expectation was that the latency would mostly be stable around some > >>> values with some jitter. But what I am seeing is huge fluctuations in > >>> the link latency whenever there is a change in queue occupancy in the > >>> switches. If the switches are congested then the link latencies would > >>> go up by a very high number. > >>> > >>> Is my expectation of having relatively stable link latency with some > >>> jitter correct in terms of how real networks work? If so, why am I > >>> seeing such behavior with the bmv2 switch? Or is my way of calculating > >>> the link latency is flawed? In that case, how should I calculate the > >>> link latency? > >>> > >>> Thank you > > _______________________________________________ > P4-discuss mailing list > P4-discuss@lists.p4.org > http://lists.p4.org/mailman/listinfo/p4-discuss_lists.p4.org >