[P4-dev] INT demo issue - Bad UDP checksum for packets headed to monitor

Avinash Herle avinash.herle at barefootnetworks.com
Tue May 30 19:42:59 EDT 2017


Thanks Rakesh for the solution. I shall update the troubleshooting steps on
the Readme.

On Sun, May 28, 2017 at 8:17 PM, Yongfeng Sun <sunnogo at gmail.com> wrote:

> Awesome, thanks!
> It works for me.
>
>
> Rakesh Kumar <kumar19 at illinois.edu>于2017年5月14日周日 上午4:59写道:
>
>> Okay. The solution to this lies in the quirks of how linux bridges get
>> filtered. Essentially, you need to go in disable all the filters so that
>> the UDP packets can get through. Details here: https://wiki.
>> linuxfoundation.org/networking/bridge
>>
>> But essentially:
>>
>>  # cd /proc/sys/net/bridge
>>  # for f in bridge-nf-*; do echo 0 > $f; done
>>
>> See attached screenshot of the working INT web UI.
>>
>>
>> -
>> rakesh
>>
>> On Thu, May 11, 2017 at 11:26 PM, Rakesh Kumar <kumar19 at illinois.edu>
>> wrote:
>>
>>> Couple of quick question Rakesh.
>>>> 1) Did you install the VXLAN_GPE driver and the other dependencies
>>>> mentioned in this readme?
>>>>
>>>
>>> Yes, I verified with dmesg/lsmod.
>>>
>>>
>>>> 2) Once you run the mininet scripts you should see a few log files
>>>> (prefixed with tmp_) generated in the path <p4factory>/mininet. Do they
>>>> report any error?
>>>>
>>>>
>>> No errors: A warning line saying "No route found for  IPv6 destination
>>> :: " in each one of the host files(i.e. tmp_h$).
>>>
>>>
>>>> 1) Run the mininet script
>>>> 2) Start a long running iperf session say, h1 perf -c 10.2.1.4 -t 100
>>>> 3) Open another window and console into
>>>>      a) switch leaf1 (since it is connected to the source h1).
>>>> Collect tcpdump on the interface leaf1-eth1 (interface connected to h1).
>>>> The host h1 is supposed to add the INT shim header. Verify if the INT
>>>> header is added.
>>>>
>>>      b) switch leaf1. Collect tcpdump on the interface leaf1-eth3
>>>> or leaf1-eth4 depending upon which interface the packets go out of.
>>>> leaf1 should have appended INT metadata.
>>>>      c) switch leaf2 (since it is connected to the destination h4).
>>>> Collect tcpdump on the interface leaf2-eth2 (interface connected to h4).
>>>> 4) Do refer the INT spec when verifying the tcpdump.
>>>>
>>>
>>> So, I did this with a  ping -c1 10.2.1.4  at h1 (instead of iperf). See
>>> attached screenshots for a captured icmp request and reply packet pair at
>>> leaf1-eth1, leaf1-eth3 and leaf2-eth2. I think it seems to be working, i.e.
>>> metadata is being added, the hop count is increasing and the length is
>>> increasing. I see this as UDP port 4790 traffic which as I understand is
>>> supposed to be carrying VXLAN-GPE...
>>>
>>>
>>>
>>>> 5) If every hop is appending it's metadata information, then proceed
>>>> with checking the monitor.
>>>>
>>> 6) Make sure you have all the dependencies for the monitor installed.
>>>> Refer readme.
>>>>
>>>
>>>
>>>>
>>>> Let me know if you have any more questions.
>>>>
>>>>
>>> From preprocessor.py, I can see UDP packets being sent via the socket to
>>> monitor. However, as I said in the previous email, these packets have bad
>>> checksum and never arrive at the interfaces named veth-t1/veth-t2 (verified
>>> this by tcpdump) on the host machine. Any ideas why these packets are not
>>> getting through the testbr1 and why they might have a bad checksum?
>>>
>>>
>>>
>>> PS: You need to build switch with switchlink.
>>>>
>>>>
>>> I did. Used this configuration as mentioned in the README (./configure
>>> --with-bmv2 --with-switchlink --enable-thrift --prefix=$HOME/install)
>>>
>>> -
>>> rakesh
>>>
>>> On Wed, May 10, 2017 at 11:54 AM, Avinash Herle <avinash.herle@
>>> barefootnetworks.com> wrote:
>>>
>>>> Couple of quick question Rakesh.
>>>> 1) Did you install the VXLAN_GPE driver and the other dependencies
>>>> mentioned in this readme?
>>>> 2) Once you run the mininet scripts you should see a few log files
>>>> (prefixed with tmp_) generated in the path <p4factory>/mininet. Do they
>>>> report any error?
>>>>
>>>> Please try the following
>>>>
>>>> 1) Run the mininet script
>>>> 2) Start a long running iperf session say, h1 perf -c 10.2.1.4 -t 100
>>>> 3) Open another window and console into
>>>>      a) switch leaf1 (since it is connected to the source h1).
>>>> Collect tcpdump on the interface leaf1-eth1 (interface connected to h1).
>>>> The host h1 is supposed to add the INT shim header. Verify if the INT
>>>> header is added.
>>>>      b) switch leaf1. Collect tcpdump on the interface leaf1-eth3
>>>> or leaf1-eth4 depending upon which interface the packets go out of.
>>>> leaf1 should have appended INT metadata.
>>>>      c) switch leaf2 (since it is connected to the destination h4).
>>>> Collect tcpdump on the interface leaf2-eth2 (interface connected to h4).
>>>> 4) Do refer the INT spec when verifying the tcpdump.
>>>> 5) If every hop is appending it's metadata information, then proceed
>>>> with checking the monitor.
>>>> 6) Make sure you have all the dependencies for the monitor installed.
>>>> Refer readme.
>>>>
>>>> Let me know if you have any more questions.
>>>>
>>>> Also, do not reload/refresh the web client when the application is
>>>> running. You might have to re-run the application in this case. The Readme
>>>> mentions a few known monitor issues and troubleshooting tips.
>>>>
>>>> PS: You need to build switch with switchlink.
>>>>
>>>> Thanks,
>>>> Avinash
>>>>
>>>> On Wed, May 10, 2017 at 12:01 AM, Yongfeng Sun <sunnogo at gmail.com>
>>>> wrote:
>>>>
>>>>> I encountered this issue, too. But it's a bit different.
>>>>>
>>>>> I find that vm-eth22 got the packet after preprocessor.py which strips
>>>>> the INT body from vxlan-gpe body. However, web client doesn't show the flow.
>>>>>
>>>>> Use tshark to capture the packet:
>>>>>
>>>>>  sunyongfeng  ~  workshop  p4factory  mininet  sudo tshark -i
>>>>> vm-eth22
>>>>> [sudo] password for sunyongfeng:
>>>>> Running as user "root" and group "root". This could be dangerous.
>>>>> tshark: Lua: Error during loading:
>>>>>  [string "/usr/share/wireshark/init.lua"]:44: dofile has been
>>>>> disabled due to running Wireshark as superuser. See
>>>>> https://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in
>>>>> running Wireshark as an unprivileged user.
>>>>> Capturing on 'vm-eth22'
>>>>>   1 0.000000000 192.168.1.44 -> 192.168.1.100 UDP 62 47895 → 54321
>>>>>  Len=20
>>>>> ...
>>>>> 118 packets captured
>>>>>  sunyongfeng  ~  workshop  p4factory  mininet  sudo tshark -i
>>>>> vm-eth22 -Vc 1
>>>>> Running as user "root" and group "root". This could be dangerous.
>>>>> tshark: Lua: Error during loading:
>>>>>  [string "/usr/share/wireshark/init.lua"]:44: dofile has been
>>>>> disabled due to running Wireshark as superuser. See
>>>>> https://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in
>>>>> running Wireshark as an unprivileged user.
>>>>> Capturing on 'vm-eth22'
>>>>> Frame 1: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on
>>>>> interface 0
>>>>>     Interface id: 0 (vm-eth22)
>>>>>     Encapsulation type: Ethernet (1)
>>>>>     Arrival Time: May 10, 2017 14:55:59.203745483 CST
>>>>>     [Time shift for this packet: 0.000000000 seconds]
>>>>>     Epoch Time: 1494399359.203745483 seconds
>>>>>     [Time delta from previous captured frame: 0.000000000 seconds]
>>>>>     [Time delta from previous displayed frame: 0.000000000 seconds]
>>>>>     [Time since reference or first frame: 0.000000000 seconds]
>>>>>     Frame Number: 1
>>>>>     Frame Length: 62 bytes (496 bits)
>>>>>     Capture Length: 62 bytes (496 bits)
>>>>>     [Frame is marked: False]
>>>>>     [Frame is ignored: False]
>>>>>     [Protocols in frame: eth:ethertype:ip:udp:data]
>>>>> Ethernet II, Src: 52:c7:b0:b9:09:a0 (52:c7:b0:b9:09:a0), Dst:
>>>>> 66:64:01:1c:df:8a (66:64:01:1c:df:8a)
>>>>>     Destination: 66:64:01:1c:df:8a (66:64:01:1c:df:8a)
>>>>>         Address: 66:64:01:1c:df:8a (66:64:01:1c:df:8a)
>>>>>         .... ..1. .... .... .... .... = LG bit: Locally administered
>>>>> address (this is NOT the factory default)
>>>>>         .... ...0 .... .... .... .... = IG bit: Individual address
>>>>> (unicast)
>>>>>     Source: 52:c7:b0:b9:09:a0 (52:c7:b0:b9:09:a0)
>>>>>         Address: 52:c7:b0:b9:09:a0 (52:c7:b0:b9:09:a0)
>>>>>         .... ..1. .... .... .... .... = LG bit: Locally administered
>>>>> address (this is NOT the factory default)
>>>>>         .... ...0 .... .... .... .... = IG bit: Individual address
>>>>> (unicast)
>>>>>     Type: IPv4 (0x0800)
>>>>> Internet Protocol Version 4, Src: 192.168.1.44, Dst: 192.168.1.100
>>>>>     0100 .... = Version: 4
>>>>>     .... 0101 = Header Length: 20 bytes
>>>>>     Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
>>>>>         0000 00.. = Differentiated Services Codepoint: Default (0)
>>>>>         .... ..00 = Explicit Congestion Notification: Not ECN-Capable
>>>>> Transport (0)
>>>>>     Total Length: 48
>>>>>     Identification: 0x0b77 (2935)
>>>>>     Flags: 0x02 (Don't Fragment)
>>>>>         0... .... = Reserved bit: Not set
>>>>>         .1.. .... = Don't fragment: Set
>>>>>         ..0. .... = More fragments: Not set
>>>>>     Fragment offset: 0
>>>>>     Time to live: 64
>>>>>     Protocol: UDP (17)
>>>>>     Header checksum: 0xab65 [validation disabled]
>>>>>         [Good: False]
>>>>>         [Bad: False]
>>>>>     Source: 192.168.1.44
>>>>>     Destination: 192.168.1.100
>>>>>     [Source GeoIP: Unknown]
>>>>>     [Destination GeoIP: Unknown]
>>>>> User Datagram Protocol, Src Port: 47895 (47895), Dst Port: 54321
>>>>> (54321)
>>>>>     Source Port: 47895
>>>>>     Destination Port: 54321
>>>>>     Length: 28
>>>>>     Checksum: 0x840e [validation disabled]
>>>>>         [Good Checksum: False]
>>>>>         [Bad Checksum: False]
>>>>>     [Stream index: 0]
>>>>> Data (20 bytes)
>>>>>
>>>>> 0000  0a 02 01 04 0a 02 01 03 08 00 78 bc 00 00 0a 01
>>>>> ..........x.....
>>>>> 0010  00 03 20 00                                       .. .
>>>>>     Data: 0a0201040a020103080078bc00000a0100032000
>>>>>     [Length: 20]
>>>>>
>>>>> 1 packet captured
>>>>>
>>>>>
>>>>> Rakesh Kumar <kumar19 at illinois.edu>于2017年5月10日周三 上午10:14写道:
>>>>>
>>>>>> Hey All,
>>>>>>
>>>>>> I was trying to recreate the INT demo on a Ubuntu 14.04 LTS VM and
>>>>>> followed the instructions here: https://github.com/
>>>>>> p4lang/p4factory/tree/master/apps/int
>>>>>>
>>>>>> The network comes up, the hosts are able to connect to each other
>>>>>> (i.e. pings and iperfs between them work), but the web application does not
>>>>>> see any counter updates (i.e. the time series all sit at 0).
>>>>>>
>>>>>> So, I tried debugging it. Ultimately, I used tcpdump on each host and
>>>>>> verified that there are UDP packets being sent to the monitor at
>>>>>> 192.168.1.100 (IP address for the interface on the native host). These
>>>>>> packets never make it to the destination interface and have a bad UDP
>>>>>> checksum. This explains why the counters are zero, because of course there
>>>>>> are no updates reaching the monitor. But I don't have many ideas about how
>>>>>> to fix this?
>>>>>>
>>>>>> Any ideas on what might be causing this? Is it normal/expected
>>>>>> behavior due to some header modifications?
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>>
>>>>>>
>>>>>> -
>>>>>> rakesh
>>>>>>
>>>>>> P.S: Is there a slack where devs hangout if I needed more interactive
>>>>>> help about anything?
>>>>>> _______________________________________________
>>>>>> P4-dev mailing list
>>>>>> P4-dev at lists.p4.org
>>>>>> http://lists.p4.org/mailman/listinfo/p4-dev_lists.p4.org
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> P4-dev mailing list
>>>>> P4-dev at lists.p4.org
>>>>> http://lists.p4.org/mailman/listinfo/p4-dev_lists.p4.org
>>>>>
>>>>
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.p4.org/pipermail/p4-dev_lists.p4.org/attachments/20170530/b78752b4/attachment-0002.html>


More information about the P4-dev mailing list