Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scapy v2.6.0 doesn't correctly read routes in containers #4674

Open
michaelvdnet opened this issue Feb 25, 2025 · 8 comments
Open

Scapy v2.6.0 doesn't correctly read routes in containers #4674

michaelvdnet opened this issue Feb 25, 2025 · 8 comments
Labels

Comments

@michaelvdnet
Copy link

michaelvdnet commented Feb 25, 2025

Brief description

The new rtnetlink implementation builds an incorrect route table on Azure Container Apps.
This breaks pretty much all inet functionality as Ether layer dst and IP layer src fields are no longer correctly set due to lookup logic involving the route table.

Scapy version

2.6.1

Python version

3.11.11

Operating system

python:11-alpine image -> alpine 3.21.3 -> linux 5.15.173.1-1.cm2

Additional environment information

No response

How to reproduce

>>> scapy.__version__
>>> conf.route  
>>> conf.route.route()
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
>>> srp1(p)

Actual result

>>> scapy.__version__
'2.6.1.dev49'
>>> conf.route
Network          Netmask          Gateway      Iface  Output IP     Metric
0.0.0.0          0.0.0.0          169.254.1.1  eth0   0.0.0.0       0     
100.100.0.14     255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
100.100.127.255  255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
127.0.0.0        255.0.0.0        0.0.0.0      lo     127.0.0.1     0     
127.0.0.1        255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
127.255.255.255  255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
169.254.1.1      255.255.255.255  0.0.0.0      eth0   0.0.0.0       0     
>>> conf.route.route()
('lo', '0.0.0.0', '0.0.0.0')
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
###[ Ethernet ]###
  dst       = None
  src       = 26:5d:4b:dd:17:a0
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 0.0.0.0
     dst       = <snip>
     \options   \
###[ ICMP ]###
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = b''

>>> srp1(p)
Begin emission
WARNING: No broadcast address found for iface eth0

..............^CWARNING: MAC address to reach destination not found. Using broadcast.

Finished sending 0 packets

Received 14 packets, got 0 answers, remaining 0 packets

Expected result

>>> scapy.__version__
'2.5.0'
>>> conf.route
Network      Netmask          Gateway      Iface  Output IP     Metric
0.0.0.0      0.0.0.0          169.254.1.1  eth0   100.100.0.33  0     
127.0.0.0    255.0.0.0        0.0.0.0      lo     127.0.0.1     1     
169.254.1.1  255.255.255.255  0.0.0.0      eth0   100.100.0.33  0     
>>> conf.route.route()
('eth0', '100.100.0.33', '169.254.1.1')
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
WARNING: No broadcast address found for iface eth0

###[ Ethernet ]### 
  dst       = aa:aa:aa:aa:aa:aa
  src       = ae:f4:4a:dd:f9:60
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 100.100.0.33
     dst       = <snip>
     \options   \
###[ ICMP ]### 
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = ''

>>> srp1(p)
Begin emission:
WARNING: No broadcast address found for iface eth0

Finished sending 1 packets.
*
Received 1 packets, got 1 answers, remaining 0 packets
<Ether  dst=ae:f4:4a:dd:f9:60 src=aa:aa:aa:aa:aa:aa type=IPv4 |<IP  version=4 ihl=5 tos=0x0 len=28 id=51819 flags= frag=0 ttl=62 proto=icmp chksum=0x4116 src=<snip> dst=100.100.0.33 |<ICMP  type=echo-reply code=0 chksum=0x0 id=0x0 seq=0x0 |>>>

Related resources

No response

@michaelvdnet
Copy link
Author

michaelvdnet commented Feb 25, 2025

After some diggin, it looks like the kernel never returns the rta_type=RTA_PREFSRC rtmsg_rtattr causing the Output IP to be set to the default 0.0.0.0 address.

Full rt message:

 <rtmsghdr  nlmsg_len=52 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740497776 nlmsg_pid=2153 |<rtmsg  rtm_family=AF_INET rtm_dst_len=0 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_MAIN rtm_protocol=RTPROT_STATIC rtm_scope=RT_SCOPE_UNIVERSE rtm_type=RTN_UNICAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_MAIN |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_GATEWAY rta_data=169.254.1.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,

Manually patching scapy/arch/linx/rtnetlink.py:898 to the hardcoded IP fixes the issue:

- addr = "0.0.0.0"
+ addr = "100.100.0.14"
>>> conf.route
Network          Netmask          Gateway      Iface  Output IP     Metric
0.0.0.0          0.0.0.0          169.254.1.1  eth0   100.100.1.14  0     
100.100.0.14     255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
100.100.127.255  255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
127.0.0.0        255.0.0.0        0.0.0.0      lo     127.0.0.1     0     
127.0.0.1        255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
127.255.255.255  255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
169.254.1.1      255.255.255.255  0.0.0.0      eth0   100.100.1.14  0     
>>> conf.route.route()
('eth0', '100.100.1.14', '169.254.1.1')
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
###[ Ethernet ]###
  dst       = None
  src       = 26:5d:4b:dd:17:a0
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 100.100.0.14
     dst       = <snip>
     \options   \
###[ ICMP ]###
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = b''

>>> srp1(p)
Begin emission
WARNING: No broadcast address found for iface eth0

.
Finished sending 1 packets
*
Received 2 packets, got 1 answers, remaining 0 packets
<Ether  dst=26:5d:4b:dd:17:a0 src=aa:aa:aa:aa:aa:aa type=IPv4 |<IP  version=4 ihl=5 tos=0x0 len=28 id=17951 flags= frag=0 ttl=62 proto=icmp chksum=0xc575 src=<snip> dst=100.100.0.14 |<ICMP  type=echo-reply code=0 chksum=0x0 id=0x0 seq=0x0 unused=b'' |>>>

Sadly today is the first day I've heard about rtmlink so can't be of much help.
Not sure if this is a problem in the Container Apps kernel or something that can be fixed on library level.
If it's a kernal problem, it might be nice to have some extra checks or workarounds available

@gpotter2
Copy link
Member

gpotter2 commented Feb 25, 2025

Could you share the full output of

from scapy.arch.linux.rtnetlink import _read_routes
_read_routes(socket.AF_INET)

and your ip route show table all result?

Thanks

@michaelvdnet
Copy link
Author

michaelvdnet commented Feb 26, 2025

Ofcourse!

>>> pprint.pp(_read_routes(socket.AF_INET))
[<rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_HOST rtm_type=RTN_LOCAL rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=100.100.0.14 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=100.100.0.14 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_LINK rtm_type=RTN_BROADCAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=100.100.127.255 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=100.100.0.14 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=8 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_HOST rtm_type=RTN_LOCAL rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=127.0.0.0 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=1 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_HOST rtm_type=RTN_LOCAL rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=1 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_LINK rtm_type=RTN_BROADCAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=127.255.255.255 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=1 |>] |>>,
 <rtmsghdr  nlmsg_len=52 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=0 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_MAIN rtm_protocol=RTPROT_STATIC rtm_scope=RT_SCOPE_UNIVERSE rtm_type=RTN_UNICAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_MAIN |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_GATEWAY rta_data=169.254.1.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,
 <rtmsghdr  nlmsg_len=52 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_MAIN rtm_protocol=RTPROT_STATIC rtm_scope=RT_SCOPE_LINK rtm_type=RTN_UNICAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_MAIN |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=169.254.1.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>]
/app # ip route show table all
default via 169.254.1.1 dev eth0 
169.254.1.1 dev eth0 scope link 
local 100.100.0.14 dev eth0 table local scope host  src 100.100.0.14 
broadcast 100.100.127.255 dev eth0 table local scope link  src 100.100.0.14 
local 127.0.0.0/8 dev lo table local scope host  src 127.0.0.1 
local 127.0.0.1 dev lo table local scope host  src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local scope link  src 127.0.0.1 
fe80::/64 dev eth0  metric 256 
local ::1 dev lo table local  metric 0 
local fe80::245d:4bff:fedd:17a0 dev eth0 table local  metric 0 
multicast ff00::/8 dev eth0 table local  metric 256

Looks like src is missing on the default route here too. I'm not sure if that's something on the container image level or at the kernel level which I can't control.

I'll test with the debian base image and see if that fixes it Did not change anything.

@michaelvdnet
Copy link
Author

A few stackoverflow resources tell me the src property is more of a hint and not a required property (connectivity still works). Would it be a solution to fall back to iface IP when none is found in the route table?

@michaelvdnet michaelvdnet changed the title Scapy v2.6.0 doesn't correctly read routes on Azure Container Apps Scapy v2.6.0 doesn't correctly read routes in containers Feb 26, 2025
@michaelvdnet
Copy link
Author

michaelvdnet commented Feb 26, 2025

Looks like the issue also arises on my macbook running rancher desktop, same container image:

root@4297ea22def2:/# ip route show table all
default via 10.4.0.1 dev eth0 
10.4.0.0/24 dev eth0 proto kernel scope link src 10.4.0.18 
local 10.4.0.18 dev eth0 table local proto kernel scope host src 10.4.0.18 
broadcast 10.4.0.255 dev eth0 table local proto kernel scope link src 10.4.0.18 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
fe80::/64 dev eth0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::ecab:afff:feec:1341 dev eth0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
>>> conf.route
Network          Netmask          Gateway   Iface  Output IP  Metric
0.0.0.0          0.0.0.0          10.4.0.1  eth0   0.0.0.0    0     
10.4.0.0         255.255.255.0    0.0.0.0   eth0   10.4.0.18  0     
10.4.0.18        255.255.255.255  0.0.0.0   eth0   10.4.0.18  0     
10.4.0.255       255.255.255.255  0.0.0.0   eth0   10.4.0.18  0     
127.0.0.0        255.0.0.0        0.0.0.0   lo     127.0.0.1  0     
127.0.0.1        255.255.255.255  0.0.0.0   lo     127.0.0.1  0     
127.255.255.255  255.255.255.255  0.0.0.0   lo     127.0.0.1  0     
224.0.0.0        240.0.0.0        0.0.0.0   eth0   10.4.0.18  250   
>>> conf.route.route()
('lo', '0.0.0.0', '0.0.0.0')
>>>
>>> p = Ether()/IP(dst="1.1.1.1")/ICMP()
>>> p.show()
###[ Ethernet ]###
  dst       = None
  src       = ee:ab:af:ec:13:41
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 10.4.0.18
     dst       = 1.1.1.1
     \options   \
###[ ICMP ]###
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = b''
>>> srp1(p, verbose=False, timeout=5)
>>> 

Weird that it does set the src address correctly on the packet

@gpotter2
Copy link
Member

gpotter2 commented Feb 26, 2025

Thanks. I'm not sure that I understand the issue. What you provided in the last comment (including the fact that the output IP is 0.0.0.0 in the routing table) is perfectly expected. You'll notice that that's not what's kept in the final packet.

Going back to your initial question, the fact that you're using APIPA adresses sounds to me like your network configuration is already broken. I'd investigate further on that end.

You should have the same result with ip route get <an ip> and conf.route.route("<the IP>"). If that's not the case then there could be a bug, but until then I'm a bit unsure of what your issue is.

@michaelvdnet
Copy link
Author

michaelvdnet commented Feb 27, 2025

It does get the correct interface, but not the correct src ip:

# ip route get 1.1.1.1
1.1.1.1 via 169.254.1.1 dev eth0 src 100.100.0.8 uid 0 
    cache 
# python3
>>> Python 3.11.11 (main, Feb 25 2025, 05:21:12) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
from scapy.all import *
>>> conf.route.route("1.1.1.1")
('eth0', '0.0.0.0', '169.254.1.1')

I've traced it down a bit and:

  1. the getmacbyip function uses the ARP packet
  2. The ARP packet uses the SourceIPField as psrc
  3. The SourceIPField uses pkt.route[1] or conf.route.route()[1] which is 0.0.0.0
  4. An ARP request like Who has 1.1.1.1. Tell 0.0.0.0 gets sent out
  5. We don't get a dst MAC so nobody responds to our packet

The IP packet also uses the SourceIPField so this not only breaks l2 but also l3.

@michaelvdnet
Copy link
Author

michaelvdnet commented Feb 27, 2025

Stepping into getmacbyip("1.1.1.1") with pdb results in:

(Pdb) where
  <stdin>(1)<module>()
  <stdin>(3)trace()
  /usr/local/lib/python3.11/site-packages/scapy/layers/l2.py(175)getmacbyip()
-> res = srp1(Ether(dst=ETHER_BROADCAST) / ARP(op="who-has", pdst=ip),
> /usr/local/lib/python3.11/site-packages/scapy/sendrecv.py(769)srp1()
-> ans, _ = srp(*args, **kargs)
(Pdb) !args[0].show()
###[ Ethernet ]###
  dst       = ff:ff:ff:ff:ff:ff
  src       = 6e:0e:fc:91:06:6a
  type      = ARP
###[ ARP ]###
     hwtype    = Ethernet (10Mb)
     ptype     = IPv4
     hwlen     = None
     plen      = None
     op        = who-has
     hwsrc     = 6e:0e:fc:91:06:6a
     psrc      = 0.0.0.0
     hwdst     = 00:00:00:00:00:00
     pdst      = 169.254.1.1

Setting psrc manually on ARP fixes it:

>>> srp1(Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(op="who-has", pdst="169.254.1.1"), timeout=4)
Begin emission

Finished sending 1 packets
......................
Received 22 packets, got 0 answers, remaining 1 packets
>>> srp1(Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(op="who-has", psrc="100.100.0.8", pdst="169.254.1.1"), timeout=4)

Finished sending 1 packets
Begin emission
......*
Received 7 packets, got 1 answers, remaining 0 packets
<Ether  dst=6e:0e:fc:91:06:6a src=aa:aa:aa:aa:aa:aa type=ARP |<ARP  hwtype=Ethernet (10Mb) ptype=IPv4 hwlen=6 plen=4 op=is-at hwsrc=aa:aa:aa:aa:aa:aa psrc=169.254.1.1 hwdst=6e:0e:fc:91:06:6a pdst=100.100.0.8 |>>

@gpotter2 gpotter2 added the bug label Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants