4:33 a.m.
For the last few weeks I am trying to pinpoint the source of a network problem.
The short story: using the standard mtu of 1500 caused sending to stall (and fail).
This is seen when uploading a file (e.g. ftp) or when sending email.
With an mtu of 1456 everything works.
Everything worked unto 02:30 on 7/Nov. At that point some uploads started hanging.
I have cron scripts that upload small and larger files. The small ones get though, the
larger ones hang.
There is enough space on the target.
The network is simple: my server connects to a modem (wireless internet over 4G).
BTW, is there a more suitable list (or forum) for discussing this issue?
I want to confirm where the problem lies. I did not change a thing on my server.
I suspect that the modem updated its fw overnight and the new fw has an issue.
However the ISP says no one else is complaining.
I did not find a way to downgrade the fw or install the old one.
I was given a new modem (with the old fw) but once connected to the 'net
the fw was updated, something I cannot control.
Below is an example of the issue.
$ sudo ifconfig eth1 mtu 1500 # this is the default, which now fails
$ ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
$ tracepath -4 -b -l 1500 -m 5 1.1.1.1
1: e5.eyal.emu.id.au (192.168.2.5) 1.109ms # this is the modem
2: no reply
3: no reply
4: no reply
5: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
## Does the above show that the server (.7) can talk to the modem (.5) but the modem
## cannot talk to the external net?
## Can it be an issue with the next hop (the 4G service)?
$ tracepath -4 -b -l 1456 -m 5 1.1.1.1
1: e5.eyal.emu.id.au (192.168.2.5) 0.760ms
2: 10.155.144.3 (10.155.144.3) 74.809ms
3: 10.246.20.29 (10.246.20.29) 39.707ms
4: 101.115.95.250 (101.115.95.250) 38.656ms
5: 203-220-16-149.tpgi.com.au (203.220.16.149) 35.127ms
Too many hops: pmtu 1456
Resume: pmtu 1456
$ sudo ifconfig eth1 mtu 1456 # with this everything works well
$ ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
$ tracepath -4 -b -l 1500 -m 5 1.1.1.1
1?: [LOCALHOST] pmtu 1456
1: e5.eyal.emu.id.au (192.168.2.5) 0.797ms
1: e5.eyal.emu.id.au (192.168.2.5) 0.614ms
2: 10.155.144.2 (10.155.144.2) 37.951ms
3: 10.246.20.29 (10.246.20.29) 36.881ms
4: 101.115.95.249 (101.115.95.249) 35.653ms
5: 203-220-16-145.tpgi.com.au (203.220.16.145) 36.673ms
Too many hops: pmtu 1456
Resume: pmtu 1456
$ tracepath -4 -b -l 1456 -m 5 1.1.1.1
1: e5.eyal.emu.id.au (192.168.2.5) 0.560ms
2: 10.155.144.2 (10.155.144.2) 33.008ms
3: 10.246.20.29 (10.246.20.29) 36.226ms
4: 101.115.95.250 (101.115.95.250) 36.278ms
5: 203-220-16-149.tpgi.com.au (203.220.16.149) 36.156ms
Too many hops: pmtu 1456
Resume: pmtu 1456
--
Eyal at Home (fedora(a)eyal.emu.id.au)
6:01 a.m.
02.12.23, 11:33 +0100, fedora(a)eyal.emu.id.au:
For the last few weeks I am trying to pinpoint the source of a
network
problem.
The short story: using the standard mtu of 1500 caused sending to stall
(and fail).
This is seen when uploading a file (e.g. ftp) or when sending email.
With an mtu of 1456 everything works.
Everything worked unto 02:30 on 7/Nov. At that point some uploads
started hanging.
I have cron scripts that upload small and larger files. The small ones
get though, the larger ones hang.
There is enough space on the target.
Assuming that the problem's not on your side of the network, the most
likely cause I can think of, is that now somewhere along the path ICMP
packets signalling the need for fragmentation get dropped, and therefore
the sender doesn't learn that it should send smaller packets.
You might want to double check that your network doesn't behave that way.
The network is simple: my server connects to a modem (wireless
internet
over 4G).
BTW, is there a more suitable list (or forum) for discussing this issue?
I want to confirm where the problem lies. I did not change a thing on my
server.
I suspect that the modem updated its fw overnight and the new fw has an
issue.
However the ISP says no one else is complaining.
I did not find a way to downgrade the fw or install the old one.
I was given a new modem (with the old fw) but once connected to the 'net
the fw was updated, something I cannot control.
If the update of the modem's FW happened around the same time the
problem arose, that would point to the new FW as being the culprit
(maybe by dropping those "evil" ICMP packets as I mused above).
And as long as you're reasonably sure that the problem isn't caused by
some change *you* made, and you can't control the modem's behavior, you
could simply accept your fate, lower the interface MTU and move on.
BTW: if your modem does PPPoE (as is common on DSL lines) it might not
be needed to lower the MTU below 1492 since the PPP header occupies only
8 bytes of the ethernet frame's payload.
Below is an example of the issue.
$ sudo ifconfig eth1 mtu 1500 # this is the default, which
now fails
$ ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
$ tracepath -4 -b -l 1500 -m 5 1.1.1.1
1: e5.eyal.emu.id.au (192.168.2.5) 1.109ms #
this is the modem
2: no reply
3: no reply
4: no reply
5: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
## Does the above show that the server (.7) can talk to the modem (.5)
but the modem
## cannot talk to the external net?
## Can it be an issue with the next hop (the 4G service)?
$ tracepath -4 -b -l 1456 -m 5 1.1.1.1
1: e5.eyal.emu.id.au (192.168.2.5) 0.760ms
2: 10.155.144.3 (10.155.144.3) 74.809ms
3: 10.246.20.29 (10.246.20.29) 39.707ms
4: 101.115.95.250 (101.115.95.250) 38.656ms
5: 203-220-16-149.tpgi.com.au (203.220.16.149) 35.127ms
Too many hops: pmtu 1456
Resume: pmtu 1456
$ sudo ifconfig eth1 mtu 1456 # with this everything works well
$ ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
$ tracepath -4 -b -l 1500 -m 5 1.1.1.1
1?: [LOCALHOST] pmtu 1456
1: e5.eyal.emu.id.au (192.168.2.5) 0.797ms
1: e5.eyal.emu.id.au (192.168.2.5) 0.614ms
2: 10.155.144.2 (10.155.144.2) 37.951ms
3: 10.246.20.29 (10.246.20.29) 36.881ms
4: 101.115.95.249 (101.115.95.249) 35.653ms
5: 203-220-16-145.tpgi.com.au (203.220.16.145) 36.673ms
Too many hops: pmtu 1456
Resume: pmtu 1456
$ tracepath -4 -b -l 1456 -m 5 1.1.1.1
1: e5.eyal.emu.id.au (192.168.2.5) 0.560ms
2: 10.155.144.2 (10.155.144.2) 33.008ms
3: 10.246.20.29 (10.246.20.29) 36.226ms
4: 101.115.95.250 (101.115.95.250) 36.278ms
5: 203-220-16-149.tpgi.com.au (203.220.16.149) 36.156ms
Too many hops: pmtu 1456
Resume: pmtu 1456
--
Regards
mks
7:27 a.m.
On 2/12/23 23:01, Markus Schönhaber wrote:
02.12.23, 11:33 +0100, fedora(a)eyal.emu.id.au:
> For the last few weeks I am trying to pinpoint the source of a network
> problem.
>
> The short story: using the standard mtu of 1500 caused sending to stall
> (and fail).
> This is seen when uploading a file (e.g. ftp) or when sending email.
> With an mtu of 1456 everything works.
>
> Everything worked unto 02:30 on 7/Nov. At that point some uploads
> started hanging.
> I have cron scripts that upload small and larger files. The small ones
> get though, the larger ones hang.
> There is enough space on the target.
Assuming that the problem's not on your side of the network, the most
likely cause I can think of, is that now somewhere along the path ICMP
packets signalling the need for fragmentation get dropped, and therefore
the sender doesn't learn that it should send smaller packets.
You might want to double check that your network doesn't behave that way.
I will double check, though the network worked well until then, so I cannot see
how my side is the cause. The system was rebooted since and the problem persisted.
> The network is simple: my server connects to a modem (wireless
internet
> over 4G).
>
> BTW, is there a more suitable list (or forum) for discussing this issue?
>
> I want to confirm where the problem lies. I did not change a thing on my
> server.
> I suspect that the modem updated its fw overnight and the new fw has an
> issue.
> However the ISP says no one else is complaining.
>
> I did not find a way to downgrade the fw or install the old one.
> I was given a new modem (with the old fw) but once connected to the 'net
> the fw was updated, something I cannot control.
If the update of the modem's FW happened around the same time the
problem arose, that would point to the new FW as being the culprit
(maybe by dropping those "evil" ICMP packets as I mused above).
And as long as you're reasonably sure that the problem isn't caused by
some change *you* made, and you can't control the modem's behavior, you
could simply accept your fate, lower the interface MTU and move on.
I did not do any change to the firewall, which is probably the only place
I could break things (like dropping ICMP).
The modem has a feature "IPv4 SPI Firewall" and turning it off did not help
here.
Anyway, if icmp if dropped then I expect simple pings will not get though either, right?
BTW: if your modem does PPPoE (as is common on DSL lines) it might
not
be needed to lower the MTU below 1492 since the PPP header occupies only
8 bytes of the ethernet frame's payload.
No, I actually tried and any higher than 1456 and it fails.
This is a wireless (4G) internet modem.
> Below is an example of the issue.
>
> $ sudo ifconfig eth1 mtu 1500 # this is the default, which
> now fails
> $ ifconfig eth1
> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
>
> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> 1: e5.eyal.emu.id.au (192.168.2.5) 1.109ms #
> this is the modem
> 2: no reply
> 3: no reply
> 4: no reply
> 5: no reply
> Too many hops: pmtu 1500
> Resume: pmtu 1500
>
> ## Does the above show that the server (.7) can talk to the modem (.5)
> but the modem
> ## cannot talk to the external net?
> ## Can it be an issue with the next hop (the 4G service)?
>
> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.760ms
> 2: 10.155.144.3 (10.155.144.3) 74.809ms
> 3: 10.246.20.29 (10.246.20.29) 39.707ms
> 4: 101.115.95.250 (101.115.95.250) 38.656ms
> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 35.127ms
> Too many hops: pmtu 1456
> Resume: pmtu 1456
>
> $ sudo ifconfig eth1 mtu 1456 # with this everything works well
> $ ifconfig eth1
> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
> inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
>
> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> 1?: [LOCALHOST] pmtu 1456
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.797ms
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.614ms
> 2: 10.155.144.2 (10.155.144.2) 37.951ms
> 3: 10.246.20.29 (10.246.20.29) 36.881ms
> 4: 101.115.95.249 (101.115.95.249) 35.653ms
> 5: 203-220-16-145.tpgi.com.au (203.220.16.145) 36.673ms
> Too many hops: pmtu 1456
> Resume: pmtu 1456
>
> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.560ms
> 2: 10.155.144.2 (10.155.144.2) 33.008ms
> 3: 10.246.20.29 (10.246.20.29) 36.226ms
> 4: 101.115.95.250 (101.115.95.250) 36.278ms
> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 36.156ms
> Too many hops: pmtu 1456
> Resume: pmtu 1456
--
Eyal at Home (fedora(a)eyal.emu.id.au)
7:43 a.m.
My experience with the MTU being wrong is that often nothing works to
inform the sending end that the MTU is wrong.
The MTU being wrong (and below 1500) is not an expected/handled
condition for the TCP stack. meaning it will break for anything that
uses a MTU above the size, especially if the wrong MTU is a few
byte(or more) than any advertised MTU in the path.
So what does your network diagram look like?
On Sat, Dec 2, 2023 at 7:28 AM <fedora(a)eyal.emu.id.au> wrote:
>
> On 2/12/23 23:01, Markus Schönhaber wrote:
> > 02.12.23, 11:33 +0100, fedora(a)eyal.emu.id.au:
> >
> >> For the last few weeks I am trying to pinpoint the source of a network
> >> problem.
> >>
> >> The short story: using the standard mtu of 1500 caused sending to stall
> >> (and fail).
> >> This is seen when uploading a file (e.g. ftp) or when sending email.
> >> With an mtu of 1456 everything works.
> >>
> >> Everything worked unto 02:30 on 7/Nov. At that point some uploads
> >> started hanging.
> >> I have cron scripts that upload small and larger files. The small ones
> >> get though, the larger ones hang.
> >> There is enough space on the target.
> >
> > Assuming that the problem's not on your side of the network, the most
> > likely cause I can think of, is that now somewhere along the path ICMP
> > packets signalling the need for fragmentation get dropped, and therefore
> > the sender doesn't learn that it should send smaller packets.
> > You might want to double check that your network doesn't behave that way.
>
> I will double check, though the network worked well until then, so I cannot see
> how my side is the cause. The system was rebooted since and the problem persisted.
>
> >> The network is simple: my server connects to a modem (wireless internet
> >> over 4G).
> >>
> >> BTW, is there a more suitable list (or forum) for discussing this issue?
> >>
> >> I want to confirm where the problem lies. I did not change a thing on my
> >> server.
> >> I suspect that the modem updated its fw overnight and the new fw has an
> >> issue.
> >> However the ISP says no one else is complaining.
> >>
> >> I did not find a way to downgrade the fw or install the old one.
> >> I was given a new modem (with the old fw) but once connected to the
'net
> >> the fw was updated, something I cannot control.
> >
> > If the update of the modem's FW happened around the same time the
> > problem arose, that would point to the new FW as being the culprit
> > (maybe by dropping those "evil" ICMP packets as I mused above).
> > And as long as you're reasonably sure that the problem isn't caused by
> > some change *you* made, and you can't control the modem's behavior, you
> > could simply accept your fate, lower the interface MTU and move on.
>
> I did not do any change to the firewall, which is probably the only place
> I could break things (like dropping ICMP).
> The modem has a feature "IPv4 SPI Firewall" and turning it off did not help
here.
>
> Anyway, if icmp if dropped then I expect simple pings will not get though either,
right?
> > BTW: if your modem does PPPoE (as is common on DSL lines) it might not
> > be needed to lower the MTU below 1492 since the PPP header occupies only
> > 8 bytes of the ethernet frame's payload.
>
> No, I actually tried and any higher than 1456 and it fails.
> This is a wireless (4G) internet modem.
>
> >> Below is an example of the issue.
> >>
> >> $ sudo ifconfig eth1 mtu 1500 # this is the default, which
> >> now fails
> >> $ ifconfig eth1
> >> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> >> inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
> >>
> >> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> >> 1: e5.eyal.emu.id.au (192.168.2.5) 1.109ms #
> >> this is the modem
> >> 2: no reply
> >> 3: no reply
> >> 4: no reply
> >> 5: no reply
> >> Too many hops: pmtu 1500
> >> Resume: pmtu 1500
> >>
> >> ## Does the above show that the server (.7) can talk to the modem (.5)
> >> but the modem
> >> ## cannot talk to the external net?
> >> ## Can it be an issue with the next hop (the 4G service)?
> >>
> >> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.760ms
> >> 2: 10.155.144.3 (10.155.144.3) 74.809ms
> >> 3: 10.246.20.29 (10.246.20.29) 39.707ms
> >> 4: 101.115.95.250 (101.115.95.250) 38.656ms
> >> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 35.127ms
> >> Too many hops: pmtu 1456
> >> Resume: pmtu 1456
> >>
> >> $ sudo ifconfig eth1 mtu 1456 # with this everything works well
> >> $ ifconfig eth1
> >> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
> >> inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
> >>
> >> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> >> 1?: [LOCALHOST] pmtu 1456
> >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.797ms
> >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.614ms
> >> 2: 10.155.144.2 (10.155.144.2) 37.951ms
> >> 3: 10.246.20.29 (10.246.20.29) 36.881ms
> >> 4: 101.115.95.249 (101.115.95.249) 35.653ms
> >> 5: 203-220-16-145.tpgi.com.au (203.220.16.145) 36.673ms
> >> Too many hops: pmtu 1456
> >> Resume: pmtu 1456
> >>
> >> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.560ms
> >> 2: 10.155.144.2 (10.155.144.2) 33.008ms
> >> 3: 10.246.20.29 (10.246.20.29) 36.226ms
> >> 4: 101.115.95.250 (101.115.95.250) 36.278ms
> >> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 36.156ms
> >> Too many hops: pmtu 1456
> >> Resume: pmtu 1456
> >
>
> --
> Eyal at Home (fedora(a)eyal.emu.id.au)
> --
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
7:57 a.m.
A few bytes or more less than any advertised.
The underlying issue is that I advertise say 1500bytes MTU (when asked
or when fragmenting) but only allow say 1496 to actually pass.
Usually when I have seen it happen it has been a switch that has an
enforced MTU limit set, but it could be any device that advertises the
MTU as X but only actually allows X-4 (or smaller).
The it10t device does not know it needs to fragment and as such it
does not/cannot report that it needs to fragment the packet, it
typically simply drops the package that it thinks is too large on the
ground and tells no one about it.
it might be something about how your network is put together and/or
how you are using it that exposes the bug in whatever device it is. I
have seen vendors have stuff that simply could not work but no one
could understand why anything was working given what was found. Code
that should not work often "works" well enough for developers to think
it works.
What kind of devices do you have between your linux node and the
providers network? Ie the modem and any routers.
On Sat, Dec 2, 2023 at 7:43 AM Roger Heflin <rogerheflin(a)gmail.com> wrote:
>
> My experience with the MTU being wrong is that often nothing works to
> inform the sending end that the MTU is wrong.
>
> The MTU being wrong (and below 1500) is not an expected/handled
> condition for the TCP stack. meaning it will break for anything that
> uses a MTU above the size, especially if the wrong MTU is a few
> byte(or more) than any advertised MTU in the path.
>
> So what does your network diagram look like?
>
>
> On Sat, Dec 2, 2023 at 7:28 AM <fedora(a)eyal.emu.id.au> wrote:
> >
> > On 2/12/23 23:01, Markus Schönhaber wrote:
> > > 02.12.23, 11:33 +0100, fedora(a)eyal.emu.id.au:
> > >
> > >> For the last few weeks I am trying to pinpoint the source of a network
> > >> problem.
> > >>
> > >> The short story: using the standard mtu of 1500 caused sending to
stall
> > >> (and fail).
> > >> This is seen when uploading a file (e.g. ftp) or when sending email.
> > >> With an mtu of 1456 everything works.
> > >>
> > >> Everything worked unto 02:30 on 7/Nov. At that point some uploads
> > >> started hanging.
> > >> I have cron scripts that upload small and larger files. The small ones
> > >> get though, the larger ones hang.
> > >> There is enough space on the target.
> > >
> > > Assuming that the problem's not on your side of the network, the most
> > > likely cause I can think of, is that now somewhere along the path ICMP
> > > packets signalling the need for fragmentation get dropped, and therefore
> > > the sender doesn't learn that it should send smaller packets.
> > > You might want to double check that your network doesn't behave that
way.
> >
> > I will double check, though the network worked well until then, so I cannot see
> > how my side is the cause. The system was rebooted since and the problem
persisted.
> >
> > >> The network is simple: my server connects to a modem (wireless
internet
> > >> over 4G).
> > >>
> > >> BTW, is there a more suitable list (or forum) for discussing this
issue?
> > >>
> > >> I want to confirm where the problem lies. I did not change a thing on
my
> > >> server.
> > >> I suspect that the modem updated its fw overnight and the new fw has
an
> > >> issue.
> > >> However the ISP says no one else is complaining.
> > >>
> > >> I did not find a way to downgrade the fw or install the old one.
> > >> I was given a new modem (with the old fw) but once connected to the
'net
> > >> the fw was updated, something I cannot control.
> > >
> > > If the update of the modem's FW happened around the same time the
> > > problem arose, that would point to the new FW as being the culprit
> > > (maybe by dropping those "evil" ICMP packets as I mused above).
> > > And as long as you're reasonably sure that the problem isn't caused
by
> > > some change *you* made, and you can't control the modem's behavior,
you
> > > could simply accept your fate, lower the interface MTU and move on.
> >
> > I did not do any change to the firewall, which is probably the only place
> > I could break things (like dropping ICMP).
> > The modem has a feature "IPv4 SPI Firewall" and turning it off did not
help here.
> >
> > Anyway, if icmp if dropped then I expect simple pings will not get though
either, right?
> > > BTW: if your modem does PPPoE (as is common on DSL lines) it might not
> > > be needed to lower the MTU below 1492 since the PPP header occupies only
> > > 8 bytes of the ethernet frame's payload.
> >
> > No, I actually tried and any higher than 1456 and it fails.
> > This is a wireless (4G) internet modem.
> >
> > >> Below is an example of the issue.
> > >>
> > >> $ sudo ifconfig eth1 mtu 1500 # this is the default, which
> > >> now fails
> > >> $ ifconfig eth1
> > >> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> > >> inet 192.168.2.7 netmask 255.255.255.0 broadcast
192.168.2.255
> > >>
> > >> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> > >> 1: e5.eyal.emu.id.au (192.168.2.5) 1.109ms
#
> > >> this is the modem
> > >> 2: no reply
> > >> 3: no reply
> > >> 4: no reply
> > >> 5: no reply
> > >> Too many hops: pmtu 1500
> > >> Resume: pmtu 1500
> > >>
> > >> ## Does the above show that the server (.7) can talk to the modem (.5)
> > >> but the modem
> > >> ## cannot talk to the external net?
> > >> ## Can it be an issue with the next hop (the 4G service)?
> > >>
> > >> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> > >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.760ms
> > >> 2: 10.155.144.3 (10.155.144.3) 74.809ms
> > >> 3: 10.246.20.29 (10.246.20.29) 39.707ms
> > >> 4: 101.115.95.250 (101.115.95.250) 38.656ms
> > >> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 35.127ms
> > >> Too many hops: pmtu 1456
> > >> Resume: pmtu 1456
> > >>
> > >> $ sudo ifconfig eth1 mtu 1456 # with this everything works
well
> > >> $ ifconfig eth1
> > >> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
> > >> inet 192.168.2.7 netmask 255.255.255.0 broadcast
192.168.2.255
> > >>
> > >> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> > >> 1?: [LOCALHOST] pmtu 1456
> > >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.797ms
> > >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.614ms
> > >> 2: 10.155.144.2 (10.155.144.2) 37.951ms
> > >> 3: 10.246.20.29 (10.246.20.29) 36.881ms
> > >> 4: 101.115.95.249 (101.115.95.249) 35.653ms
> > >> 5: 203-220-16-145.tpgi.com.au (203.220.16.145) 36.673ms
> > >> Too many hops: pmtu 1456
> > >> Resume: pmtu 1456
> > >>
> > >> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> > >> 1: e5.eyal.emu.id.au (192.168.2.5) 0.560ms
> > >> 2: 10.155.144.2 (10.155.144.2) 33.008ms
> > >> 3: 10.246.20.29 (10.246.20.29) 36.226ms
> > >> 4: 101.115.95.250 (101.115.95.250) 36.278ms
> > >> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 36.156ms
> > >> Too many hops: pmtu 1456
> > >> Resume: pmtu 1456
> > >
> >
> > --
> > Eyal at Home (fedora(a)eyal.emu.id.au)
> > --
> > _______________________________________________
> > users mailing list -- users(a)lists.fedoraproject.org
> > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> > Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
3:16 p.m.
On 3/12/23 00:43, Roger Heflin wrote:
My experience with the MTU being wrong is that often nothing works
to
inform the sending end that the MTU is wrong.
The MTU being wrong (and below 1500) is not an expected/handled
condition for the TCP stack. meaning it will break for anything that
uses a MTU above the size, especially if the wrong MTU is a few
byte(or more) than any advertised MTU in the path.
So what does your network diagram look like?
A rather simple setup.
server
\ \
\ eth1 192.168.2.7 --- 192.168.2.5 - modem --- 4G
eth0 192.168.3.7 --- switch -- home network
modem is TP-Link VX420-G2h supplied by the ISP.
My main clue is that everything worked for a year with this modem,
then unexpectedly, overnight, the problem appeared and still persist.
[trimmed]
--
Eyal at Home (fedora(a)eyal.emu.id.au)
6:30 p.m.
On Sat, Dec 2, 2023 at 3:16 PM <fedora(a)eyal.emu.id.au> wrote:
On 3/12/23 00:43, Roger Heflin wrote:
> My experience with the MTU being wrong is that often nothing works to
> inform the sending end that the MTU is wrong.
>
> The MTU being wrong (and below 1500) is not an expected/handled
> condition for the TCP stack. meaning it will break for anything that
> uses a MTU above the size, especially if the wrong MTU is a few
> byte(or more) than any advertised MTU in the path.
>
> So what does your network diagram look like?
A rather simple setup.
server
\ \
\ eth1 192.168.2.7 --- 192.168.2.5 - modem --- 4G
eth0 192.168.3.7 --- switch -- home network
modem is TP-Link VX420-G2h supplied by the ISP.
My main clue is that everything worked for a year with this modem,
then unexpectedly, overnight, the problem appeared and still persist.
[trimmed]
Have you logged into the modem and checked if it has any settings?
You might also test a device on wifi and/or login to the router and
send a test ping from inside the router with a large size.
What does the network config for 192.168.2.7 look like? And you do
the pings from the server connected to both?
It could be the reason no one else is seeing this is that they are not
using the wired network and that is what they screwed up.
6:50 p.m.
On 3/12/23 11:30, Roger Heflin wrote:
On Sat, Dec 2, 2023 at 3:16 PM <fedora(a)eyal.emu.id.au> wrote:
>
> On 3/12/23 00:43, Roger Heflin wrote:
>> My experience with the MTU being wrong is that often nothing works to
>> inform the sending end that the MTU is wrong.
>>
>> The MTU being wrong (and below 1500) is not an expected/handled
>> condition for the TCP stack. meaning it will break for anything that
>> uses a MTU above the size, especially if the wrong MTU is a few
>> byte(or more) than any advertised MTU in the path.
>>
>> So what does your network diagram look like?
>
> A rather simple setup.
>
> server
> \ \
> \ eth1 192.168.2.7 --- 192.168.2.5 - modem --- 4G
> eth0 192.168.3.7 --- switch -- home network
>
> modem is TP-Link VX420-G2h supplied by the ISP.
>
> My main clue is that everything worked for a year with this modem,
> then unexpectedly, overnight, the problem appeared and still persist.
>
> [trimmed]
Have you logged into the modem and checked if it has any settings?
I cannot log in, but the menu offered by the modem does not show any relevant settings.
You might also test a device on wifi and/or login to the router and
send a test ping from inside the router with a large size.
WiFi is turned off on the modem, but I might give it a try.
What does the network config for 192.168.2.7 look like? And you do
the pings from the server connected to both?
$ ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
ether 00:1b:21:3a:56:72 txqueuelen 1000 (Ethernet)
RX packets 4009354 bytes 4646140675 (4.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1465729 bytes 195820660 (186.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18 memory 0x515c0000-515e0000
I do the pings and tracepath from this server.
It could be the reason no one else is seeing this is that they are
not
using the wired network and that is what they screwed up.
The ISP finally confirmed that other users have issues too. Good news I guess?
--
Eyal at Home (fedora(a)eyal.emu.id.au)
7:17 p.m.
I cannot log in, but the menu offered by the modem does not show any relevant settings.
> You might also test a device on wifi and/or login to the router and
> send a test ping from inside the router with a large size.
WiFi is turned off on the modem, but I might give it a try.
> What does the network config for 192.168.2.7 look like? And you do
> the pings from the server connected to both?
$ ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
ether 00:1b:21:3a:56:72 txqueuelen 1000 (Ethernet)
RX packets 4009354 bytes 4646140675 (4.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1465729 bytes 195820660 (186.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18 memory 0x515c0000-515e0000
I do the pings and tracepath from this server.
> It could be the reason no one else is seeing this is that they are not
> using the wired network and that is what they screwed up.
The ISP finally confirmed that other users have issues too. Good news I guess?
That is good news. It means they will talk to whoever their vendor
support is. What firmware version is on the device?
I think I found what the device is and there were 2 firmware updates
recently, and the 2nd one was 3 days after the first and that is
usually a sign that something serious was wrong with the first update.
10:04 p.m.
On 3/12/23 12:17, Roger Heflin wrote:
>
> I cannot log in, but the menu offered by the modem does not show any relevant
settings.
>
>> You might also test a device on wifi and/or login to the router and
>> send a test ping from inside the router with a large size.
>
> WiFi is turned off on the modem, but I might give it a try.
>
>> What does the network config for 192.168.2.7 look like? And you do
>> the pings from the server connected to both?
>
> $ ifconfig eth1
> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
> inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
> ether 00:1b:21:3a:56:72 txqueuelen 1000 (Ethernet)
> RX packets 4009354 bytes 4646140675 (4.3 GiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 1465729 bytes 195820660 (186.7 MiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
> device interrupt 18 memory 0x515c0000-515e0000
>
> I do the pings and tracepath from this server.
>
>> It could be the reason no one else is seeing this is that they are not
>> using the wired network and that is what they screwed up.
>
> The ISP finally confirmed that other users have issues too. Good news I guess?
>
That is good news. It means they will talk to whoever their vendor
support is. What firmware version is on the device?
The modem is a TP-Link VX420-G2h 4G.
The original firmware (on original and replacement modem) is:
Firmware Version:2.0.0 0.9 v603c.0 Build 220126 Rel.13161n as recorded when
received
and the current one is:
Firmware Version:2.0.0 0.9 v603c.0 Build 221114 Rel.39966n as is now
I think I found what the device is and there were 2 firmware updates
recently, and the 2nd one was 3 days after the first and that is
usually a sign that something serious was wrong with the first update.
--
Eyal at home (fedora(a)eyal.emu.id.au)
10:47 a.m.
02.12.23, 14:27 +0100, fedora(a)eyal.emu.id.au:
I did not do any change to the firewall, which is probably the only
place
I could break things (like dropping ICMP).
The modem has a feature "IPv4 SPI Firewall" and turning it off did not
help here.
Well, I'm out of ideas, then.
Anyway, if icmp if dropped then I expect simple pings will not get
though either, right?
Not necessarily. A packet filter might let one type of ICMP packets
through while blocking other ICMP types/codes.
But you can simply try if pings get through.
What you could also do is make a recording of network traffic when
trying to transmit files with a high MTU setting and look if you find
any ICMP type 3 code 4 packets in there.
--
Regards
mks
6:43 p.m.
On 3/12/23 03:47, Markus Schönhaber wrote:
02.12.23, 14:27 +0100, fedora(a)eyal.emu.id.au:
> I did not do any change to the firewall, which is probably the only place
> I could break things (like dropping ICMP).
> The modem has a feature "IPv4 SPI Firewall" and turning it off did not
> help here.
Well, I'm out of ideas, then.
> Anyway, if icmp if dropped then I expect simple pings will not get
> though either, right?
Not necessarily. A packet filter might let one type of ICMP packets
through while blocking other ICMP types/codes.
But you can simply try if pings get through.
What you could also do is make a recording of network traffic when
trying to transmit files with a high MTU setting and look if you find
any ICMP type 3 code 4 packets in there.
I have done so and can see how there are no ICMP replies beyond the modem.
One example:
$ sudo ifconfig eth1 mtu 1500
$ date ; tracepath -4 -n -l 1500 -m 3 1.1.1.1 ; date
Sun Dec 3 10:43:49 AEDT 2023
1: 192.168.2.5 1.147ms
2: no reply
3: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
Sun Dec 3 10:43:55 AEDT 2023
2023-12-03 10:43:49.201984 IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17),
length 1500)
192.168.2.7.46132 > 1.1.1.1.44444: UDP, length 1472
2023-12-03 10:43:49.202503 IP (tos 0xc0, ttl 64, id 22137, offset 0, flags [none], proto
ICMP (1), length 576)
192.168.2.5 > 192.168.2.7: ICMP time exceeded in-transit, length 556
IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
192.168.2.7.46132 > 1.1.1.1.44444: UDP, length 1472
2023-12-03 10:43:49.202768 IP (tos 0x0, ttl 2, id 0, offset 0, flags [DF], proto UDP (17),
length 1500)
192.168.2.7.46132 > 1.1.1.1.44445: UDP, length 1472
2023-12-03 10:43:50.203883 IP (tos 0x0, ttl 2, id 0, offset 0, flags [DF], proto UDP (17),
length 1500)
192.168.2.7.46132 > 1.1.1.1.44446: UDP, length 1472
2023-12-03 10:43:51.205034 IP (tos 0x0, ttl 2, id 0, offset 0, flags [DF], proto UDP (17),
length 1500)
192.168.2.7.46132 > 1.1.1.1.44447: UDP, length 1472
2023-12-03 10:43:52.206217 IP (tos 0x0, ttl 3, id 0, offset 0, flags [DF], proto UDP (17),
length 1500)
192.168.2.7.46132 > 1.1.1.1.44448: UDP, length 1472
2023-12-03 10:43:53.207372 IP (tos 0x0, ttl 3, id 0, offset 0, flags [DF], proto UDP (17),
length 1500)
192.168.2.7.46132 > 1.1.1.1.44449: UDP, length 1472
2023-12-03 10:43:54.207878 IP (tos 0x0, ttl 3, id 0, offset 0, flags [DF], proto UDP (17),
length 1500)
192.168.2.7.46132 > 1.1.1.1.44450: UDP, length 1472
Some good news: I just had a call from my ISP who finally admitted that I am not the only
user with a similar issue. I emailed them my test logs. Maybe they will resolve this.
--
Eyal at Home (fedora(a)eyal.emu.id.au)
7:28 a.m.
There is nothing wrong with leaving the mtu lower. At worst you need
one more packet.
Knowing hardware providers and internet providers. "no one else is
reporting this" means we have not noticed that anyone else reported
this, and/or there are only 2 others using this and/or we have not put
together that others have this same issue. I would just leave the
current fix and maybe every so often see if it is still broken.
Getting through vendor tech support is typically almost impossible,
they asusmping is there are no issues like this and that nobody should
be passed on to the next level. And from my experience (debugging
this same issue) is even if you get to a 3rd line network engineer
they will argue that it is not their problem and you will have to work
hard to get it past them (I am known as the troubleshooting expert in
my company, and the networking experts argued that my test and results
was wrong, it had to be a host problem--"expert" specialist rule #1 if
I cannot find any issue then nothing is broken and then it must be
someone else's issue, blame someone else and close the ticket).
90+% of the "experts" aren't experts, they are trained monkeys that
know process, but don't have the slightest idea what the magic process
does, and often have extra useless steps in their process that they
don't understand that they think are doing something.
On Sat, Dec 2, 2023 at 4:34 AM <fedora(a)eyal.emu.id.au> wrote:
>
> For the last few weeks I am trying to pinpoint the source of a network problem.
>
> The short story: using the standard mtu of 1500 caused sending to stall (and fail).
> This is seen when uploading a file (e.g. ftp) or when sending email.
> With an mtu of 1456 everything works.
>
> Everything worked unto 02:30 on 7/Nov. At that point some uploads started hanging.
> I have cron scripts that upload small and larger files. The small ones get though,
the larger ones hang.
> There is enough space on the target.
>
> The network is simple: my server connects to a modem (wireless internet over 4G).
>
> BTW, is there a more suitable list (or forum) for discussing this issue?
>
> I want to confirm where the problem lies. I did not change a thing on my server.
> I suspect that the modem updated its fw overnight and the new fw has an issue.
> However the ISP says no one else is complaining.
>
> I did not find a way to downgrade the fw or install the old one.
> I was given a new modem (with the old fw) but once connected to the 'net
> the fw was updated, something I cannot control.
>
> Below is an example of the issue.
>
> $ sudo ifconfig eth1 mtu 1500 # this is the default, which now
fails
> $ ifconfig eth1
> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
>
> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> 1: e5.eyal.emu.id.au (192.168.2.5) 1.109ms # this is the
modem
> 2: no reply
> 3: no reply
> 4: no reply
> 5: no reply
> Too many hops: pmtu 1500
> Resume: pmtu 1500
>
> ## Does the above show that the server (.7) can talk to the modem (.5) but the modem
> ## cannot talk to the external net?
> ## Can it be an issue with the next hop (the 4G service)?
>
> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.760ms
> 2: 10.155.144.3 (10.155.144.3) 74.809ms
> 3: 10.246.20.29 (10.246.20.29) 39.707ms
> 4: 101.115.95.250 (101.115.95.250) 38.656ms
> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 35.127ms
> Too many hops: pmtu 1456
> Resume: pmtu 1456
>
> $ sudo ifconfig eth1 mtu 1456 # with this everything works well
> $ ifconfig eth1
> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1456
> inet 192.168.2.7 netmask 255.255.255.0 broadcast 192.168.2.255
>
> $ tracepath -4 -b -l 1500 -m 5 1.1.1.1
> 1?: [LOCALHOST] pmtu 1456
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.797ms
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.614ms
> 2: 10.155.144.2 (10.155.144.2) 37.951ms
> 3: 10.246.20.29 (10.246.20.29) 36.881ms
> 4: 101.115.95.249 (101.115.95.249) 35.653ms
> 5: 203-220-16-145.tpgi.com.au (203.220.16.145) 36.673ms
> Too many hops: pmtu 1456
> Resume: pmtu 1456
>
> $ tracepath -4 -b -l 1456 -m 5 1.1.1.1
> 1: e5.eyal.emu.id.au (192.168.2.5) 0.560ms
> 2: 10.155.144.2 (10.155.144.2) 33.008ms
> 3: 10.246.20.29 (10.246.20.29) 36.226ms
> 4: 101.115.95.250 (101.115.95.250) 36.278ms
> 5: 203-220-16-149.tpgi.com.au (203.220.16.149) 36.156ms
> Too many hops: pmtu 1456
> Resume: pmtu 1456
>
> --
> Eyal at Home (fedora(a)eyal.emu.id.au)
> --
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
178
days inactive
179
days old
12 comments
3 participants
participants (3)
-
fedora＠eyal.emu.id.au -
Markus Schönhaber -
Roger Heflin