[PVE-User] System freeze after kernel update to 4.15.15-1-pve

[Harald Leithner]

Hi,

yesterday we did a kernel update on one of the cluster nodes to
4.15.15-1-pve, after some hours the node freezes and got fenced.

There was no kernel panic...

While rebooting the kernel crashed, you can find the screenshots at:

https://privatebin.at/?450815b1e2b6977f#OZF5/DEC8pwJHM8NVMR0/ODogOPS3U1o738Riz813F4=

https://privatebin.at/?37d273119ba7acf4#YeJY6Y3K52vQPrncUvBfe+Ah8rzS+FD+6arkQHlRxAI=

I booted the old kernel 4.13.16-2-pve without problems.

Specs:
3 node cluster with ceph.
Supermicro SYS-1029U-TR4T
2x Intel Xeon Silver 4108 1,8 GHz Box Sockel 3647
64 GB DDR4

Btw. live migration from older kernel seams to fail too, but that hasn't
been tested much.

thx

Harald

[Thomas Lamprecht]

Hi,

Post by Harald Leithner
Hi,
yesterday we did a kernel update on one of the cluster nodes to
4.15.15-1-pve, after some hours the node freezes and got fenced.

There seem to be some IO regressions in this kernel release,
we updated it and pushed a version to pvetest which should include fixes
or revert the respective commits. Can you please try
pve-kernel-4.15.17-1-pve_4.15.17-8 [1]

if possible. Users in our Forum reported that this version fixed their
issues.

Post by Harald Leithner
There was no kernel panic...
https://privatebin.at/?450815b1e2b6977f#OZF5/DEC8pwJHM8NVMR0/ODogOPS3U1o738Riz813F4=
https://privatebin.at/?37d273119ba7acf4#YeJY6Y3K52vQPrncUvBfe+Ah8rzS+FD+6arkQHlRxAI=
I booted the old kernel 4.13.16-2-pve without problems.
3 node cluster with ceph.
Supermicro SYS-1029U-TR4T
2x Intel Xeon Silver 4108 1,8 GHz Box Sockel 3647
64 GB DDR4
Btw. live migration from older kernel seams to fail too, but that hasn't
been tested much.

Hmm, from 4.13 to 4.15? I'll look into that...

cheers,
Thomas

[1]:
http://download.proxmox.com/debian/pve/dists/stretch/pvetest/binary-amd64/pve-kernel-4.15.17-1-pve_4.15.17-8_amd64.deb

[Nurullah Ciftci]

Hi,

Linux Kernel 4.15 is end-of-life. Greg Kroah-Hartman, a linux kernel
maintainer, said that all users of the 4.15 kernel series must upgrade.
Is there any timeline ugrading 4.16 kernel to Proxmox?

https://news.softpedia.com/news/linux-kernel-4-15-reached-end-of-life-users-urged-to-move-to-linux-4-16-now-520787.shtml

Post by Thomas Lamprecht
Hi,

Hi, yesterday we did a kernel update on one of the cluster nodes to 4.15.15-1-pve, after some hours the node freezes and got fenced.

There seem to be some IO regressions in this kernel release,
we updated it and pushed a version to pvetest which should include fixes
or revert the respective commits. Can you please try
pve-kernel-4.15.17-1-pve_4.15.17-8 [1]
if possible. Users in our Forum reported that this version fixed their
issues.

There was no kernel panic... While rebooting the kernel crashed, you can find the screenshots at: https://privatebin.at/?450815b1e2b6977f#OZF5/DEC8pwJHM8NVMR0/ODogOPS3U1o738Riz813F4= [1] https://privatebin.at/?37d273119ba7acf4#YeJY6Y3K52vQPrncUvBfe+Ah8rzS+FD+6arkQHlRxAI= [2] I booted the old kernel 4.13.16-2-pve without problems. Specs: 3 node cluster with ceph. Supermicro SYS-1029U-TR4T 2x Intel Xeon Silver 4108 1,8 GHz Box Sockel 3647 64 GB DDR4 Btw. live migration from older kernel seams to fail too, but that hasn't been tested much.

Hmm, from 4.13 to 4.15? I'll look into that...
cheers,
Thomas
http://download.proxmox.com/debian/pve/dists/stretch/pvetest/binary-amd64/pve-kernel-4.15.17-1-pve_4.15.17-8_amd64.deb [3]
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user [4]

Links:
------
[1]
https://privatebin.at/?450815b1e2b6977f#OZF5/DEC8pwJHM8NVMR0/ODogOPS3U1o738Riz813F4=
[2]
https://privatebin.at/?37d273119ba7acf4#YeJY6Y3K52vQPrncUvBfe+Ah8rzS+FD+6arkQHlRxAI=
[3]
http://download.proxmox.com/debian/pve/dists/stretch/pvetest/binary-amd64/pve-kernel-4.15.17-1-pve_4.15.17-8_amd64.deb
[4] https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

------------------------------------------------------------------------------------------
Bu elektronik posta ve onunla iletilen bütün ekler (varsa) sadece göndericisi tarafýndan almasý amaçlanan yetkili gerçek ya da tüzel kiþinin kullanýmý içindir. Eðer söz konusu yetkili alici deðilseniz bu elektronik postayý derhal silmeniz gerekmektedir.

This e-mail and attachments (if any) transferred are strictly confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient the e-mail should immediately be deleted.

[Thomas Lamprecht]

Hi,

Post by Nurullah Ciftci
Linux Kernel 4.15 is end-of-life. Greg Kroah-Hartman, a linux kernel
maintainer, said that all users of the 4.15 kernel series must upgrade.
Is there any timeline ugrading 4.16 kernel to Proxmox?
https://news.softpedia.com/news/linux-kernel-4-15-reached-end-of-life-users-urged-to-move-to-linux-4-16-now-520787.shtml

It's EOL for kernel.org, the highest kernel upstream. We base ours on
Ubuntu's 18.04 LTS, which will get support (i.e. bug (security and
others) fixes, new hardware support, etc. until 2023 (new hardware
"only" until 2020, though)[1] by Ubuntu's and our kernel people.

So that's not a problem. We'll keep the 4.15 kernel through the
remaining lifetime of PVE 5.X, which will be EOL'ed probably someday in
2020, if I had to guess [2].

[1]: https://www.ubuntu.com/info/release-end-of-life
[2]: https://pve.proxmox.com/wiki/FAQ (Point 10)

cheers,
Thomas

Post by Nurullah Ciftci

Post by Thomas Lamprecht
Hi,

Hi, yesterday we did a kernel update on one of the cluster nodes to 4.15.15-1-pve, after some hours the node freezes and got fenced.

There seem to be some IO regressions in this kernel release,
we updated it and pushed a version to pvetest which should include fixes
or revert the respective commits. Can you please try
pve-kernel-4.15.17-1-pve_4.15.17-8 [1]
if possible. Users in our Forum reported that this version fixed their
issues.

There was no kernel panic... While rebooting the kernel crashed, you can find the screenshots at: https://privatebin.at/?450815b1e2b6977f#OZF5/DEC8pwJHM8NVMR0/ODogOPS3U1o738Riz813F4= [1] https://privatebin.at/?37d273119ba7acf4#YeJY6Y3K52vQPrncUvBfe+Ah8rzS+FD+6arkQHlRxAI= [2] I booted the old kernel 4.13.16-2-pve without problems. Specs: 3 node cluster with ceph. Supermicro SYS-1029U-TR4T 2x Intel Xeon Silver 4108 1,8 GHz Box Sockel 3647 64 GB DDR4 Btw. live migration from older kernel seams to fail too, but that hasn't been tested much.

Hmm, from 4.13 to 4.15? I'll look into that...
cheers,
Thomas
http://download.proxmox.com/debian/pve/dists/stretch/pvetest/binary-amd64/pve-kernel-4.15.17-1-pve_4.15.17-8_amd64.deb

[Harald Leithner]

Hi,

thx, atleast it boots now.

If it freezes again, I will report back.

Migration worked as well, maybe it was related but not visible for me.

Harald

Post by Thomas Lamprecht
Hi,

Post by Harald Leithner
Hi,
yesterday we did a kernel update on one of the cluster nodes to
4.15.15-1-pve, after some hours the node freezes and got fenced.

There seem to be some IO regressions in this kernel release,
we updated it and pushed a version to pvetest which should include fixes
or revert the respective commits. Can you please try
pve-kernel-4.15.17-1-pve_4.15.17-8 [1]
if possible. Users in our Forum reported that this version fixed their
issues.

Post by Harald Leithner
There was no kernel panic...
https://privatebin.at/?450815b1e2b6977f#OZF5/DEC8pwJHM8NVMR0/ODogOPS3U1o738Riz813F4=
https://privatebin.at/?37d273119ba7acf4#YeJY6Y3K52vQPrncUvBfe+Ah8rzS+FD+6arkQHlRxAI=
I booted the old kernel 4.13.16-2-pve without problems.
3 node cluster with ceph.
Supermicro SYS-1029U-TR4T
2x Intel Xeon Silver 4108 1,8 GHz Box Sockel 3647
64 GB DDR4
Btw. live migration from older kernel seams to fail too, but that hasn't
been tested much.

Hmm, from 4.13 to 4.15? I'll look into that...
cheers,
Thomas
http://download.proxmox.com/debian/pve/dists/stretch/pvetest/binary-amd64/pve-kernel-4.15.17-1-pve_4.15.17-8_amd64.deb