Hi Marcus,
no, I haven't disabled Spectre/Meltdown mitigations (yet).
For 4 of my hosts I have the Intel microcode from 2018-05-08 running which is the most up-to-date version from Ubuntu.
Using the spectre-meltdown-checker.sh script from https://github.com/speed47/spectre-meltdown-checker gives below output which
let's me believe that performace should be goot with mitigations enabled.
#### output for sandybridge based hosts ####
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch: NO
* Kernel has mask_nospec64 (arm64): NO
STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization)
CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW)
* Mitigation 1
* Kernel is compiled with IBRS support: YES
* IBRS enabled and active: YES (for kernel and firmware code)
* Kernel is compiled with IBPB support: YES
* IBPB enabled and active: YES
* Mitigation 2
* Kernel has branch predictor hardening (arm): NO
* Kernel compiled with retpoline option: YES
* Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation)
STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability)
CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface: YES (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI): YES
* PTI enabled and active: YES
* Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced)
* Running as a Xen PV DomU: NO
STATUS: NOT VULNERABLE (Mitigation: PTI)
CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
* CPU microcode mitigates the vulnerability: YES
STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability)
CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
* Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
* Kernel supports speculation store bypass: YES (found in /proc/self/status)
STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
* Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)
STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)
##### end #####
The other 2 hosts are base on Westmere architecture and won't get microcode updates as fas as I know. Disabling mitigations on
these hosts might be worth a try. Same script from above gives:
#### output of westmere based hosts ####
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch: NO
* Kernel has mask_nospec64 (arm64): NO
STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization)
CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline)
* Mitigation 1
* Kernel is compiled with IBRS support: YES
* IBRS enabled and active: NO
* Kernel is compiled with IBPB support: YES
* IBPB enabled and active: NO
* Mitigation 2
* Kernel has branch predictor hardening (arm): NO
* Kernel compiled with retpoline option: YES
* Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation)
STATUS: NOT VULNERABLE (Full retpoline is mitigating the vulnerability)
IBPB is considered as a good addition to retpoline for Variant 2 mitigation, but your CPU microcode doesn't support it
CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface: YES (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI): YES
* PTI enabled and active: YES
* Reduced performance impact of PTI: YES (CPU supports PCID, performance impact of PTI will be reduced)
* Running as a Xen PV DomU: NO
STATUS: NOT VULNERABLE (Mitigation: PTI)
CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
* CPU microcode mitigates the vulnerability: NO
STATUS: VULNERABLE (an up-to-date CPU microcode is needed to mitigate this vulnerability)
CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
* Mitigated according to the /sys interface: NO (Vulnerable)
* Kernel supports speculation store bypass: YES (found in /proc/self/status)
STATUS: VULNERABLE (Your CPU doesn't support SSBD)
CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
* Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)
STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)
##### end #####
Hi,
(taken from the ceph list)
This is PTI, I think. Try to add "noibrs noibpb nopti nospectre_v2" to
kernel cmdline and reboot.
Did this make a difference ?
We are struggeling with proxmox/ceph too and we have the suspect that it is kernel related or network related,
but could not narrow it down to a specific reason.
But the effects are different... We encountered stuck I/O on rdb devices.
And kernel says it is losing a mon connection and hunting for a new mon all the time (when backup takes
place and heavy I/O is done).
Marcus Haarmann
----------------------------------------------------------------------------------------------------------------------------------
*Gesendet: *Mittwoch, 22. August 2018 10:50:19
*Betreff: *Re: [PVE-User] PVE kernel (sources, build process, etc.)
Post by Thomas LamprechtPost by Uwe SauterPost by Thomas LamprechtPost by Uwe Sauter* pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ?
Yes. (Note that this may not get much updates anymore)
Post by Uwe Sauter* pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ?
Yes. We're normally on the latest stable release tagged on the master branch.
I'll checkout both and compare the myri10ge drivers…
What's your exact issue, if I may ask?
cheers,
Thomas
Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those
are gone.
Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs.
Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different
settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well.
https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html
https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html
https://marc.info/?l=ceph-devel&m=153449419830984&w=2
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user