How to handle the tcp_mark_head_lost error reported by the Linux system
|
Problem Description Recently, a host reported the following kernel information: Jul 8 10:47:42 cztest kernel: ------------[ cut here ]------------ Jul 8 10:47:42 cztest kernel: WARNING: at net/ipv4/tcp_input.c:2269 tcp_mark_head_lost+0x113/0x290() Jul 8 10:47:42 cztest kernel: Modules linked in: iptable_filter ip_tables binfmt_misc cdc_ether usbnet mii xt_multiport dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_p clmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif ipmi_devintf ipmi_si mei_me pcspkr iTCO_wdt mxm_wmi iTCO_vendor_support dcdbas mei sg sb_edac edac_core ipmi_msghandler shpchp lpc_ich wmi acpi_p ower_meter xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt fb_sys_fops igb ttm ptp drm ahci pps_core libahci dca i2c_algo_bit libat a megaraid_sas i2c_core fjes [last unloaded: ip_tables] Jul 8 10:47:42 cztest kernel: CPU: 10 PID: 0 Comm: swapper/10 Tainted: GW ------------ 3.10.0-514.16.1.el7.x86_64 #1 Jul 8 10:47:42 cztest kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016 Jul 8 10:47:42 cztest kernel: 0000000000000000 dd79fe633eacd853 ffff88103e743880 ffffffff81686ac3 Jul 8 10:47:42 cztest kernel: ffff88103e7438b8 ffffffff81085cb0 ffff8806d5c57800 ffff88010a4e6c80 Jul 8 10:47:42 cztest kernel: 0000000000000001 00000000f90e778c 00000000000000001 ffff88103e7438c8 Jul 8 10:47:42 cztest kernel: Call Trace: Jul 8 10:47:42 cztest kernel: <IRQ> [<ffffffff81686ac3>] dump_stack+0x19/0x1b Jul 8 10:47:42 cztest kernel: [<ffffffff81085cb0>] warn_slowpath_common+0x70/0xb0 Jul 8 10:47:42 cztest kernel: [<ffffffff81085dfa>] warn_slowpath_null+0x1a/0x20 Jul 8 10:47:42 cztest kernel: [<ffffffff815c3663>] tcp_mark_head_lost+0x113/0x290 Jul 8 10:47:42 cztest kernel: [<ffffffff815c3f47>] tcp_update_scoreboard+0x67/0x80 Jul 8 10:47:42 cztest kernel: [<ffffffff815c964d>] tcp_fastretrans_alert+0x6dd/0xb50 Jul 8 10:47:42 cztest kernel: [<ffffffff815ca49d>] tcp_ack+0x8dd/0x12e0 Jul 8 10:47:42 cztest kernel: [<ffffffff815cb3a8>] tcp_rcv_established+0x118/0x760 Jul 8 10:47:42 cztest kernel: [<ffffffff815d5f8a>] tcp_v4_do_rcv+0x10a/0x340 Jul 8 10:47:42 cztest kernel: [<ffffffff812a84c6>] ? security_sock_rcv_skb+0x16/0x20 Jul 8 10:47:42 cztest kernel: [<ffffffff815d76d9>] tcp_v4_rcv+0x799/0x9a0 Jul 8 10:47:42 cztest kernel: [<ffffffffa0140036>] ? iptable_filter_hook+0x36/0x80 [iptable_filter] Jul 8 10:47:42 cztest kernel: [<ffffffff815b1094>] ip_local_deliver_finish+0xb4/0x1f0 Jul 8 10:47:42 cztest kernel: [<ffffffff815b1379>] ip_local_deliver+0x59/0xd0 Jul 8 10:47:42 cztest kernel: [<ffffffff815b0fe0>] ? ip_rcv_finish+0x350/0x350 Jul 8 10:47:42 cztest kernel: [<ffffffff815b0d1a>] ip_rcv_finish+0x8a/0x350 Jul 8 10:47:42 cztest kernel: [<ffffffff815b16a6>] ip_rcv+0x2b6/0x410 Jul 8 10:47:42 cztest kernel: [<ffffffff815700d2>] __netif_receive_skb_core+0x582/0x800 Jul 8 10:47:42 cztest kernel: [<ffffffff815dc694>] ? tcp4_gro_receive+0x134/0x1b0 Jul 8 10:47:42 cztest kernel: [<ffffffff811dc861>] ? __slab_free+0x81/0x2f0 Jul 8 10:47:42 cztest kernel: [<ffffffff81570368>] __netif_receive_skb+0x18/0x60 Jul 8 10:47:42 cztest kernel: [<ffffffff815703f0>] netif_receive_skb_internal+0x40/0xc0 Jul 8 10:47:42 cztest kernel: [<ffffffff81571578>] napi_gro_receive+0xd8/0x130 Jul 8 10:47:42 cztest kernel: [<ffffffffa018b237>] igb_clean_rx_irq+0x387/0x700 [igb] Jul 8 10:47:42 cztest kernel: [<ffffffff8155e862>] ? skb_release_data+0xf2/0x140 Jul 8 10:47:42 cztest kernel: [<ffffffffa018b933>] igb_poll+0x383/0x770 [igb] Jul 8 10:47:42 cztest kernel: [<ffffffff815d3120>] ? tcp_write_timer_handler+0x200/0x200 Jul 8 10:47:42 cztest kernel: [<ffffffff81570c00>] net_rx_action+0x170/0x380 Jul 8 10:47:42 cztest kernel: [<ffffffff8108f63f>] __do_softirq+0xef/0x280 Jul 8 10:47:42 cztest kernel: [<ffffffff81698c1c>] call_softirq+0x1c/0x30 Jul 8 10:47:42 cztest kernel: [<ffffffff8102d365>] do_softirq+0x65/0xa0 Jul 8 10:47:42 cztest kernel: [<ffffffff8108f9d5>] irq_exit+0x115/0x120 Jul 8 10:47:42 cztest kernel: [<ffffffff816997b8>] do_IRQ+0x58/0xf0 Jul 8 10:47:42 cztest kernel: [<ffffffff8168e86d>] common_interrupt+0x6d/0x6d Jul 8 10:47:42 cztest kernel: <EOI> [<ffffffff81514a22>] ? cpuidle_enter_state+0x52/0xc0 Jul 8 10:47:42 cztest kernel: [<ffffffff81514b69>] cpuidle_idle_call+0xd9/0x210 Jul 8 10:47:42 cztest kernel: [<ffffffff810350ee>] arch_cpu_idle+0xe/0x30 Jul 8 10:47:42 cztest kernel: [<ffffffff810e82a5>] cpu_startup_entry+0x245/0x290 Jul 8 10:47:42 cztest kernel: [<ffffffff8104f07a>] start_secondary+0x1ba/0x230 Jul 8 10:47:42 cztest kernel: ---[ end trace 6bc65b0c591c1794 ]--- The host environment is as follows:
Processing instructions The stack printing process is similar to the xfs alarm processing. The general process is that after the kernel turns on the sack and fack functions, the fast retransmission and selective retransmission required during network transmission will be processed by the tcp_mark_head_lost function of the tcp_input.c file. It mainly marks the number of packets lost during transmission. As shown below, the kernel stack information reported by the system is triggered by the tcp_verify_left_out function call in the tcp_mark_head_lost function:
// source/include/net/tcp.h
#define tcp_verify_left_out(tp) WARN_ON(tcp_left_out(tp) > tp->packets_out)
static inline unsigned int tcp_left_out(const struct tcp_sock *tp)
{
return tp->sacked_out + tp->lost_out;
}
// source/include/asm-generic/bug.h
#define __WARN() warn_slowpath_null(__FILE__, __LINE__)
#ifndef WARN_ON
#define WARN_ON(condition) ({ \
__WARN(); \
})
#endif
// source/net/ipv4/tcp_input.c
/* Detect loss in event "A" above by marking head of queue up as lost.
* For FACK or non-SACK(Reno) senders, the first "packets" number of segments
* are considered lost. For RFC3517 SACK, a segment is considered lost if it
* has at least tp->reordering SACKed seqments above it; "packets" refers to
* the maximum SACKed segments to pass before reaching this limit.
*/
static void tcp_mark_head_lost(struct sock *sk, int packets, int mark_head)
{
struct tcp_sock *tp = tcp_sk(sk);
....
tcp_verify_left_out(tp); // trigger dump_stack
}
...
static void tcp_update_scoreboard(struct sock *sk, int fast_rexmit)
{
struct tcp_sock *tp = tcp_sk(sk);
if (tcp_is_reno(tp)) {
tcp_mark_head_lost(sk, 1, 1);
} else if (tcp_is_fack(tp)) {
int lost = tp->fackets_out - tp->reordering;
if (lost <= 0)
lost = 1;
tcp_mark_head_lost(sk, lost, 0);
} else {
int sacked_upto = tp->sacked_out - tp->reordering;
if (sacked_upto >= 0)
tcp_mark_head_lost(sk, sacked_upto, 0);
else if (fast_rexmit)
tcp_mark_head_lost(sk, 1, 1);
}
}According to the description in redhat-536483, this error message is usually caused by a TCP bug, which may be triggered when the kernel uses a released TCP socket buffer list:
Processing Upgrading the kernel As shown below, redhat may have fixed the use after free bug of tcp_* related functions in version 3.10.0-520. You can try to upgrade to solve this problem: * Thu Nov 03 2016 Rafael Aquini <[email protected]> [3.10.0-520.el7] - [net] tcp: fix use after free in tcp_xmit_retransmit_queue() (Mateusz Guzik) [1379531] {CVE-2016-6828} Disable fack/sack function From the documentation of Red Hat Knowledge Base, the tcp_mark_head_lost function is mainly used to mark the number of packets lost during fast retransmission and selective confirmation, so it may be possible to temporarily disable the fack/sack parameters to avoid this problem: sysctl -w net.ipv4.tcp_fack=0 sysctl -w net.ipv4.tcp_sack=0 You can try the second method first, and if the problem persists, consider upgrading the kernel version. refer to redhat-536483 bug-1367091 CVE-2016-6828 kernel-commit Summarize The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM. You may also be interested in:
|
<<: MySQL green decompression version installation and configuration steps
>>: Discuss the application of mixin in Vue
Recommend
An example of how Vue implements four-level navigation and verification code
Effect: First create five vue interfaces 1.home.v...
mysql create database, add users, user authorization practical method
1. Create a MySQL database 1. Create database syn...
Detailed explanation of Vue project optimization and packaging
Table of contents Preface 1. Routing lazy loading...
Explanation of the working principle and usage of redux
Table of contents 1. What is redux? 2. The princi...
Vue implements a scroll bar style
At first, I wanted to modify the browser scroll b...
Detailed explanation of the meaning of N and M in the MySQL data type DECIMAL(N,M)
A colleague asked me what N and M mean in the MyS...
Example code for using CSS to darken the font color of the selected area when scrolling
When encapsulating the date picker, you need to d...
Reasons why MySQL queries are slow
Table of contents 1. Where is the slowness? 2. Ha...
JavaScript implements a box that follows the mouse movement
This article shares the specific code of JavaScri...
Recommend a cool interactive website made by a front-end engineer
Website link: http://strml.net/ By Samuel Reed Ti...
A quick guide to MySQL indexes
The establishment of MySQL index is very importan...
Vue axios interceptor commonly used repeated request cancellation
introduction The previous article introduced the ...
HTML background image and background color_PowerNode Java Academy
1. HTML Image <img> 1. The <img> tag ...
Summary of 7 reasons why Docker is not suitable for deploying databases
Docker has been very popular in the past two year...
Centos builds chrony time synchronization server process diagram
My environment: 3 centos7.5 1804 master 192.168.1...