首頁>技術>

文末有彩蛋,這麼多年從來沒這樣處理過問題

背景

作業系統是 centos 6,RabbitMQ 3.8.1.

伺服器突然無法連線,重啟後檢視/var/log/messages,報錯日誌如下:

epmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memoryepmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memoryepmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memoryepmd: erlang mapper daemon cant fork 2'nd time Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memorysshd[1121]: error: fork: Cannot allocate memory

使用locate erl_crash.dump找到崩潰日誌 erl_crash.dump,使用 https://github.com/ferd/recon/edit/master/script/erl_crashdump_analyzer.sh 分析結果如下:

analyzing erl_crash.dump, generated on:  Wed Apr 22 16:39:10 2020 Slogan: init terminating in do_boot ({error,{cannot_read_enabled_plugins_file,/etc/rabbitmq/enabled_plugins,eacces}})Memory:===  processes: 17 Mb  processes_used: 17 Mb  system: 16 Mb  atom: 0 Mb  atom_used: 0 Mb  binary: 0 Mb  code: 6 Mb  ets: 1 Mb  ---  total: 34 MbDifferent message queue lengths (5 largest different):===     89 0Error logger queue length:===0File descriptors open:===  UDP:  0  TCP:  2  Files:  0  ---  Total:  2Number of processes:===89Processes Heap+Stack memory sizes (words) used in the VM (5 largest different):===      1 46422      1 10958      2 6772      2 4185      2 2586Processes OldHeap memory sizes (words) used in the VM (5 largest different):===      1 318187      1 75113      1 46422      1 10958      2 2586Process States when crashing (sum): ===      1 CONNECTED      2 CONNECTED|BINARY_IO      2 CONNECTED|BINARY_IO|PORT_LOCK      1 Current Process Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING | ACTIVE_SYS      1 Current Process Running      1 Internal ACT_PRIO_HIGH | USR_PRIO_HIGH | PRQ_PRIO_HIGH | OFF_HEAP_MSGQ      3 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_MAX      1 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_MAX | OFF_HEAP_MSGQ      1 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_NORMAL     76 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL      1 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING | ACTIVE_SYS      6 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | OFF_HEAP_MSGQ      1 Running     88 Waiting

沒排查出問題來,這臺機器是8c32g的機器,按道理應該不該出現記憶體不足的問題。

彩蛋在此

後面開發說RabbitMQ不用了,服務停掉。。。問題消失了。

29
最新評論
  • BSA-TRITC(10mg/ml) TRITC-BSA 牛血清白蛋白改性標記羅丹明
  • 在酷家樂做面試官的日子