Dealing with PHP-FPM Segfaults in Unbuntu

Introduction

You’re getting 502 errors on your site, you have an Ubuntu server (this was a Ubuntu 22 server with GridPane) and you’re digging deeper and see that Nginx is getting a connection reset by peer from PHP-FPM in the sites error log.

024/10/07 08:52:28 [error] 753728#753728: *2097 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 0.0.0.0, server: domain.com, request: "GET /add-slimpod-to-homescreen/ HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php81-fpm-domain.com.sock:", host: "domain.com", referrer: "https://domain.com/help/"

You start digging more and more, and start looking in /var/log/syslog and find a segfault during the same time.

Oct 7 08:52:28 server1 kernel: [1658840.187828] php-fpm8.1[755922]: segfault at 7fe1822a7fe4 ip 00007fe18357fbc9 sp 00007ffe3002eb20 error 4 in mbstring.so[7fe18357d000+29000]
Oct 7 08:52:28 server1 kernel: [1658840.187839] Code: 00 00 00 00 48 c7 05 c2 2f 0f 00 00 00 00 00 48 c7 05 c7 2f 0f 00 00 00 00 00 48 c7 05 c4 2f 0f 00 00 00 00 00 48 85 ff 74 13 <8b> 57 04 f6 c2 40 74 2f 48 c7 05 34 30 0f 00 00 00 00 00 31 c0 44

Investigating Segfaults

We know the process was segfaulting, but how do we figure out why? Well, we need to have the kernel create something called a core dump when a process segfaults.

Check ulimit -c

First we need to confirm that the system will create core dumps. Run the following command to verify if core dumps can be created.

> ulimit -c
0

The returned value is 0, which means no core file can be created. To enable core dumps, you can run the following

ulimit -c unlimited

This won’t persist however, so you will need to update the /etc/security/limits.conf

* soft core unlimited
root soft core unlimited

There is no need to restart your system, the above will apply to all users and the root user when a new shell is started.

Installing coredumpctl via systemd-coredump

You can use systemd to log core dumps, and then utilize the coredumpctl command to list the core dumps.

apt-get install systemd-coredump

Ubuntu has apport, but unfortunately it’s not enabled by default and has some automation with opening bug reports. I didn’t do a deep dive on it, and used systemd-coredump package with coredumpctl instead.

Reviewing Core Dumps with gdb

You can use the gdb program to review a core dump.

gdb <executable_name> <core_dump_file>

Since we installed coredumpctl, you can simply run coredumpctl and it will do all the work.

coredumpctl debug <PID>
coredumpctl debug php-fpm8.1