Table of Contents
- Introduction
- Investigating Segfaults
- Reviewing Core Dumps with gdb
- PHP Debug Symbols
- Changelog
Content Error or Suggest an Edit
Notice a grammatical error or technical inaccuracy? Let us know; we will give you credit!
Introduction
You’re getting 502 errors on your site, you have an Ubuntu server (this was a Ubuntu 22 server with GridPane) and you’re digging deeper and see that Nginx is getting a connection reset by peer from PHP-FPM in the sites error log.
024/10/07 08:52:28 [error] 753728#753728: *2097 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 0.0.0.0, server: domain.com, request: "GET /add-slimpod-to-homescreen/ HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php81-fpm-domain.com.sock:", host: "domain.com", referrer: "https://domain.com/help/"
You start digging more and more, and start looking in /var/log/syslog and find a segfault during the same time.
Oct 7 08:52:28 server1 kernel: [1658840.187828] php-fpm8.1[755922]: segfault at 7fe1822a7fe4 ip 00007fe18357fbc9 sp 00007ffe3002eb20 error 4 in mbstring.so[7fe18357d000+29000] Oct 7 08:52:28 server1 kernel: [1658840.187839] Code: 00 00 00 00 48 c7 05 c2 2f 0f 00 00 00 00 00 48 c7 05 c7 2f 0f 00 00 00 00 00 48 c7 05 c4 2f 0f 00 00 00 00 00 48 85 ff 74 13 <8b> 57 04 f6 c2 40 74 2f 48 c7 05 34 30 0f 00 00 00 00 00 31 c0 44
Investigating Segfaults
We know the process was segfaulting, but how do we figure out why? Well, we need to have the kernel create something called a core dump when a process segfaults.
Configure System to Create Core Dumps (ulimit -c)
First we need to confirm that the system will create core dumps. Run the following command to verify if core dumps can be created.
> ulimit -c 0
The returned value is 0, which means no core file can be created. To enable core dumps, you can run the following
ulimit -c unlimited
This won’t persist however, so you will need to update the /etc/security/limits.conf
* soft core unlimited root soft core unlimited
There is no need to restart your system, the above will apply to all users and the root user when a new shell is started.
Installing coredumpctl via systemd-coredump
You can use systemd to log core dumps, and then utilize the coredumpctl command to list the core dumps.
apt-get install systemd-coredump
Ubuntu has apport, but unfortunately it’s not enabled by default and has some automation with opening bug reports. I didn’t do a deep dive on it, and used systemd-coredump package with coredumpctl instead.
Reviewing Core Dumps with gdb
Now that we can store core-dumps you can use the gdb program to review a core dump.
gdb <executable_name> <core_dump_file>
If you installed coredumpctl, you can simply run coredumpctl and it will do all the work.
coredumpctl debug <PID> coredumpctl debug php-fpm8.1
PHP Debug Symbols
In order to further debug PHP, we need to have the appropriate PHP debug symbols installed.
Enabling Debug Ubuntu Debug Symbol Packages
If you’re using Ubuntu, then you can enable debug symbol packages
If you want to debug a crash – whether in a project you are developing yourself or from a third-party package – or if you frequently need the debug symbols for specific libraries, it might be helpful to install them permanently on your system if you can’t use debuginfod.
This document describes how to set up the debugging symbol packages (
*-dbg.deband*-dbgsym.ddeb). You might need to do this when you are performing tasks like a Backtrace or using Valgrind.
Debuginfod Service Ubuntu 22 or Later (Preferred Method)
If you’re running Ubuntu 22 or later, you can use the Debuginfod service provided by Ubuntu. I suggest using this method.
If you are on Ubuntu Jammy (22.04) or later, you don’t need to worry about installing debug symbol packages since the Ubuntu project maintains a Debuginfod server. GNU Debugger (GDB) and other debuginfo-consumer applications support Debuginfod (mostly) out of the box. For more information about it, please refer to our Debuginfod guide.
Here’s the commands to run.
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com" set debuginfod enabled on
Installing Ubuntu dbgsym.ddeb Packages
You can read the full guide at the following link or follow the steps below.
1 – Import the Signing Key
sudo apt install ubuntu-dbgsym-keyring
2 – Create a ddebs.list file
Create an /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal:
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \ sudo tee -a /etc/apt/sources.list.d/ddebs.list
3 – Update package list
Run the following to update your package list or click the Reload button if you used the Synaptic Package Manager:
sudo apt-get update
4 – Install PHP debug symbol Packages
Now install all the PHP debug symbol packages for the PHP packages you have installed.
apt-get install php8.1-dbgsym php8.1-common-dbgsym php8.1-fpm-dbgsym
Installing Ubuntu Ondrej PHP dbgsym.ddeb Packages (Required for GridPane)
If you’re using Ondrej PHP PPA then you need to add a new repo line to /etc/apt/sources.list.d/ondrej-ubuntu-php-jammy.list
deb https://ppa.launchpadcontent.net/ondrej/php/ubuntu/ jammy main deb https://ppa.launchpadcontent.net/ondrej/php/ubuntu/ jammy main/debug
Changelog
- 12-21-2024 – Added more instructions for DebuginfoD
Jordan Trask
With 20 years of experience in technology spanning Linux, Data Center Infrastructure, and related services and applications, Jordan consults with small, medium, and large organizations tackling technology problems and needs. With 10 years of experience in WordPress, he’s created Managing WP as a means of dumping his brain on the web.
Jordan operates LMT Solutions providing Technology Consulting Services and We Power WP for WordPress Care Plans and Support.