Table of Contents
- Introduction
- What happened?
- Why was my server hacked? Why didn’t CyberPanel announce anything?
- Don’t Clean up your Infected CyberPanel Server, Restore from Backups
- Blocking CyberPanel RCE Attack
- Cleaning up Infected CyberPanel Servers
- Restoring SSH Access
- Malware Clean-up
- Kinsing Malware Clean-up
- Additional Steps to Clean up Infection
- Change Log
Content Error or Suggest an Edit
Notice a grammatical error or technical inaccuracy? Let us know; we will give you credit!
Introduction
On October 28th 2024 Cyberpanel posted the following on their Facebook page.
Hello everyone!
We’ve recently made some important security updates which you can read in our blog.
It is highly recommended to upgrade CyberPanel as soon as possible ->
Unfortunately there was no blog post. Until today (October 30th 2024) which goes on to explain the issue, it’s light and discusses how to resolve it the issue.
What happened?
There is currently a root RCE (Remote Code Execution) bug within CyberPanel. It was discovered and disclosed to CyberPanel by DreyAnd or @dreyand_ (Twitter).
You can read the full disclosure on their blog.
Few months ago I was assigned to do a pentest on a target running CyberPanel. It seemed to be installed by default by some VPS providers & it was also sponsored by Freshworks.
I was clueless on how to pwn the target as the functionalities were very limited, so I thought about it differently, let’s just find a 0day ¯_(ツ)_/¯ .
This lead to a 0-click pre-auth root RCE on the latest version (2.3.6 as of now). It’s currently still “unpatched”, as in, the maintainers have been notified, a patch has been done but still waiting for the CVE & for the fix to make the make it to he main release. You can find the patch commit at https://github.com/usmannasir/cyberpanel/commit/5b08cd6d53f4dbc2107ad9f555122ce8b0996515 .
Why was my server hacked? Why didn’t CyberPanel announce anything?
CyberPanel was informed of the vulnerability and the PoC (proof of concept) which was released by the researcher on October 27th 2024 after confirmation from the CyberPanel team.
They even made a commit to patch the RCE on October 23rd 2024, but unfortunately never created a release.
Confirmation on CyberPanel Updates and Releases
As mentioned above, CyberPanel has yet to make an official release for this security issue. Usman Nasir has posted the following on the Facebook CyberPanel group.
I also want to address some misinformation circulating. A recent blog post claimed that we committed and released a patch but did not make it an official release. This is incorrect. CyberPanel is installed directly from GitHub, so every commit to the branch is automatically included in the latest upgrades and installs, which has been in effect for over a week now.
CyberPanel Website Changelog is from September 2023
If you visit the cyberpanel.net site and view the changelog posted, the last update was September 19th 2024.
CyberPanel Update Method
Unfortunately, CyberPanel doesn’t follow the typical development cycle that you might find with other open-source projects. As you can see from the screenshot below, the last release of CyberPanel was v1.9.1 on October 15th 2019.
However, this is misleading because CyberPanel updates based on branches. So to see the latest version of CyberPanel, you must view all branches. This is where you can see that v2.3.8 seems to be the latest. However, there is no changelog, unfortunately, for this release.
Searching for Vulnerable CyberPanel Instances
How did the attack happen so quickly? Well, it’s pretty easy to figure out CyberPanel instances exposed to the internet via TCP port 8090 using fofa.info
FOFA is a search engine for mapping the cyberspace, aimed at helping users search for internet assets on the public network. Simply put, FOFA operates like Google or Baidu, where users can input keywords to match data containing those keywords. However, unlike Google or Baidu, the data searched by FOFA includes assets such as cameras, printers, databases, and operating systems.
Cyberspace mapping can be considered a “map” of the cyberspace. Similar to how Google Maps map out terrains through satellite images, cyberspace mapping also detects global network assets through technological probes.
However, the cyberspace is divided into public and private networks. FOFA mainly detects assets on the public network, similar to the buildings (IP addresses) on Google Maps. Although the exterior structure and scale of the building can be seen, the interior cannot be understood. This is because violating the privacy of the private network is illegal.
Here is the direct link to the search
https://en.fofa.info/result?qbase64=YXBwPSJDeWJlclBhbmVsIg%3D%3D
Here’s the search query
app="CyberPanel"
Number of Vulnerable Servers going down
From this Twitter post, the number of vulnerable servers has gone down. It would seem someone is using the RCE to patch affected hosts.
Ransomware Malware Utilized During Attack
Some CyberPanel servers have been hit with ransomware malware, there’s actually multiple different types. And depending you could have been infected by the first, second and third and sometimes there is no key around to decrypt.
Here are the twitter posts.
It’s so bad, files were encrypted 2 times, 1 time for each group.
TLDR:
R1 encrypted files to .locked
R2 encrypted files to .encryp ( even the .encryp ones done by R1)
R3 (PSAUX) encrypted files (badly) and crashed everything
Blog Posts and News Articles
- What Are My OPTIONS? CyberPanel v2.3.6 pre-auth RCE – dreyand.rs
- Multiple Vulnerabilities in CyberPanel – iamaldi.github.io
- Massive PSAUX ransomware attack targets 22,000 CyberPanel instances
Don’t Clean up your Infected CyberPanel Server, Restore from Backups
It’s not advised to clean up your server. Instead you should restore from backups and utilize a different control panel at this time. If you have have sites with Stripe keys etc, you should re-generate those keys.
Blocking CyberPanel RCE Attack
Upgrade CyberPanel to the Latest Version, you can do so by visiting CyberPanel’s Upgrade guide.
Here is the one-liner from the above doc.
sh <(curl https://raw.githubusercontent.com/usmannasir/cyberpanel/stable/preUpgrade.sh || wget -O - https://raw.githubusercontent.com/usmannasir/cyberpanel/stable/preUpgrade.sh)
Upgrade CyberPanel code database/views.py One Liner
You can run the following script and it will update the affected file database/views.py with the latest Github version. Use this if you’re unable to upgrade CyberPanel
if ! grep -B5 'currentACL = ACLManager.loadedACL(userID)' /usr/local/CyberCP/databases/views.py | grep -q upgrademysqlstatus; then echo "Downloading and replacing views.py..." wget -q https://raw.githubusercontent.com/usmannasir/cyberpanel/refs/heads/stable/databases/views.py -O /usr/local/CyberCP/databases/views.py else echo "views.py is already up-to-date." fi
Cleaning up Infected CyberPanel Servers
If you’re still looking to clean-up your server. Then I’ll provide some details below.
Kinsing Malware
It looks as though some of the infections are automated, and uses the kinsing Malware.
CyberPanel Community Forum Infection Clean-up Post
The following CyberPanel Community Forum Post goes into details about cleaning up your system.
Restoring SSH Access
Proxmox Guest
If you’re using cloud-init, you can enable the option to set a password. On Ubuntu systems this will set the ubuntu user password and then you can run sudo su -
Malware Clean-up
There seems to be a couple of variants of malware that you could be potentially be infected with.
- kinsing
Ransomware
Another piece of malware actually seems to be ransomware looks to encrypt all files and databases and provides a README on how to send money to decrypt the database.
Help with Malware Clean-up
If you’re looking to have your server cleaned-up, I do need some infected servers to test this script on. Visit our contact page to get in touch. Best method is through Discord.
Kinsing Malware Clean-up
The following section will go through the Kinsing Malware Clean-up
Kinsing Malware Clean-up Shell Script
The following is a shell script that can help to automate the clean-up. It’s a work in progress, so make sure to create an issue on the Github.
Kinsing Malware Manual Clean-up Steps
I’ve created the following guide to help clean-up your CyberPanel server after it’s been infected with the Kinsing malware. I’ve take the information from the CyberPanel Community Forum post and add more details.
Step 1: Disable Cron
The malware will re-infect the server using the system cron, so for now we will stop cron.
systemctl stop cron
Step 2: Delete Malware Files
Remove the following files to handicap the malware.
rm -f /etc/data/kinsing rm -f /etc/kinsing rm -f /tmp/kdevtmpfsi rm -rf /usr/lib/secure rm -f /usr/lib/secure/udiskssd rm -f /usr/bin/network-setup.sh rm -f /usr/.sshd-network-service.sh rm -rf /usr/.network-setup rm -f /usr/.network-setup/config.json rm -f /usr/.network-setup/xmrig-*tar.gz rm -f /usr/.network-watchdog.sh' rm -f /tmp/kdevtmpfsi rm -f /etc/data/libsystem.so rm -f /etc/data/kinsing rm -f /dev/shm/kdevtmpfsi
If any of the files fail to delete, you may want to remove immutable flag from the file and directory.
chattr -i secure/udiskssd chattr -i secure
Step 3: Remove Suspicious Service
systemctl stop bot.service systemctl disable bot.service rm /lib/systemd/system/bot.service systemctl daemon-reload systemctl stop systemd_s.service systemctl disable systemd_s.service rm /etc/systemd/system/systemd_s.service systemctl stop sshd-network-service.service systemctl disable sshd-network-service.service rm /etc/systemd/system/sshd-network-service.service systemctl stop network-monitor.service systemctl disable network-monitor.service rm /etc/systemd/system/network-monitor.service
Step 4: Kill Suspicious Processes
ps -aux | grep -E 'kinsing|udiskssd|kdevtmpfsi|bash2|.network-setup|syshd|atdb' | awk {' print $2 '} | xargs kill -9
Step 5: Unload pre-loaded libraries (Delete /etc/ld.so.preload)
- You might file the file /etc/ld.so.preload exists with the following:
/etc/data/libsystem.so
- Delete this file
rm /etc/ld.so.preload
- Locate all processes that loaded libsystem.so and kill them.
lsof | grep libsystem.so | awk {' print $2 '} | xargs kill -9
Step 6: Delete Suspicious Cron Jobs
Malware often adds tasks to crontab to automatically restart itself. To remove suspicious crontab entries:
- Open the root
crontab:
sudo crontab -e
- Delete any unknown or suspicious lines.
* Crons for user root: in /var/spool/cron/crontabs/root: */3 * * * * /usr/lib/secure/atdb
If you get an error deleting the crontab, then run the following
chattr -ia /var/spool/cron/crontabs/root chattr -ia /var/spool/cron/root
Additional Steps to Clean up Infection
You should also consider completing the following steps to ensure that you’ve cleared out the infection entirely.
1 – Install chkrootkit and rkhunter
You can do this after you’ve stop the infection from spawning over and over and to furter cleanup the system.
apt-get install chkrootkit rkhunter
Then do a scan
rkhunter --check
Then review the log for “Warn”.
2 – Using find -mtime
You can use find to locate files that have been modified within the last 2 days.
find . -mtime 2
You can change the 2 to 1 for files modified in the last day before or 0 for last 24 hours.
Change Log
- 10-30-2024 – Added Ransomware Malware section
- 10-30-2024 – Added section on number of vulnerable servers decrease.
- 10-29-2024 – Added section for Kinsing shell script.
- 10-29-2024 – Add Proxmox section for Restoring SSH Access via cloud-init
Jordan Trask
With 20 years of experience in technology spanning Linux, Data Center Infrastructure, and related services and applications, Jordan consults with small, medium, and large organizations tackling technology problems and needs. With 10 years of experience in WordPress, he’s created Managing WP as a means of dumping his brain on the web.
Jordan operates LMT Solutions providing Technology Consulting Services and We Power WP for WordPress Care Plans and Support.