Category: cPanel

  • How a PostgreSQL Vulnerability Led to a Crypto Mining Malware Infection: An Incident Analysis

    How a PostgreSQL Vulnerability Led to a Crypto Mining Malware Infection: An Incident Analysis

    Date of Incident: May 5, 2025
    Affected System: AlmaLinux 9.5 with cPanel & PostgreSQL

    Incident Summary

    A Linux server running cPanel with PostgreSQL was compromised through a misconfigured PostgreSQL service, which allowed an attacker to upload and execute a malicious binary called cpu_hu. This ELF executable is part of a known crypto mining malware campaign, which abuses PostgreSQL’s permissions to spawn unauthorized processes.


    Indicators of Compromise (IOCs)

    • Suspicious binary: /var/lib/pgsql/data/base/13494/cpu_hu
    • Crontab entries for postgres user triggering binary execution
    • Audit logs showing executions of /usr/bin/s-nail, /usr/sbin/sendmail, and /usr/sbin/exim with UID 26 (PostgreSQL user)
    • Kernel logs in /var/log/messages showing:Killing process <PID> (cpu_hu) with signal SIGKILL
    • PostgreSQL failing to restart due to improper permissions after remediation

    Attack Vector & Execution

    The attacker exploited a misconfigured PostgreSQL installation with either:

    • trust authentication enabled
    • publicly accessible port 5432
    • the ability to execute arbitrary shell commands via SQL extensions like COPY TO PROGRAM

    Once inside, the attacker used the postgres user to:

    1. Upload the ELF binary
    2. Schedule its execution via cron

    Containment & Mitigation Steps

    Step 1: Kill Malware Processes

    pkill -f cpu_hu

    Step 2: Remove Binary

    find / -type f -name '*cpu_hu*' -delete

    Step 3: Clean Up Crontab

    crontab -u postgres -r

    Step 4: Lock Down PostgreSQL

    Edit postgresql.conf and add:

    session_preload_libraries = ''

    Restart service.

    -- Inside psql:
    ALTER USER postgres PASSWORD 'new-strong-password';
    REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM PUBLIC;
    REVOKE EXECUTE ON FUNCTION pg_read_file(text) FROM PUBLIC;
    REVOKE EXECUTE ON FUNCTION pg_stat_file(text) FROM PUBLIC;

    Step 5: Set Correct Permissions

    chown -R postgres:postgres /var/lib/pgsql/data
    chmod 700 /var/lib/pgsql/data
    systemctl restart postgresql

    Step 6: Block Public Access

    iptables -A INPUT -p tcp --dport 5432 -s <trusted_ip> -j ACCEPT
    iptables -A INPUT -p tcp --dport 5432 -j DROP

    Lessons Learned

    • Always restrict PostgreSQL to localhost or VPN access
    • Disable dangerous features like COPY TO PROGRAM unless absolutely required
    • Use auditd rules to track mail/sendmail invocations by non-root users:
    audictl -a always,exit -F arch=b64 -S execve -F uid=26 -F path=/usr/bin/s-nail -k mail_postgres_exec
    • Regularly inspect crontabs and non-root user binaries in /var/lib/ and /tmp

    References


    Status: Resolved

  • Fix cPanel ownership and permissions [Script]

    Fix cPanel ownership and permissions [Script]

    When a cPanel server experiences file permission issues-after a migration, manual file operations, or a misbehaving script-websites may become inaccessible, emails may fail, or security might be at risk. This script automates the process of fixing file ownership and permissions for one or more cPanel users, ensuring everything is back to a secure and functional state.

    Use Case

    You may need to run this script when:

    • Website files show 403 Forbidden errors
    • Email delivery fails due to etc/ permissions
    • Files were copied or restored without --preserve flags
    • CageFS directories have incorrect modes

    How to run the script

    for i in `ls -A /var/cpanel/users` ; do ./fixperms $i ; done

    The Script (save as ./fixperms and chmod +x fixperms)

    #!/bin/bash
    # Script to fix permissions and ownerships for one or more cPanel users
    
    if [ "$#" -lt "1" ]; then
      echo "Must specify at least one user"
      exit 1
    fi
    
    USERS=$@
    
    for user in $USERS; do
      HOMEDIR=$(getent passwd "$user" | cut -d: -f6)
    
      if [ ! -f /var/cpanel/users/"$user" ]; then
        echo "User file missing for $user, skipping"
        continue
      elif [ -z "$HOMEDIR" ]; then
        echo "Could not determine home directory for $user, skipping"
        continue
      fi
    
      echo "Fixing ownership and permissions for $user"
    
      # Ownership
      chown -R "$user:$user" "$HOMEDIR" >/dev/null 2>&1
      chmod 711 "$HOMEDIR" >/dev/null 2>&1
      chown "$user:nobody" "$HOMEDIR/public_html" "$HOMEDIR/.htpasswds" 2>/dev/null
      chown "$user:mail" "$HOMEDIR/etc" "$HOMEDIR/etc/"*/shadow "$HOMEDIR/etc/"*/passwd 2>/dev/null
    
      # File permissions (parallel)
      find "$HOMEDIR" -type f -print0 2>/dev/null | xargs -0 -P4 chmod 644 2>/dev/null
      find "$HOMEDIR" -type d ! -name cgi-bin -print0 2>/dev/null | xargs -0 -P4 chmod 755 2>/dev/null
      find "$HOMEDIR" -type d -name cgi-bin -print0 2>/dev/null | xargs -0 -P4 chmod 755 2>/dev/null
    
      chmod 750 "$HOMEDIR/public_html" 2>/dev/null
    
      # CageFS fixes
      if [ -d "$HOMEDIR/.cagefs" ]; then
        chmod 775 "$HOMEDIR/.cagefs" 2>/dev/null
        chmod 700 "$HOMEDIR/.cagefs/tmp" "$HOMEDIR/.cagefs/var" 2>/dev/null
        chmod 777 "$HOMEDIR/.cagefs/cache" "$HOMEDIR/.cagefs/run" 2>/dev/null
      fi
    
    done

    This is a improved script from: https://www.casbay.com/guide/kb/script-to-fix-cpanel-account-permissions-2

  • Recovering MySQL Databases on a Crashed cPanel Server Without Backups

    Recovering MySQL Databases on a Crashed cPanel Server Without Backups

    When a cPanel server experiences catastrophic failure without any valid backups, restoring websites and databases manually becomes the only option. In my case, the server had completely failed and could only be accessed via a rescue environment. No backups were available in /backup, and the system was non-bootable due to critical library corruption.

    To recover, I mounted the failed system, manually transferred essential directories such as /var/lib/mysql and /home to a freshly installed cPanel server using rsync, and fixed ownership and permissions. This restored websites and database files physically, but cPanel/WHM did not recognize the MySQL databases or users.

    Problem: cPanel Doesn’t Recognize Existing MySQL Databases

    Although the database folders were correctly placed in /var/lib/mysql/ and all MySQL users were present in the mysql.user table, cPanel GUI showed no databases or users associated with any account.

    This is expected behavior — cPanel stores mappings between accounts, databases, and MySQL users in its own internal metadata files, which were not recoverable.

    Solution: Rebuild cPanel MySQL Mapping Using dbmaptool

    To restore MySQL database and user associations for each cPanel account without recreating them manually, I used the official cPanel utility:

    /usr/local/cpanel/bin/dbmaptool

    I created a script that:

    • Loops through all cPanel users (found in /var/cpanel/users)
    • For each user, finds all MySQL databases starting with the user’s prefix (e.g. user_db1)
    • Finds all MySQL users belonging to that prefix (e.g. user_dbuser1)
    • Automatically maps them using dbmaptool
    #!/bin/bash
    
    for user in $(ls /var/cpanel/users); do
      dbs=$(mysql -N -e "SHOW DATABASES LIKE '${user}\_%';" | tr '\n' ',' | sed 's/,\$//')
      dbusers=$(mysql -N -e "SELECT User FROM mysql.user WHERE User LIKE '${user}\_%';" | tr '\n' ',' | sed 's/,\$//')
    
      if [[ -n "$dbs" || -n "$dbusers" ]]; then
        echo "Mapping for user: $user"
        /usr/local/cpanel/bin/dbmaptool "$user" --type 'mysql' --dbs "$dbs" --dbusers "$dbusers"
      fi
    done

    Final Cache Refresh

    After running the script, I executed:

    /scripts/update_db_cache
    /scripts/updateuserdatacache

    This forced cPanel to reload and re-index the updated metadata, and all previously invisible databases and MySQL users reappeared in the cPanel UI for each respective account.

    Even in total system failure scenarios with no backups, if the /home and /var/lib/mysql directories are intact and MySQL users are present, it’s entirely possible to recover a cPanel environment manually. The key is to re-establish metadata associations using dbmaptool, which tells cPanel which databases and users belong to which accounts.