News

  1. Home
  2. Operating SystemsLinux
Linux, Unix by Team Ajara

Slowness due to df k defunct issue on apps server

df -k defunct issue on apps  server causing connectivity problems

Over view of the issue

Issue on Application Server where “Apps is experiencing clocking” and it  happened because of defunct df –k processes which cosumed High CPU Usage.

Did not find any issue in the Database level.

Troubleshooting / Verification:

  • From DBA end verified the remote connectivity , top processes and memory usage ,found no CPU and memory and connectivity issues on DB Server.
  • Application and platform team(Unix) Team tried different options to resolve the Issue
  • They have found the issue related with “df -k” defunct processes that present more than 100 processes which consumed the memory on servers and caused service interruption. Few lines from Processes output shown below.
$ ps -ef | grep df
root 3145742 4588268 15 Nov 05 – 452:18 df -k
root 5636188 5439720 15 Nov 05 – 452:39 df -k
root 5701812 7143656 16 Nov 05 – 451:09 df -k
7012384 /usr/sbin/cron
39780422 /bin/ksh /u01/scripts/chkvolume.sh
40370390 awk { print $4 }
45678756 df -k

# ps -T 7012384
PID TTY TIME CMD
7012384 – 0:40 cron
8323116 – 0:00 |\–ksh
14876712 – 0:00 | \–awk

Actions Taken:

  • All Defunct processes had been  killed  but still the issue was persist
  • As none of the work arounds didn’t work,  server had been  rebooted  and disabled crontab script “/u01/scripts/chkvolume.sh” on apps servers.
  • After server Reboot, the services started working fine,  sanity checks performed and users confirmed the system was working as expected.

Reason for the Issue

There was storage (Netapp) Migration occurred on the Apps servers ,  something missed in Execution part in the scripts.

Action Items to prevent this Issue in Future

Need to review scripts whenever there is a major change and need to verify after the change is complete.

Conclusion

Normally we receive these kind of issues after any major change in the environment (by any Team) .Always need to verify/compare the
pre change and implementation and post implementation stages carefully. It is always best to take Pre image of  disk usage (df -k) , services on Apps server and DB servers and AWR reports etc.

 

See also