0

So here are mt symptoms. Over the last few days I start the day with say 33 GB of space and then come in the next day to see that I have only 1GB of space left. I am assessing space by using "df"

hari@hari-Precision-WorkStation-T7500:~$ df
Filesystem                                                              1K-blocks       Used  Available Use% Mounted on
/dev/sde1                                                               230398116  216818952    1852572 100% /
none                                                                            4          0          4   0% /sys/fs/cgroup
udev                                                                      6135136          4    6135132   1% /dev
tmpfs                                                                     1229864       1588    1228276   1% /run
none                                                                         5120          0       5120   0% /run/lock
none                                                                      6149304       5808    6143496   1% /run/shm
none                                                                       102400         56     102344   1% /run/user
//server01.mycompany.com/archive$/Biology/RAW-Data 8458860528 4685045352 3773815176  56% /home/rawdata2
/dev/sda1                                                              1953512032 1724815580  228696452  89% /media/hari/FreeAgent GoFlex Drive1

The only change I did to my system was mount a 9 TB fileshare using the cifs (samba) module before this problem started.

When I run the disk usage analyser "baobab", it reports on the disk without suggesting that any single directory under my home has grown overnight significantly - i.e the percentages don't go up significantly.

I am running a private webserver on this machine and not running much else on this machine. There are no logs that are growing significantly overnight nor are there any processes that have much disk io as investigated by "htop" "top".

I tried to use "sudo iotop" but I dont know how to interpret its output.

Where is my disk space going. I am scared it is some indexer that is trying to index the 9TB fileshare that is holding on to some space since the problem started after I added the 9TB fileshare.

I am running Ubuntu 13.10 , 3.11.0-15-generic

How do I troubleshoot whats using all this space. I am scared its some indexer or some other background process.

harijay
  • 217
  • 1
  • 5
  • 21

2 Answers2

3

If the disk space is always freed after reboot that most likely means that the space is used up by some unlinked files. Unlinked files are deleted = unlinked from the file system, but a process still keeps them open, so the disk space used up by such a file is not freed until the process releases the file. The process holding the unlinked file may also keep writing to the file and using up more and more disk space undetectable in the file system.

To view unlinked files and the process which holds them you can use lsof. Until a long time I always used some grep-ing on the lsof output to view the unlinked files, but a while ago I found this super answer on how to view unlinked files easily. You just have to use

lsof +L1

as this will list all open files which have link count less than 1, hence they are unlinked.

So use lsof +L1 and check if there are a lot of files or big files unlinked on your machine, you will also see which process holds them.

Community
  • 1
falconer
  • 15,334
1

If you are in a situation where reboot isn't always freeing the spacing, I would reboot into rescue mode and use du as described above to get a feel for where the space is going. I tend to do "du -sc * | sort -n" as a quick-and-dirty way of doing this sort of investigation (dirty due to issues with hidden files, etc. as discussed above.) Start in the root and work down.

I would agree this is a very odd problem. Perhaps try selectively disabling services to see if you can get it to stop? As far as indexing goes, the only things running system-wide as standard that I can think of should be locate/mlocate. There's also apt-xapian which indexes packages, mandb which indexes man pages.

Are you logging in to this machine with a desktop session? If so, does the problem stop if you do log in (at least graphically)?

Steve Dee
  • 837