Follow
Also, my sort job that has been running all week finally finished 😅 it turns out indexing 40TB of web archive data can take some time!
@eb yeah, me too to be honest :-) It is definitely the bottleneck since I had to use it for reading, writing, and also for sorting (TMPDIR). It's an NFS share that is provisioned by campus IT, and I have no idea what is on the other side. I probably should know...
@edsu automated /commercial solution or rolled your own? indexing file contents?
@edsu I'm curious about your storage solution 👀