💀 The Day 16TB Vanished — And Everyone Looked at Me

☕ It Started Like Any Normal Day

Coffee. Tickets. The usual background noise of enterprise support work at Wipro for Dell EMC. You know the drill — log in, check queue, sip tea (or coffee if you're still pretending to be productive).

Then my colleague picked up a case from one of India's top banks.

I remember the energy shift in the room. That specific kind of silence when someone says something that shouldn't be possible.

Because here's the thing about Dell EMC NetWorker — it doesn't randomly wake up one morning, look at 16TB of banking data, and think "you know what, today I delete." That's just not how it works. But try explaining that to a bank that's already halfway through drafting escalation emails.

🏦 The Setup (aka "This Should Never Fail")

Before we dive into the chaos, let me explain what was supposed to happen. The bank had a carefully designed backup workflow for one of their critical applications — one that couldn't be taken offline (no cold backups allowed). Whenever the application team confirmed low usage, backup admins would trigger this OnDemand workflow.

Basically, the kind of setup that auditors love and engineers quietly fear touching. And this workflow had been running perfectly for months. Which is exactly why this incident made zero sense.

🚨 The Weird Part Nobody Could Explain

During what should have been a routine backup trigger, the admin noticed something deeply wrong:

The admin panicked and stopped the job immediately. Smart move.

Let me explain why "relabelled" is such a big deal, because this is where the tape library logic matters:

So either NetWorker had a serious bug (nightmare scenario for everyone), or something had caused the saveset to expire prematurely. My colleague did the initial analysis and found… nothing. No trace. No clear reason. Clean logs saying backup completed successfully.

The case was already escalating. Customer was ready to go nuclear. And the data situation looked like this:

😤 I Took Over the Case

Not because I had answers. I absolutely did not. But because nobody else did either, and someone had to own it.

I set two goals, in priority order:

Here's where most engineers would look at the incomplete saveset and give up. An incomplete backup in standard recovery scenarios is like a key with half its teeth missing — it technically exists, but it won't open anything useful.

Most tools won't touch it. Most procedures don't cover it.

But then I remembered something I'd mostly ignored for years:

💡 UASM — The Tool I Underestimated

The plan I came up with was unusual but logical:

We executed. Tape retrieved, loaded, scanned. UASM command fired from the storage node CLI. And then… we waited.

No dramatic "problem solved" moment. Just steady, slow progress. Chunk by chunk. File by file.

That should have been the end of it. Close the case, write the RCA as "unknown cause," and move on. But I've never been comfortable with "unknown."

🔍 Going Back in Time (No, Really)

I reached out to the L3 SME team. Their suggestion:

My first reaction was what? Go to the past? Are you writing a script for Doctor Strange?

But they weren't being philosophical. They were talking about Bootstrap Backups.

The idea was brilliant once it clicked: if a saveset was deleted from the current media database, we can't see it now. But if we restore bootstrap backups from before the deletion, we can see it in the historical database. Then we replay day by day until it disappears — and that pinpoints exactly when and why it expired.

We pulled a full month of bootstrap backups, restored them in an isolated folder, compressed them, and brought them to a lab environment. Carefully. Methodically. Day by day.

And we saw it. Clear as day. The savesets were valid — then suddenly expired — days before the tape incident.

Something changed the retention.

🎭 The Audit Log Confession

We got on a session with the customer and pulled the audit logs.

And there it was.

No bug. No corruption. No mystery. Just a UI option that looks innocuous until it isn't.

The same thing had happened to multiple savesets, not just this one. So we spent the next few weeks loading every affected tape, running scanner against all the savesets, and manually resetting retention back to the correct compliance period.

🧠 What This Case Taught Me

Backup systems don't fail dramatically. They fail quietly — when a small mistake slips through and nobody notices until it's too late. No alarm, no error email, no red screen. Just a checkbox that meant something different than what you thought.

And ever since that case, whenever someone says "we didn't change anything" — I stop believing them immediately. Not because people lie, but because sometimes they genuinely don't remember that one dropdown three weeks ago that felt routine.

Always check the audit logs. Always.

Shibu A

Support Engineer & Infrastructure Admin with 5+ years handling enterprise backup systems, storage, and Linux environments. Currently based in Doha, Qatar.

LinkedIn Twitter