Antipattern: Premature diagnosis

I preach against this, but got caught myself doing this. Taking a set of facts and developing a conclusion then using confirmation bias to make sure that's the conclusion that wins.Here's the deal. We have a system that runs in the background on a server and pulls up customer records over time to send them reminder emails. Our system also encrypts a lot of the sensitive user info as well.So things are going all honky-dory until one week when things go amiss.

• We change up the client credentials (best practice to rotate these things occasionally)
• We notice some errors from this system, specifically around encryption
• Somehow the metrics associated with processing change

Ok, obviously we screwed something up around the encryption. So we futz around and make sure everything is up-to-date on that server.

• Metrics go up again

At this point we're spiking the ball in the end-zone. Problem solved, lets just backfill the stuff that errored out.But the errors from the system don't stop.Huh?Between fighting other fires (crazy week) we try to fix the obvious on-again-off-again encryption problem to no avail. No matter what we do the errors don't stop.Yesterday I finally had a chance to dig a bit deeper into the problem. Up until then I hadn't had a chance to come up for air. While I was occupied another coworker that was in deeper on the encryption end had been chasing the problem when he had time.What I found was interesting. The error that we were getting, wasn't really an error; it was more of a warning when I looked at the code that was generating it. But since everything fit what we were thinking we just ran with it.It's a classic case of "ready, fire, aim."And it's not that I didn't know it before. This is a good reminder to validate your assumptions. Just because what you are seeing fits with your theory doesn't mean you're right. It's always worth taking a step back to assess the situation a bit more even if you think you know what the problem is.Now to figure out what the problem actually is.<sigh/>Live and learn....and then forget... and learn again.