Technical Strategy & Debt

Persistence

The hidden tax of custom code and the cost of building your own plumbing.

Custom code is rarely an asset. It is usually a debt that grows teeth. We celebrate the day we finish a custom module. We feel like architects. We feel like we saved the company money.

This is a lie we tell to sleep better. Every line of custom code is a future headache. It is a promise to fix things later. It is a burden for the next developer. We think we are building a bridge. Often, we are just digging a hole.

The Rhythmic Failure

. The Slack channel is a graveyard of productivity. There are 34 replies in one thread. A screenshot shows a customer’s phone. It has three “Order Confirmed” messages.

The timestamps are eleven seconds apart. This is a rhythmic failure. It is the signature of a retry loop. Someone suggests looking at the retry module. There is a long silence. Someone else says they would rather not touch it. They would rather do anything else. They would rather go to the dentist.

11:14:01

Confirmed

11:14:12

Confirmed

11:14:23

Confirmed

The rhythmic failure: Three identical confirmations sent exactly 11 seconds apart.

The Ghost in the Code

The module is called Persistence.js. It has 732 lines. A contractor wrote it in one afternoon. He was under a tight deadline. He was a “one-man army.” He is now working in another country. Nobody has his phone number.

The first half of the file makes sense. It defines variables. It sets simple timeouts. The second half is a dark forest. It contains nested loops. It has “idempotency handling.” It has custom logic for edge cases.

Last month, it failed. It sent 900 customers three identical confirmations. The team did not fix the code. They just reduced the retry count. They hoped for the best. Hope is not a technical strategy.

I understand the fear of the code. I once joined a video call by accident. My camera was on. I was eating a massive, messy burrito. I felt exposed. I felt unprepared.

Looking at Persistence.js feels the same way. You see the flaws. You see the shortcuts. You feel the vulnerability of the system. You want to look away. But you cannot. The system is live.

🌯

Personal Exposure

The messy burrito moment.

📄

Technical Exposure

The Persistence.js codebase.

The Real Cost of “Cheap”

The module exists because it was cheap. It was built in three weeks. Buying a solution felt expensive then. The quarterly budget was tight. The contractor cost five thousand dollars. A subscription would have cost a few hundred.

In that quarter, the math looked good. The savings were real. But the cost was just moved. It was moved into the future. It moved into the maintenance risk. It moved into the onboarding difficulty.

New developers see the module and panic. They take three days to understand one bug. That time is expensive. The bill has arrived.

Quarter 1: One-time Contractor

$5,000

Future: Maintenance & Risk

$$$ Unknown

Invisible Debt: The immediate “savings” vs. the compounding future maintenance cost.

Anatomy of a Retry Module

The retry module has three main parts:

1. The Backoff Strategy: It manages wait times.

2. The Deduplication Table: It prevents double actions.

3. The Ghost Logic: It handles unknown errors.

Let us define the Backoff Strategy. It is the delay between attempts. Imagine a crowded room. If everyone talks at once, no one is heard. If they wait and try again, it works. We use “Exponential Backoff” for this. The wait time doubles each time.

Five seconds becomes ten. Ten becomes twenty. This prevents the server from crashing. If you use a professional whatsapp api, these layers are managed for you. You do not have to write them. You do not have to fear them.

The Idempotency Paradox

The Deduplication Table is even harder. It requires an Idempotency Key. Define an Idempotency Key. It is a unique tag for an action. Imagine a light switch. Flipping it “on” twice has the same result. The first flip works. The second flip does nothing.

The key tells the server the request is old. If the server sees the same key, it ignores it. But what if the table is full? What if the database is slow? The logic breaks. The duplicate messages fly out. The customers get angry.

💡

The Idempotency Key

Action 1: “Turn Light ON” → SUCCESS

Action 2 (Retry): “Turn Light ON” → NO CHANGE (Idempotent)

Ella L.-A. is a mediator I know. She handles complex human conflicts. She told me something once.

“A compromise made in fear is just a conflict waiting for a better time to explode.”

– Ella L.-A., Mediator

This applies to code. The contractor was afraid of the deadline. He made a compromise. He wrote complex logic to save time. He did not write tests. He did not write documentation. He just made it work. Now, the conflict has exploded. The 900 duplicate messages were the explosion.

The Invisible Accounting

We often optimize for the local moment. We look at the immediate task. We ignore the aggregate cost. This is a rational decision in a vacuum. It is irrational in a business.

No accounting system records this trade. The CFO does not see the “Maintenance Risk” line. They only see the “Contractor Expense” line. They see the “Software Subscription” line. They want to lower the second one. They do not realize they are raising a third, invisible one.

What the CFO sees:

[✓] Contractor Invoice

PAID

[✓] SaaS Monthly Fee

$0 (SAVED)

What the CFO misses:

[!] Operational Fragility

HIGH

[!] Developer Attrition Risk

CRITICAL

The retry logic is a black box. It sits in the middle of the stack. It handles every message. It handles every notification. It is the most important part of the app. Yet, it is the part everyone fears.

People treat it like a sleeping bear. You walk past it quietly. You do not poke it. You do not try to clean its cage. You just hope it stays asleep. But systems do not stay asleep. They grow. They change. The network fluctuates. The database scales. Eventually, the bear wakes up.

Persistence.js: The sleeping bear of your infrastructure.

Last week, a junior developer tried to fix it. He spent six hours. He added three lines of code. He deleted two lines. Then, the staging environment crashed. He rolled back the changes. He apologized in Slack.

The senior lead told him it was okay. “Nobody understands the second half of that file,” he said. This is a tragedy. We have built a system we cannot control. We have saved five thousand dollars. We have lost our peace of mind.

Focus on the House

Building plumbing is a trap. Sending a message seems easy. Handling the failure of that message is hard. The network is a chaotic place. Packets get lost. Servers go down. APIs time out. You have to handle every scenario.

You have to manage the state of every attempt. This is not “feature” work. This is plumbing. Most companies should not build their own plumbing. They should focus on their house. They should focus on the product.

Plumbing

Retries, Logics, Network Errors, Latency

VS

The House

Product Experience, Customer Value, Features

When you build your own gateway, you own the debt. You own the 700 lines of logic. You own the bugs. You own the midnight alerts. When you use a service, the debt is theirs.

They have an entire team for retries. They have developers who love backoff logic. They have tests for every edge case. They have documentation. You pay a monthly fee.

You are not just buying a tool. You are buying an insurance policy. You are buying the right to not care about Persistence.js.

The Real Invoices

The contractor is gone. The code remains. Every time there is a network hiccup, the team holds their breath. They watch the logs. They wait for the triple messages. They wait for the Slack thread to start.

This is the hidden tax of “doing it yourself.” It is a tax on your focus. It is a tax on your sleep. It is a tax on your company’s agility. You cannot move fast if you are afraid of your own code.

We need to stop praising local optimization. We need to look at the aggregate. We need to ask what a module will cost in three years. Not just what it costs today. If the answer is “we will be afraid to touch it,” then we should not build it.

We should buy it. We should find a provider who specializes in that specific problem. We should keep our codebase small. We should keep our minds clear.

The bill for the contractor is never really paid. It is just refinanced. We pay it every day we avoid the file. We pay it every time we skip a feature because we are busy fixing a ghost. We pay it in the frustration of our team. The savings were real for one quarter. The cost is real for a lifetime.

The contractor’s invoice was paid in dollars, but the interest is paid in every duplicate message that haunts the customer’s phone.

We must choose our complexity carefully. Some complexity is necessary. It is the core of the business. It is the secret sauce. Other complexity is just a burden. It is the plumbing. It is the retry logic. It is the idempotency handling.

Do not build your own labyrinth. You might get lost in it. Or worse, you might have to live in it. It is better to have a clean house. It is better to have code you actually understand. It is better to sleep through the night.

Let the specialists handle the black boxes. You have more important things to do. You have a business to run. You have customers to serve. Do not let 700 lines of fear hold you back.

Categories:

Tags:

Comments are closed