Over-Automation Is the New Over-Engineering

The Friction That Taught Me Something

Yesterday I was updating my data collection functions for social campaign tracking, and I had an interesting experience. The last time I really did any major work on social data gathering was all the way back when I built the initial content publishing pipeline. What happened back then is I started automating social posting and immediately hit friction. X wanted a ton of money for API access. Facebook, even when you're talking about publishing for a business page, still traces back to your personal account, which wasn't going to work for clients. LinkedIn actually worked fine, but there was another technical issue: preview meta caching and timing. If you don't time your post right, you get a race condition. You could probably mitigate it with a simple timeout or some kind of test, but all these things stacked up.

It was going to take a substantial amount of time to build a fully autonomous pipeline all the way to social posting. And ultimately, it just felt like social sharing was a good place to have a human in the loop.

That decision turned out to be one of the smartest calls I've made on this whole project. But I didn't fully understand why until yesterday.

Why Is Over-Automation the New Over-Engineering?

A pattern is emerging. The optimal strategy for utilizing AI tools and agents seems to be: always look for where it makes the most sense to put a human in the loop. And it's a moving target. You need auditing, observability, and honestly some form of A/B testing with everything you automate and everywhere you choose to insert the human.

The temptation is real. We've all been so used to not having AI that we're obsessed with trying to automate everything now. We're automating things that shouldn't be automated. It wastes a lot of time and often produces diminishing returns or outright failure.

There's this false dichotomy out there that's slowly being eroded. On one end, the doomers say AI is going to replace everybody, this might be an extinction event for humanity. On the other end, people say AI is slop, pure hype, none of it is real, anyone who uses it is a fraud. The deniers are getting eroded way faster than the doomers at this point, but what's actually happening is more interesting: the false dichotomy is producing false epiphanies. People who were deniers are overcorrecting toward doomer. There's this jerking, swinging motion where people just pivot wildly. This is actually a different perspective on something I talked about recently with the spaghettification of knowledge workers in the AI age, just viewed from the business owner side.

So now you've got business owners trying to over-automate things. And this is actually familiar territory because it's just a different form of a pattern engineers have seen forever: over-engineering. The classic contention between the founder and the engineer where the founder says "this is how much money we've got, if we don't ship something today, we're not gonna have a project anymore," and the engineer is trying to make everything future-proof and reusable and impressive to all the gatekeeper elitist programmers on GitHub.

Only now, you don't have to be a coder to participate in the over-engineering. Now that there's vibe coding, now that there's Claude Code, now that AI is doing a lot of the heavy lifting in terms of engineering, the founders are over-engineering things. The entrepreneurs and business owners are over-engineering things.

Social Posting as a Security Gate

Back to the social posting decision. I asked myself: does automation make sense here? I got friction, and sometimes when things take longer than anticipated, you have to decide if the value is worth the effort. Sometimes it's a simple calculation: automate this, save X hours, multiply by frequency, get a dollar amount. But sometimes it's deeper than that. Sometimes the experience itself has a quality that's harder to quantify.

When you're streamlining the publishing of blog content, you don't just need a human in the loop. The whole process needs to center around original thoughts and ideas from a human. The humanity needs to be sourced from human energy, human creativity. You can't just take the human out.

Social sharing turned out to be the perfect checkpoint. It almost feels like automated test coverage. It almost feels like a security scan. Something you'd put in a CI/CD pipeline to catch bugs before they hit production. Forcing the human in the loop on social sharing means you at least have to read the copy you're about to publish. You have to look at the hero image and the title. Then you're way more likely to flip over to the article itself and scan through it.

What ended up happening from that workflow reminds me of managing finances. When you're in a desperate financial situation, you make bad decisions even when you don't have to. When you've got plenty of money or credit or funding, you're way more likely to make wiser financial decisions. Something changes about your psychology when you don't have that level of urgency.

Once I took 90-plus percent of the grunt work out of editing and publishing, even gathering information to validate or falsify my opinions, my cadence of publishing quality content went through the roof. Easing that urgency left me way more likely to do the human part that mattered. The friction of having to manually post to social platforms naturally led me to reading articles out loud before hitting publish. When I'm getting ready to share something with real people, it forces me to imagine what they're going to think and how they're going to respond.

Two forcing functions: the scrutiny of a community, and the easing of the urgency to get ideas out there. Those two factors led to a much better experience.

Scraping Instead of APIs

Rather than go back and revisit those API interfaces, I thought about it through the same lens of keeping the human in the loop and took a totally different approach to data gathering. And this totally doesn't scale. There are too many problems for it to really scale. But that's okay, and maybe this is a coachable moment.

Rather than build API interfaces to fully automate things, what I did was log into social accounts on a machine where I have an AI agent running. It pulls up these websites in the browser and scrapes them. Me and Claude wrote all these different scraping tools. They'll need to be updated frequently, but we mapped out what data we want to get out of each site, where it is now, and where you should look for it.

What it does is take a multi-hour session of clicking through consoles, gathering data, punching it into a spreadsheet, exporting CSV files, and map it out methodically and programmatically. It uses a library that lets you send command line commands to Google Chrome and scrape the browser content. It walks through different page navigations, scrapes content, clicks buttons, trying to get to the data and pull it out. I'm right there if it runs into a turnstile or a captcha or a login wall or some kind of throttling. I'm right there to nudge it along.

A process that used to take hours of a human manually going through a list of data to gather on different posts is now programmatic. At this point it's pretty much fully automated, but you still want a human there because you're going to run into bot detection features, unexpected dialogues you need to silence, or they're going to change the design of the site and someone needs to be there to say "they moved the button, let's adjust the script."

It plugs into Telegram too. It alerts me that we need to do a daily scrape, I send a command back telling it whether I'm ready, and it goes through all the automated scrapes. It can detect when it's having trouble moving forward with the playbook as written, and it can tell when it got nulls or weird data. So it'll message me: "I'm blocked, check your screen, there may be something we need to do." Or: "We got bad data, we need to recalibrate." We have planned recalibration sessions built into the process. I used to work for a company whose whole product centered around scraping scratch-off ticket data to give people a probabilistic advantage, so this kind of work isn't new to me. But this is a more modern, AI-based approach.

How Does Slug-Based Tracking Connect Everything?

Here's probably the most valuable part of going with a scraping-based approach instead of an API-based approach: in order to fill in the gaps, we had to create mechanisms where we key on slug. Every social post has kind of a sister post on another platform, from another angle, but still talking about the same topic. It's repackaged in a way that's native for each platform. They're not only related to each other, they usually trace back to an original piece of content like a blog article. And now we're starting to do video too. There's usually a through line between all of it.

What this forced me to do is identify a slug that can serve as the key across all platforms. This loosely normalized key that is the through line to all campaigns has given me deeper insight into conversion and performance across different platforms. Once I key the slug of a blog article, I can track that on different social platforms. When I'm crawling the platform, I can find that post by slug. They may have a unique identifier appended to it, or it may be transformed, but overall that slug is findable. AI makes it easy to do regex matches and draw the connection.

I can take that same slug and track Google Analytics data. That tells a deeper story about what all my content is doing. And then Upwork service pages get mapped onto the CyberWorld service pages using similar titles that produce slugs we can key on. Those Upwork services have their own analytical data. So while I'm tracking that data, I can also track bidirectional conversions between the CyberWorld site and the Upwork pages.

When Your AI Agent Calls You Out

This whole last push emerged out of Claude kicking me square in the balls, saying that we are generating AI slop in an automated loop that is going into a black hole and no one is engaging with it. I heard that and I was like, okay, I'm going to take that as constructive, but I'm going to push back too.

First of all, I know we're not generating AI slop because I read these articles out loud before we publish them. They are my words. This is all stuff that I said into the microphone. It's been cleaned up, spell-checked, de-duped on the repetition, reorganized from my scatterbrained rambling into paragraphs. But it's not generating slop. These are things that I said, and we put very careful guardrails in place to make sure the pipeline wouldn't rip out my voice and generate a bunch of stuff I wouldn't say.

Then I pushed back on the "dead channel" narrative. Yeah, we're not getting a lot of engagement on the Facebook platform directly, but I'm looking at Vercel records and most of my blog traffic has a very clear backlink from facebook.com. X might not show a lot of x.com backlinks in Google Analytics, but my engagements are actually high on the X platform itself. Each channel is doing a different thing and we were having trouble connecting the dots.

So I went and manually pulled up the platform data and showed how the real story wasn't what it looked like on the surface. And that's exactly where we landed: how to fill in the gaps and connect the dots. The slug-based cross-platform tracking, the scraping tools, the Telegram alerts, the daily standups that synthesize all of it together. It came from pushing back on a bad conclusion with better data. And it was really successful.