My First Tech Job: The Evolution of the DocWorks EMR System (2011-2013)

I don't usually get to tell these stories. When you've got profiles online and you want people to know about you quickly, you stick with the most recent specialties, the most recent tech. That makes sense because those are the most relevant. But I never get time to dig into the older stuff, and some of these stories are just fun to talk about. The way problems were solved, what was smart and what was stupid, things I did and things I was compelled to do.

One story I always wanted to tell is about one of the first jobs I ever had in technology: building and maintaining an electronic medical records system called DocWorks at a doctor's office here in town. This was around 2011 to 2013, maybe bleeding a bit into 2014. I was a full-time employee, primarily a programmer but also kind of general IT staff because it was a doctor's office on a hospital campus. Pretty much what I did all day was code. Full-stack development.

The Stack That Time Forgot

The stack was Windows Server. Most were Windows Server 2008, with some newer installs running 2012 and one really old holdout still on 2003. There were between 10 and 15 total clients. Each client was a doctor's office, and every office had their own server on-premises. All running IIS as a web server.

Here's where it gets interesting: they were not using any .NET at all. No Visual Basic, no C#. C# was actually new at the time and hardly used. The system was primarily a Perl app. Most of the codebase was written in Perl, and most of the data was stored in flat files. Files like exam.dat or patient.dat. Perl would use regular expressions and delimiters to format and store data. It wasn't XML, JSON, or CSV. It was a made-up standard, and it wasn't even consistent. As the system evolved, each script that retrieved data would vary from file to file.

For a long time, the entire system was built around reading files and had no database. None of the Perl code had any database connection logic because the database didn't exist for most of the development. They had a fully functional EMR with no database for a pretty good stretch of time.

The main languages ended up being Perl, Ruby, and PHP. There was very little JavaScript too. This was back when JavaScript had a stigma. In a professional enterprise context, it was basically a rule not to have JavaScript enabled. No respectable web developer wrote real scripting in JavaScript. You had to be compatible with devices that had it disabled.

The Parade of Developers

The history matters when talking about the stack because it evolved based on whoever was writing code at the time. Each developer left their fingerprints all over this thing.

The original developer was Dr. Chuck, a doctor who learned enough Perl to build a proof of concept. Once he had that, he got a programmer named Brandon to help build out the MVP. Brandon was the one who put the hours in and built the Perl codebase into something functional.

Then Jeff comes along. He's young, a hotshot, wants to learn the latest technology. Ruby on Rails was on fire. MVC architecture was gaining popularity. And honestly, it was atrocious that they had authentication and patient data not in a database. Jeff finds a way to bring Ruby on Rails into the mix, adds a MySQL database, and builds out the immunization system as its own standalone service. Early days of microservices, basically.

One problem: we're on Windows servers. In production, we were running a Linux emulator called Cygwin to run Ruby on Rails, which won't run in a Windows environment. We used a load balancer called Pound, configured the ports that Ruby was using, and forwarded everything in immunizations to port 80. I didn't build it, but I troubleshot it a lot.

Part of our workflow for triaging issues with the immunization system: restart Pound, edit the Ruby code, restart Ruby, flush the Ruby caches, restart Pound. We had a script for it. It was a routine part of the day. While you're trying to build new features and work through a backlog of bug fixes, calls come in that immunizations aren't working, so you remote into their server and run the restart script.

When they should have cleaned up the Ruby, integrated the rest into MVC patterns, and ported to Linux, instead Jeff goes off to college and uses his experience to get real jobs with exciting companies building Ruby apps. Good for Jeff.

Enter Terry and the PHP Era

They hired Terry, who is a big fan of PHP. Terry hates MVC. He wants procedural like the Perl codebase but hates Perl and loves MySQL. As a web developer in a small town, naturally he's in love with PHP. He hates frameworks, not interested in microservices. In a lot of ways, he wants to go back to Perl days but use MySQL. He brings PHP into the mix and starts rewriting everything.

This is around the time of Obamacare, the Affordable Care Act, and a lot of incentive money. That's when Jennifer comes along. Jennifer is an employee at a local computer shop doing contract work to help support the system. She's making a lot of sales; that was her strong suit. She comes in and helps close a bunch of deals using the incentive money, putting pressure on Terry to rewrite the system for compliance and to scale the user base.

There's no real version control. Deployments are bundling up files and FTPing them out to servers. On top of that, a growing user base who all want custom features. You've got disparate codebase versions across clients, no version control, no sane deployment process.

To keep clients and make sales, Terry is playing Wizard of Oz. Whenever someone needs a bug fix or a feature, he remotes in and makes the change at that location, on the fly, in production. Jennifer is making sales, and while we're developing new features and patching bugs, she's calling saying so-and-so has a problem. So-and-so is most likely failing to use the product properly. Every time we try to get immersed in real work, we have the salesperson telling us to stop and fix this one problem. It has to be the fastest fix, so we log in, make the change, and get back to the backlog.

Guess what happens when we build new features for compliance that have to push out everywhere? Anybody that got a feature or bug fix on the fly gets nuked. No version control and no system preferences. Gone.

The Most Epic Franken-System

I saw this pattern later in my career but never to this degree. DocWorks is still the most epic clusterfuck of a Franken-system. That's the term I coined for it. You run across these codebases where multiple junior developers learned to program on your production system. Each one brought their favorite language, their favorite architecture, their favorite opinions. And every single one of them is baked into the code you're maintaining.

There was even a native Windows Forms application for a printer and scanner thing where Jeff had hacked some drivers together. A lot of it was real hacker-style work, all running in production. I ended up building a web-based print feature to get around having to reverse-engineer the compiled code. I got a Visual Studio subscription and started looking at what he did, but we were trying to build production enterprise software that could scale, not just hack drivers.

I've still got that codebase archived somewhere. One of the last things I did when I left was get it running on my local machine, packaged it with all the executables and installers. All I'd need is a Windows 2008 VM and I could run it. With what I know now about DevOps and infrastructure, I bet I could get it running in Linux. It would be fun to stand it up, put it on the web, and invite people who used it to come tell their stories about it.

I don't usually get to tell these war stories because they're irrelevant to the work I'm doing now. I'm usually working with startups or driven enterprise teams where there's so much going on that nobody has time to talk about the past. But there are real engineering lessons buried in this mess, and honestly, it's just fun to gawk at the train wreck.