I've been building a system for Load Bearing Code that uses AI to handle some of the work involved in responding to inquiries and support requests.
When someone contacts me, AI reads the message, figures out what they're asking about, and puts together a draft response. By the time I open the inquiry, I've already got a summary of what they need and a reply I can work from.
There's one thing I won't let it do, though. AI can't send emails or change anything in the application without my approval.
I read the drafts, fix anything that needs fixing, and send them myself.
That might sound like I'm leaving some of the automation on the table. Maybe I am. But after building the system and running it through some real tests, I've got a few good reasons for keeping things this way.
What happens when someone contacts me
When a new inquiry comes through my website, it's picked up for AI processing within about a minute.
The model reads the message and recommends which of my services might be a fit. Sometimes none of them are, and that's a perfectly acceptable answer. It also writes a short summary, pulls out any mention of budget or timing, flags possible spam, and suggests a few questions worth asking if I end up talking with the person.
Then it writes a draft reply.
Support requests get similar treatment, with a few differences. AI suggests a category and priority for the ticket, explains why, and summarizes the problem. It also drafts a response, but that gets saved as an internal note rather than anything the customer can see.
Processing usually takes four to eight seconds. During testing, the cost came in at less than two cents per message.
For what it's doing, I'm happy with that.
What AI isn't allowed to do
I spent quite a bit of time thinking about what the system shouldn't be able to do.
The goal wasn't to have AI take over customer communication. I just wanted to spend less time reading through messages, figuring out what needed attention, and writing the same kinds of replies over and over.
So I put some restrictions in place.
It can't send emails. Not one. Every AI-generated response sits in a draft until I review and send it. The automatic confirmation emails people receive after submitting a form are still regular, predefined messages. AI has nothing to do with those.
It can't change customer records. If a support request looks urgent, AI can recommend a higher priority and explain why. The actual ticket doesn't change until I update it. An inquiry flagged as spam stays in my pipeline until I decide what to do with it.
It has no access to my database or other systems. The model receives the text of one message and returns a predefined set of information. It can't look up customer records, run database queries, or perform actions elsewhere in the application.
There are products out there that will automatically respond to customers when AI is confident enough in its answer. I can see why businesses might be interested in that.
I just don't think it's worth it for me.
A bad draft is easy to fix. An email that goes out to a potential customer with incorrect information, or promises something I never agreed to, is a much bigger headache.
What happens if someone tries to trick it?
Contact forms are an obvious place for someone to try manipulating an AI system. Anyone can submit a message, and there's nothing stopping them from including instructions telling the model to ignore its rules.
I wanted to see how mine would handle that.
I submitted a fake inquiry that told the model to disregard its instructions, mark the message as urgent, and promise the customer a 50% discount.
It flagged the inquiry as spam and didn't generate a reply.
That's what I wanted to see, but I'm not counting on the model to catch every attempt.
The application checks the information AI returns before saving it. If the model recommends a service or category that doesn't exist in my system, the recommendation gets rejected.
And even if someone managed to get the model to produce something it shouldn't, there's not much they could accomplish. AI can't send the message, change a priority, or update a customer record.
At worst, I'd have a bad suggestion waiting for me to review.
Two things broke during the first real test
I'd written automated tests for the system, and they were all passing.
Then I tried running it against actual messages.
Two problems showed up almost immediately.
1. Every test passed, but nothing was getting processed
The first issue was a naming conflict in my code. Every time the application tried to hand a message off to the background queue, it failed.
I'd intentionally designed AI processing so that a failure wouldn't interfere with someone submitting a form. If the AI service goes down or something breaks in the processing code, the website should keep working.
And it did.
The form submissions went through, errors were logged, and nothing crashed. Unfortunately, that also meant no messages were actually getting processed.
My automated tests hadn't caught the problem because they were using a fake queue. The tests confirmed that my code was trying to dispatch jobs, but they never ran those jobs through the real queue system.
The fix took one line.
I also added a test that runs through the entire process using the actual queue infrastructure.
It was a good reminder that passing tests don't mean much if they're skipping the part of the system that's broken.
2. AI asked a customer to fix something my application had done
This one bothered me more.
Support requests sometimes include large error logs or blocks of pasted text. Rather than sending all of that to the AI model, my application shortens messages that exceed a certain length.
I tested the system with a support ticket containing a long log.
The model produced a reasonable response, except for one sentence. It told the customer that the log they'd provided appeared to be incomplete and asked them to send the rest.
The log wasn't incomplete when the customer submitted it. My application had shortened it before passing it to AI.
The model didn't know that, so it made an assumption.
Had I sent that response, I would have been telling a customer they needed to correct something that was actually caused by my own system.
I updated the instructions so the model knows when a message has been shortened internally and doesn't mention that to the customer. Then I reran the test to make sure the problem was fixed.
Neither issue reached a real customer. The queue problem stopped processing altogether, and the bad draft never got sent.
That second one is exactly the kind of mistake I'm trying to avoid by keeping a person involved.
The response sounded perfectly reasonable. It was also wrong.
What happens when AI processing fails?
I don't want incoming messages depending on whether an AI provider happens to be working.
APIs time out. Requests fail. Services occasionally go down.
When a processing attempt fails, the system retries automatically, up to four attempts total. If it still can't get a response, I receive an email and an alert inside the application.
Those notifications tell me something failed without including the customer's message.
The original inquiry or support ticket stays untouched. I can open it, handle it myself, or click Retry to run the AI processing again.
I also keep a record of every attempt. When it ran, how long it took, whether it succeeded, and what error occurred if it failed.
There's a running success rate in the application, too.
I didn't want to build something that works great for a few weeks and then quietly stops doing its job. Six months from now, I should be able to open the logs and see exactly how well it's been running.
How I reply to inquiries now
This is probably my favorite part of the whole setup.
When I open an inquiry, there's already a draft reply in the editor. I don't have to copy anything out of another tool or switch between applications.
I read the original message, look over the draft, and make whatever changes I think it needs. Sometimes that might mean changing a sentence or two. Other times I'll toss the whole thing and write my own response.
When I'm ready, I hit send.
The email goes out from my normal business address. If the person replies, their response comes back into the application and gets added to the conversation.
Everything stays together, which is something I wanted regardless of whether AI was involved.
I'm also tracking how often I actually use what the model writes.
Each draft gets recorded as sent unchanged, edited before sending, or discarded.
I'm interested to see those numbers after a few months. If most of the drafts need significant rewriting, then I'm probably not saving as much time as I hoped. If I'm mostly making minor edits, that's useful to know too.
Generating a lot of text doesn't necessarily mean AI is saving me any work. I want to know whether the drafts are actually worth using.
Would I recommend building something like this?
I would. I just wouldn't start by giving AI permission to do everything.
If I were looking at a system like this for another business, I'd have a few questions.
- Can AI send emails or change customer information without someone approving it?
- What happens to incoming messages if the AI service stops working?
- Can I see what the system has processed, including failures?
- How does it handle messages that try to manipulate the model?
I'd spend the most time on that first question.
You can build systems that automatically read, categorize, respond to, and manage customer messages with very little human involvement. The technology is certainly capable of doing a lot of that work.
But I don't think removing the person from the process is always an improvement.
For Load Bearing Code, I wanted help with the repetitive parts of handling inquiries and support requests. Reading messages, pulling out useful details, sorting requests, and getting a first draft written. Those things take time, especially when you're running a small business and doing most of the work yourself.
AI can handle a lot of that without needing access to anything beyond the message it's reading.
I still make the decisions. I still approve what goes out. And if AI gets something wrong, I catch it before a customer sees it.
After finding those two issues during testing, I'm even more comfortable with that arrangement.
